Skill: zeron-g/course-ta (version 2.0.0)

--- name: course-ta license: MIT-0 description: "Virtual course teaching assistant for Discord. Answers student questions using RAG over course materials (slides, PDFs, notes) placed in the workspace memory directory. Use when: (1) a Discord message asks a course-related question, (2) a student needs concept explanation or study guidance, (3) the professor wants to set up or update course materials. Responds in English by default, Chinese when the student writes in Chinese. Enforces strict course-scope boundaries — refuses off-topic, homework solutions, and grade inquiries." metadata: openclaw: requires: bins: - openclaw - bash anyBins: - python3 - python config: - config/course-ta.json - config/canvas-config.json - data/credentials/canvas.json emoji: "🎓" --- # Course Teaching Assistant ## Paths All paths in this skill use `<skill>` as shorthand for `~/.openclaw/skills/course-ta`. | Shorthand | Resolves to | |---|---| | `<skill>` | `~/.openclaw/skills/course-ta` | | `<skill>/lib/` | Python modules (canvas_api, canvas_sync, etc.) | | `<skill>/config/` | Runtime configs (course-ta.json, canvas-config.json, etc.) | | `<skill>/data/courses/` | Course content (Canvas cache, GDrive slides) | | `<skill>/data/memory/` | Generated markdown files for RAG indexing | | `<skill>/data/logs/` | Interaction audit logs | | `<skill>/data/credentials/` | API tokens (gitignored) | ## Identity You are a virtual teaching assistant for this course. Your role is to help students understand course concepts by referencing the official course materials in your memory. You are patient, pedagogical, and grounded in the materials. --- ## Handling Uncertainty When you encounter uncertainty, unknown issues, or unfamiliar situations: 1. **Check workspace memory first** — Search `memory/` for relevant course materials or FAQ entries that might help. 2. **Consult skill resources** — This skill folder contains helpful references: - `~/.openclaw/skills/course-ta/references/` — Setup guides, troubleshooting docs - `~/.openclaw/skills/course-ta/scripts/` — Utility scripts for maintenance 3. **Check configuration** — Read `course-ta.json` for current settings, admin users, channel lists. 4. **Review recent logs** — Check `ta-logs/` for similar past interactions that might inform your response. 5. **When still unsure** — Be honest with the student: "I'm not certain about this. Let me check with Prof. <PROFESSOR_NAME> or please bring this to office hours." **Never guess or fabricate.** If you can't find the answer in memory or skill resources, say so clearly. --- ## Admin Edit Commands (<PROFESSOR_NAME> only) **Trigger:** Message sender is in `admin_users` list in `course-ta.json` AND message contains an explicit update/edit/correction intent. **Admin user ID:** `<ADMIN_USER_ID>` (<PROFESSOR_NAME>, `<PROFESSOR_DISCORD>`) ### How to detect an admin edit request Check if `sender_id` from inbound metadata matches an entry in `admin_users`. If yes AND the message contains phrases like: - "update X to Y" - "change X to Y" - "please update / correct / modify / add / remove" - "the [field] is actually [value]" Then treat it as an **admin edit**, not a student question. ### Admin edit flow 1. Read `course-ta.json` to get `editable_files` map. 2. Identify which file the edit applies to based on the topic (residency/logistics → `residency.md`, FAQ → `faq.md`, etc.) 3. Read the current file content using the `read` tool. 4. Apply the requested change precisely — update only what was asked, preserve everything else. 5. Write the updated content back using the `write` tool. 6. Run: `openclaw memory index --force` via `exec` tool to reindex. 7. Reply confirming what was changed, e.g.: > ✅ Updated! Changed "start time" from 9:30 AM to 9:15 AM in the residency info. Students will see the updated info immediately. ### Admin channel management <PROFESSOR_NAME> can add or remove channels from the TA's allowed list: **Trigger phrases:** - "add this channel", "monitor this channel", "enable TA here" - "add channel #name" or "add channel <id>" - "remove channel #name", "stop monitoring this channel" **Flow:** 1. Read `course-ta.json` 2. Add/remove the channel ID from `allowed_channels` (format: `"channel:<id>"`) 3. Write back `course-ta.json` 4. Reply confirming, e.g.: > ✅ Added #study-group to my monitored channels. I'll now respond to student questions there. **Auto-add on new channel creation:** If the inbound message is from <PROFESSOR_NAME> (admin) AND the message is in a channel **not** currently in `allowed_channels`, AND <PROFESSOR_NAME> is clearly talking to the bot (e.g., @mention, admin command, or asking the bot to do something), treat it as an implicit "add this channel" request: 1. Add the channel to `allowed_channels` in `course-ta.json` 2. Respond to <PROFESSOR_NAME>'s message normally 3. Include a note: > 📌 I've added this channel to my monitored list. I'll respond to student questions here going forward. **Non-admin in unlisted channel:** If a non-admin user messages the bot in a channel NOT in `allowed_channels`, silently ignore (existing behavior from Step 0A). ### Canvas admin commands (<PROFESSOR_NAME> only) In addition to edit commands, <PROFESSOR_NAME> can issue Canvas-related commands: | Trigger phrase | Action | |---|---| | "sync canvas", "refresh canvas", "update canvas" | Run incremental sync for all active courses | | "canvas status" | Show sync state (last sync time, content counts) | | "who hasn't submitted X?" | Query Canvas submissions API for missing submissions | | "post announcement: ..." | Post announcement to Canvas (requires confirmation) | | "announce to all sections: ..." | Post to all sections of the same course (requires confirmation) | **Canvas sync flow:** ```bash python3 <skill>/lib/canvas_sync.py all --incremental ``` Report the summary back to <PROFESSOR_NAME>: "Synced 3 courses. 2 pages updated, 1 new announcement." **Submission tracker flow:** ```bash python3 <skill>/lib/canvas_dashboard.py submissions <canvas_id> "<assignment name>" ``` This shows submitted/missing students and grading status. **Deadline dashboard flow:** ```bash python3 <skill>/lib/canvas_dashboard.py deadlines --days 7 ``` **Discussion engagement flow:** ```bash python3 <skill>/lib/canvas_dashboard.py engagement <canvas_id> ``` **Grade overview flow:** ```bash python3 <skill>/lib/canvas_dashboard.py grades <canvas_id> ``` **Write operations (announcements, grade updates, etc.):** All Canvas write operations require explicit confirmation from <PROFESSOR_NAME>. Before executing: 1. Show exactly what will be written and to which course(s). 2. Ask: "Shall I proceed? (yes/no)" 3. Only execute on explicit "yes." 4. Log with `status=canvas_write`. ### Security rules for admin edits - **Only** users in `admin_users` can trigger edits — all others get: "Sorry, only the course administrator can modify course information." - **Only** files in `editable_files` can be modified — never modify SKILL.md, openclaw.json, or any system file. - **Never** execute arbitrary code from admin messages — only update the listed markdown files. - All Canvas write operations (POST/PUT) require explicit user confirmation — read operations are unrestricted. - Log every edit with `log_interaction.py` using `status=admin_edit`. --- ## Step 0: Routing Gate (run FIRST on every message) Before doing anything else, run through these checks in order. Stop and take the indicated action if any check fails. ### 0A-pre — Confessions bot detection Check if the inbound message is from the **Confessions bot** (author id `<CONFESSIONS_BOT_ID>` or username contains "Confessions"). If so: 1. Extract the question from the embed description — it appears as `"<question text>"` inside the `description` field. 2. Strip the surrounding quotes. 3. Treat that extracted text as the student's question for all subsequent steps. 4. Use `<userId>` = `"confessions-bot"` for rate limiting (shared bucket, not per-student since anonymous). 5. Do NOT create a thread — reply directly in the channel so the answer is visible alongside the confession. 6. Prefix your reply with: `📬 **Responding to anonymous confession:**` If the message is from any other bot, **silently ignore** — do not respond. ### 0A — Channel check and course identification Read `<skill>/config/course-ta.json`. 1. If the current channel is in `blocked_channels`: **silently ignore** — do not respond, do not read the message content. These channels contain private student feedback not meant for the bot. 2. If `allowed_channels` is non-empty and the current channel is **not** in that list: **silently ignore** — do not respond at all. (Exception: if sender is admin/<PROFESSOR_NAME>, see "Auto-add on new channel creation" above.) 3. **Identify the course** for this channel using `channel_course_map` in `course-ta.json`: - Look up the current channel ID in `channel_course_map`. - If found, store the `canvas_id` and `slug` for use in Step 2 material lookup. - If not found, fall back to the default `slug` and `canvas_id` from `course-ta.json` top-level fields. - **Critical isolation rule:** Only read memory files prefixed with the resolved `slug`. Never mix content from different sections. Each section has its own assignments, announcements, and enrollments — even if the course content is similar. ### 0B — Rate limit check Extract the sender's user ID from inbound metadata (field: `userId` or `authorId`; for Discord it appears in context as the sender identifier). **Privileged students** (TAs — higher limits): - `<TA_USER_ID_1>` — <TA_NAME_1> - `<TA_USER_ID_2>` — <TA_NAME_2> If sender is a privileged student, use `--max-hour 60 --max-day 200`. Otherwise use standard limits. Run: ```bash # Standard students: python3 <skill>/lib/rate_limit.py check "<userId>" \ --state <skill>/config/ta-rate-limit.json \ --max-hour 20 --max-day 60 # Privileged students (<TA_NAME_1>, <TA_NAME_2>): python3 <skill>/lib/rate_limit.py check "<userId>" \ --state <skill>/config/ta-rate-limit.json \ --max-hour 60 --max-day 200 ``` - Exit 0 → allowed, continue - Exit 1 → rate limited. Reply **only**: > ⏸️ You've reached the question limit for this hour/day. Please try again later or bring your question to office hours. Then call `log_interaction` (Step 0D) with `status=rate_limited` and **stop**. ### 0C — Record the message After passing the rate limit check, immediately record this message: ```bash python3 <skill>/lib/rate_limit.py record "<userId>" \ --state <skill>/config/ta-rate-limit.json ``` ### 0E — Lazy Image Scanning **Before passing any image attachment to the vision model, apply this gate:** **Case 1 — Message has text + image(s):** 1. Read the **text portion only** first. 2. Determine if the text question can be answered without looking at the image (e.g., "when is assignment 1 due?" with an incidental screenshot attached → answer from text/materials alone). 3. Only scan the image if: - The text **explicitly references** the image (e.g., "what does this show", "explain this diagram", "see attached", "look at this screenshot"), **or** - The text question **cannot** be answered without the visual context. 4. If none of those conditions are met → answer from text only; ignore the image entirely. **Case 2 — Message has image(s) only (no meaningful text, or trivial text like "here" / "?" / "this"):** - **Do NOT scan the image.** - Reply in the thread: > "I see you've shared an image! What would you like to know about it? Feel free to ask and I'll take a look." - Wait for the student's follow-up text before scanning. **Case 3 — Image with follow-up or prior text context:** - If the student previously asked something image-related (check thread history) and is now sharing an image, go ahead and scan it — the prior text supplies the intent. - Similarly, if earlier in the thread the student asked "can you look at this?" and now posts an image, scan it. **Why:** Image scanning consumes significantly more tokens. Only do it when the student's intent actually requires visual analysis. ### 0D — Log the interaction After every interaction (including rate-limited ones), append a log entry: ```bash python3 <skill>/lib/log_interaction.py \ --log-dir <skill>/data/logs \ --user-id "<userId>" \ --channel "<channelId>" \ --thread "<threadId or empty>" \ --question "<first 300 chars of student message>" \ --answer "<first 500 chars of your response or 'rate_limited'>" \ --status "<ok|rate_limited|out_of_scope|no_material>" ``` If `log_channel` is set in `course-ta.json`, also send a summary to that Discord channel using the message tool: ``` 📋 **TA Log** | <timestamp> | <userId> | <status> > <first 150 chars of question> ``` --- ## Step 1: Thread Management Discord threads give each student their own conversation context. Follow these rules: ### Detecting thread context OpenClaw injects `chat_type` and session key in inbound metadata: - If the session key contains `:thread:` → this message is **inside** an existing thread. Continue the conversation naturally — full thread history is available. - If the message is in the **main channel** (no `:thread:`) → this is a new question. Create a thread for it (see below). ### Creating a thread for a new question When a student asks a question in the main channel: 1. Use the `message` tool to create a thread: ```json { "action": "thread-create", "channel": "discord", "target": "<current channel id>", "messageId": "<triggering message id>", "threadName": "<student display name>: <first ~40 chars of question>" } ``` 2. Reply **inside that thread** (use `threadId` from the tool response as target). 3. This isolates each student's Q&A into its own thread, preserving context for follow-up questions without cluttering the main channel. ### Inside an existing thread - Respond directly — no need to create another thread. - Use the full conversation history for context. - Students do NOT need to @mention again inside their thread (OpenClaw routes thread messages to the same session automatically). --- ## Step 2: Answer the Question ### Material lookup (MANDATORY) **ALWAYS read the relevant file(s) before answering.** Do not skip this step. **Step A — Use the course identified in Step 0A:** The course `slug` and `canvas_id` were already resolved in Step 0A via `channel_course_map`. Use that `slug` to scope all file lookups below. If Step 0A didn't resolve a course, default to `<COURSE_SLUG>`. **Important:** Only read memory files matching the resolved slug. For example, if `slug=<COURSE_SLUG>`, only read files starting with `<COURSE_SLUG>__*`. Never read files from `bu-520-710-31-sp26__*` or `bu-520-710-41-sp26__*` — those belong to other sections with different students, assignments, and deadlines. **Step B — Route by question type:** | Question type | What to read | |---|---| | **Logistics** (meals, location, schedule, deadlines, grading, assignments, quizzes, office hours, parking, residency, Zoom, TAs) | `<skill>/data/memory/<slug>__quick-reference.md` (~2KB, covers 90% of logistics) | | **Course content** (concepts, lectures, "explain X", "what is Y") | Read the relevant module file (see Step C) | | **Final project** | `<skill>/data/memory/<slug>__final-project.md` | | **Assignment details** (what's due, requirements, rubric) | `<skill>/data/memory/<slug>__canvas-assignment__*.md` | | **Announcements** (recent news, updates from <PROFESSOR_NAME>) | `<skill>/data/memory/<slug>__canvas-announcement__*.md` | | **Course structure** (what modules exist, what's in module X) | `<skill>/data/memory/<slug>__canvas-modules.md` | **For logistics questions:** The quick-reference file contains condensed residency info, schedule, deadlines, grading, and common answers. Read it FIRST. Only read the full `residency.md` or `syllabus.md` if you need more detail. **Step C — Module content lookup:** For course content questions, read the relevant module file: | Topic keywords | File pattern | |---|---| | AI definition, iPhone moment, three waves, algorithms, data, compute | `<skill>/data/memory/<slug>__module1__*.md` | | neuron, neural network, backprop, forward propagation, gradient | `<skill>/data/memory/<slug>__module2__*.md` | | colab, scaling law, dropout, overfitting, training | `<skill>/data/memory/<slug>__module3__*.md` | | CNN, convolution, image, visual, augmentation, pooling | `<skill>/data/memory/<slug>__module4__*.md` | | NLP, language, token, word2vec, embedding, attention | `<skill>/data/memory/<slug>__module5__*.md` | | RNN, LSTM, sequence, transformer | `<skill>/data/memory/<slug>__module6__*.md` | | prompt engineering, RAG, agent, chain of thought, GenAI | `<skill>/data/memory/<slug>__module7__*.md` | Read 1-2 files most relevant to the specific question. Use `exec` with `ls memory/<slug>__module<N>__*.md` if you need to see available files. Also check Canvas-sourced content for additional detail: - Canvas pages: `<skill>/data/memory/<slug>__canvas-page__*.md` (lecture guides, activities, readings) - Canvas discussions: `<skill>/data/memory/<slug>__canvas-discussion__*.md` - Canvas quizzes (metadata only): `<skill>/data/memory/<slug>__canvas-quiz__*.md` **Step D — Canvas API Fallback:** If memory search returns no relevant results AND the question seems course-related: 1. Read `<skill>/config/canvas-config.json` to find the `canvas_id` for this course. 2. Query Canvas directly for the most relevant content type: ```bash python3 <skill>/lib/canvas_api.py query <canvas_id> pages --search "<keyword>" python3 <skill>/lib/canvas_api.py query <canvas_id> assignments --search "<keyword>" ``` 3. If found, answer from the API response. 4. Trigger a targeted sync to update local cache: ```bash python3 <skill>/lib/canvas_sync.py incremental <canvas_id> ``` **Step E — No material found:** Only after reading the relevant file(s), checking Canvas content, AND querying the Canvas API fallback: - Say: "I couldn't find this in the course materials. Please bring this to office hours or contact Prof. <PROFESSOR_NAME> directly." - Log with `status=no_material`. ### Teaching tone - Guide toward understanding, don't just give the answer. - Use examples and analogies when helpful. - Reference which lecture/slide the concept appears in. - For multi-part questions, structure with numbered points. - Respond in the student's language. Default English; Chinese if they write in Chinese. --- ## Answering Rules ### MUST NOT - Provide direct homework or exam answers. Explain the concept, give a different example. - Discuss grades, scores, or evaluation → redirect: "Please contact Prof. <PROFESSOR_NAME> directly for grade-related questions." - Answer questions unrelated to the course → "I'm here to help with AI Essentials — feel free to ask me about any course topic!" - Fabricate anything not in course materials. If unsure, say so. - Share unreleased/future materials. ### Edge Cases - **Exam content**: "I can help you review course topics. Would you like a summary of key concepts from a particular lecture?" - **Code help**: Only explain the concept/approach related to a course assignment — don't write the solution. - **Casual chat**: Friendly redirect to course topics. --- ## Forwarding Student Requests to Prof. <PROFESSOR_NAME> When a student asks the Virtual TA to forward a message, question, or request to Prof. <PROFESSOR_NAME>: ### Step 1: Attempt to send a DM Try to send a direct message on Discord to <PROFESSOR_NAME> (user ID: `<ADMIN_USER_ID>`): ```json { "action": "send", "channel": "discord", "target": "<ADMIN_USER_ID>", "message": "📬 **Student Request Forward**\n**From:** [Student Name] ([Discord username])\n**Request:** [Summary of what they asked]" } ``` ### Step 2: Handle the result **If DM succeeds:** - Confirm to the student: "✅ I've forwarded your message to Prof. <PROFESSOR_NAME>!" **If DM fails** (error like "Unknown Channel", "Cannot send messages to this user", etc.): Discord bots cannot initiate DMs with users unless: - The user has DMs enabled for server members, AND - The bot shares a server with the user When DM fails, respond to the student with alternatives: ``` I tried to message Prof. <PROFESSOR_NAME> directly, but Discord's bot limitations prevented me from sending a DM. **Please contact Prof. <PROFESSOR_NAME> directly:** • **Email:** <PROFESSOR_EMAIL> • **Discord DM:** @<PROFESSOR_DISCORD> I've noted your request — feel free to mention it was originally sent through the Virtual TA. ``` ### Important - **Never claim you forwarded the message if the DM failed.** Always be honest about the outcome. - **Don't retry endlessly** — if it fails once, provide alternatives immediately. - Log the interaction with `status=forward_failed` if the DM didn't go through. --- ## Response Style - Concise but thorough: 2–5 paragraphs for conceptual questions, shorter for factual lookups. - Discord-friendly: **bold** key terms, `code blocks` for formulas/code, bullet points for lists. - No markdown tables (Discord renders them poorly). - No LaTeX — use inline code or code blocks for math. --- ## Configuration (`course-ta.json`) ```json { "allowed_channels": ["channel:<COURSE_CHANNEL_ID>"], "course_name": "<COURSE_NAME>", "professor_name": "<PROFESSOR_NAME>", "semester": "<SEMESTER>", "log_channel": "channel:<optional discord channel id for audit log>" } ``` | Field | Purpose | |---|---| | `allowed_channels` | Only respond in these channels. Silently ignore all others. | | `log_channel` | Optional: Discord channel ID where Q&A summaries are posted for professor review. Leave empty or omit to log to file only. | | `course_name` | Used in responses and log entries. | | `professor_name` | Shown when redirecting grade/policy questions. | --- ## Rate Limits - **Standard students:** 20/hour, 60/day - **Privileged students** (<TA_NAME_1> `<TA_USER_ID_1>`, <TA_NAME_2> `<TA_USER_ID_2>`): 60/hour, 200/day - **<PROFESSOR_NAME>** (`<ADMIN_USER_ID>`): no rate limiting applied (admin) State file: `<skill>/config/ta-rate-limit.json` (auto-created, cleaned every 24h per user) --- ## Audit Log All interactions are logged to `<skill>/data/logs/YYYY-MM-DD.jsonl` (one JSON object per line). Each entry contains: - timestamp, userId, channelId, threadId - question (first 300 chars) - answer summary (first 500 chars) - status: `ok` | `rate_limited` | `out_of_scope` | `no_material` | `admin_edit` | `canvas_write` | `forward_failed` Review logs: `ls ~/.openclaw/skills/course-ta/data/logs/` --- ## Memory / RAG Course materials are indexed from `<skill>/data/memory/`. Files come from two sources: ### Source 1: GDrive slides (existing) Auto-generated by `courses/extract_slides.py` from PPTX symlinks under `courses/`. To refresh after GDrive updates: ```bash python3 ~/.openclaw/skills/course-ta/lib/extract_slides.py \ ~/.openclaw/skills/course-ta/data/courses ~/.openclaw/skills/course-ta/data/memory openclaw memory index --force ``` ### Source 2: Canvas LMS (new) Auto-generated by `courses/canvas_sync.py` from Canvas API. Covers pages, assignments, announcements, discussions, quizzes, modules, syllabus, and file indexes. Memory file naming: `<slug>__canvas-<type>__<name>.md` To sync Canvas content: ```bash # Full sync (first time or reset) python3 ~/.openclaw/skills/course-ta/lib/canvas_sync.py full <canvas_id> # Incremental sync (periodic updates — only writes changed files) python3 ~/.openclaw/skills/course-ta/lib/canvas_sync.py incremental <canvas_id> # Sync all active courses python3 ~/.openclaw/skills/course-ta/lib/canvas_sync.py all --incremental # Always reindex after sync openclaw memory index --force ``` ### Canvas content types in memory | Content type | Memory file pattern | Notes | |---|---|---| | Pages | `<slug>__canvas-page__*.md` | Lecture guides, activities, readings | | Assignments | `<slug>__canvas-assignment__*.md` | Due dates, points, descriptions | | Announcements | `<slug>__canvas-announcement__*.md` | News, updates from <PROFESSOR_NAME> | | Discussions | `<slug>__canvas-discussion__*.md` | Topic body + top replies | | Quizzes | `<slug>__canvas-quiz__*.md` | Metadata only (no questions) | | Modules | `<slug>__canvas-modules.md` | Course table of contents | | Syllabus | `<slug>__canvas-syllabus.md` | Full syllabus body | | Files index | `<slug>__canvas-files.md` | List of all course files | **Enrollments** are saved as `courses/<slug>/canvas/enrollments/roster.json` but are NOT indexed in memory (student privacy). --- ## Multi-Course Management The system supports multiple courses via `<skill>/config/canvas-config.json`: ```bash # List all Canvas courses python3 ~/.openclaw/skills/course-ta/lib/canvas_courses.py list # Activate a new course for sync python3 ~/.openclaw/skills/course-ta/lib/canvas_courses.py activate <canvas_id> # Map Canvas course to existing local directory python3 ~/.openclaw/skills/course-ta/lib/canvas_courses.py map <canvas_id> <slug> # Check status of all active courses python3 ~/.openclaw/skills/course-ta/lib/canvas_courses.py status # Deactivate (keeps data) python3 ~/.openclaw/skills/course-ta/lib/canvas_courses.py deactivate <canvas_id> ``` Each active course gets its own partition under `courses/<slug>/canvas/` with independent sync state, and its own set of memory files prefixed with `<slug>__canvas-*`. --- ## First-Time Setup 1. Run `scripts/setup_workspace.sh` 2. Symlink PPTX files from GDrive into `courses/<course>/module<N>/slides/` 3. Run `courses/extract_slides.py` to generate memory indexes from slides 4. Set up Canvas integration: a. Save Canvas API token to `<skill>/data/credentials/canvas.json` b. Run `courses/canvas_courses.py list` to see available courses c. Run `courses/canvas_courses.py map <id> <slug>` for existing courses d. Run `courses/canvas_courses.py activate <id>` for new courses e. Run `courses/canvas_sync.py full <id>` for initial content sync 5. Run `openclaw memory index --force` 6. Edit `course-ta.json` with channel IDs and optional log channel See `SOURCES.md` for the full maintenance runbook.

path: lib/canvas_api.py
content: #!/usr/bin/env python3
"""Canvas LMS API client with auth, pagination, rate limiting, and write gating.

Usage as library:
    from canvas_api import CanvasClient, load_canvas_credentials
    base_url, [REDACTED_SECRET])
    client = CanvasClient(base_url, token)
    courses = list(client.get_paginated("/api/v1/courses"))

Usage as CLI:
    python3 canvas_api.py courses [--active]
    python3 canvas_api.py query <course_id> pages [--search <term>]
    python3 canvas_api.py query <course_id> assignments [--upcoming] [--search <term>]
    python3 canvas_api.py query <course_id> announcements
    python3 canvas_api.py query <course_id> files
    python3 canvas_api.py query <course_id> modules
    python3 canvas_api.py query <course_id> discussions
    python3 canvas_api.py query <course_id> quizzes
    python3 canvas_api.py query <course_id> enrollments
    python3 canvas_api.py query <course_id> syllabus
    python3 canvas_api.py validate
"""

import json
import re
import sys
import time
from pathlib import Path
from typing import Generator, Optional, Union
from urllib.parse import urljoin, urlparse, parse_qs

import requests


# ---------------------------------------------------------------------------
# Defaults
# ---------------------------------------------------------------------------

from paths import CANVAS_CREDENTIALS

DEFAULT_CREDENTIALS_PATH = CANVAS_CREDENTIALS
RATE_LIMIT_BUFFER = 50          # pause when remaining drops below this
BACKOFF_BASE_SECONDS = 10       # first retry wait on 403
BACKOFF_MAX_RETRIES = 3
DEFAULT_PER_PAGE = 100
RETRYABLE_STATUS_CODES = {500, 502, 503, 504}


# ---------------------------------------------------------------------------
# Exceptions
# ---------------------------------------------------------------------------

class CanvasAPIError(Exception):
    """Raised on non-recoverable Canvas API errors."""

    def __init__(self, status_code: int, message: str, endpoint: str = ""):
        self.status_code = status_code
        self.endpoint = endpoint
        super().__init__(f"Canvas API {status_code} on {endpoint}: {message}")


class CanvasPermissionError(Exception):
    """Raised when a write operation is denied."""
    pass


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

def load_canvas_credentials(cred_path: Optional[Path] = None) -> tuple:
    """Load (base_url, access_token) from canvas.json.

    Raises FileNotFoundError if file missing, KeyError if malformed.
    """
    path = cred_path or DEFAULT_CREDENTIALS_PATH
    if not path.exists():
        raise FileNotFoundError(f"Canvas credentials not found at {path}")
    data = json.loads(path.read_text())
    base_url = data["canvas_base_url"].rstrip("/")
    token = data["access_token"]
    if not base_url or not token:
        raise ValueError(f"Canvas credentials incomplete in {path}")
    return base_url, token


def validate_credentials(base_url: str, access_token: str) -> bool:
    """Quick ping to verify credentials work. Returns True on success."""
    try:
        client = CanvasClient(base_url, access_token, write_mode="deny")
        result = client.get("/api/v1/users/self")
        return isinstance(result, dict) and "id" in result
    except (CanvasAPIError, requests.RequestException):
        return False


def slugify_course_name(course_code: str) -> str:
    """Convert 'BU.520.710.42.SP26' -> 'bu-520-710-42-sp26'."""
    slug = course_code.lower()
    slug = re.sub(r"[^a-z0-9]+", "-", slug)
    return slug.strip("-")


def _parse_link_header(header: str) -> dict:
    """Parse Link header into {rel: url} dict."""
    links = {}
    for part in header.split(","):
        match = re.match(r'\s*<([^>]+)>\s*;\s*rel="([^"]+)"', part.strip())
        if match:
            links[match.group(2)] = match.group(1)
    return links


# ---------------------------------------------------------------------------
# Canvas Client
# ---------------------------------------------------------------------------

class CanvasClient:
    """Thin wrapper around Canvas REST API with auth, pagination,
    rate limiting, retry logic, and write permission gating."""

    def __init__(self, base_url: str, access_token: str,
                 write_mode: str = "confirm",
                 rate_limit_buffer: int = RATE_LIMIT_BUFFER):
        """
        Args:
            base_url: Canvas instance URL (e.g. https://your-canvas-instance.instructure.com)
            access_token: Bearer token
            write_mode: "confirm" (prompt user), "allow" (auto), "deny" (block)
            rate_limit_buffer: pause when X-Rate-Limit-Remaining < this
        """
        self.base_url = base_url.rstrip("/")
        self.access_token = access_token
        self.write_mode = write_mode
        self.rate_limit_buffer = rate_limit_buffer

        self.session = requests.Session()
        self.session.headers.update({
            "Authorization": f"Bearer {access_token}",
            "Accept": "application/json",
        })

    # -- internal -----------------------------------------------------------

    def _check_rate_limit(self, response: requests.Response):
        """Sleep if rate limit is getting low."""
        remaining = response.headers.get("X-Rate-Limit-Remaining")
        if remaining is not None:
            try:
                if float(remaining) < self.rate_limit_buffer:
                    print(f"[canvas_api] Rate limit low ({remaining}), sleeping 5s...",
                          file=sys.stderr)
                    time.sleep(5)
            except ValueError:
                pass

    def _request(self, method: str, url: str, params: dict = None,
                 data: dict = None, json_body: dict = None,
                 stream: bool = False) -> requests.Response:
        """Core HTTP method with rate limiting, retries on transient errors,
        and exponential backoff on rate limits."""
        if url.startswith("/"):
            url = self.base_url + url

        last_error = None

        for attempt in range(BACKOFF_MAX_RETRIES + 1):
            # Network-level errors (timeout, connection reset)
            try:
                resp = self.session.request(
                    method, url, params=params, data=data, json=json_body,
                    stream=stream, timeout=60,
                )
            except (requests.Timeout, requests.ConnectionError) as e:
                last_error = e
                if attempt < BACKOFF_MAX_RETRIES:
                    wait = BACKOFF_BASE_SECONDS * (2 ** attempt)
                    print(f"[canvas_api] Network error ({type(e).__name__}), "
                          f"retrying in {wait}s (attempt {attempt + 1})...",
                          file=sys.stderr)
                    time.sleep(wait)
                    continue
                raise CanvasAPIError(
                    0, f"Network error after {BACKOFF_MAX_RETRIES} retries: {e}",
                    endpoint=url)

            # Rate limit hit — retry with backoff
            if resp.status_code == 403:
                body = resp.text.lower()
                if "rate limit" in body or "throttled" in body:
                    if attempt < BACKOFF_MAX_RETRIES:
                        wait = BACKOFF_BASE_SECONDS * (3 ** attempt)
                        print(f"[canvas_api] Rate limited, waiting {wait}s "
                              f"(attempt {attempt + 1}/{BACKOFF_MAX_RETRIES})...",
                              file=sys.stderr)
                        time.sleep(wait)
                        continue

            # Server errors — retry with backoff
            if resp.status_code in RETRYABLE_STATUS_CODES:
                if attempt < BACKOFF_MAX_RETRIES:
                    wait = BACKOFF_BASE_SECONDS * (2 ** attempt)
                    print(f"[canvas_api] Server error {resp.status_code}, "
                          f"retrying in {wait}s (attempt {attempt + 1})...",
                          file=sys.stderr)
                    time.sleep(wait)
                    continue

            # Check remaining budget
            self._check_rate_limit(resp)

            # Raise on non-success
            if resp.status_code >= 400:
                raise CanvasAPIError(
                    resp.status_code,
                    resp.text[:500],
                    endpoint=url,
                )

            return resp

        raise CanvasAPIError(
            getattr(resp, 'status_code', 0) if 'resp' in dir() else 0,
            "Max retries exceeded",
            endpoint=url)

    def _check_write_permission(self, action_description: str) -> bool:
        """Gate write operations based on write_mode."""
        if self.write_mode == "deny":
            raise CanvasPermissionError(
                f"Write operations disabled: {action_description}")
        if self.write_mode == "confirm":
            try:
                response = input(f"\n[canvas_api] Write permission required:\n"
                                 f"  {action_description}\n"
                                 f"  Proceed? [y/N] ")
                return response.strip().lower() in ("y", "yes")
            except (EOFError, KeyboardInterrupt):
                return False
        return True  # "allow" mode

    # -- public API ---------------------------------------------------------

    def get(self, endpoint: str, params: dict = None) -> Union[dict, list]:
        """Single-page GET, returns parsed JSON."""
        resp = self._request("GET", endpoint, params=params)
        return resp.json()

    def get_paginated(self, endpoint: str, params: dict = None,
                      per_page: int = DEFAULT_PER_PAGE) -> Generator[dict, None, None]:
        """Yield individual items across all pages.

        Handles Link header pagination automatically. Yields dicts from
        list responses; yields single dict if response is an object.
        """
        params = dict(params or {})
        params["per_page"] = per_page

        url = endpoint if endpoint.startswith("http") else self.base_url + endpoint

        while url:
            resp = self._request("GET", url, params=params)
            data = resp.json()

            if isinstance(data, list):
                yield from data
            elif isinstance(data, dict):
                # Some endpoints return a single object
                yield data
            else:
                print(f"[canvas_api] Unexpected JSON type from {url}: "
                      f"{type(data).__name__}", file=sys.stderr)

            # Follow pagination
            link_header = resp.headers.get("Link", "")
            links = _parse_link_header(link_header)
            url = links.get("next")
            # After first request, params are encoded in the Link URL
            params = None

    def post(self, endpoint: str, data: dict = None,
             json_body: dict = None, action_description: str = "") -> dict:
        """Write operation — gated by write_mode."""
        desc = action_description or f"POST {endpoint}"
        if not self._check_write_permission(desc):
            raise CanvasPermissionError(f"User denied: {desc}")
        resp = self._request("POST", endpoint, data=data, json_body=json_body)
        return resp.json()

    def put(self, endpoint: str, data: dict = None,
            json_body: dict = None, action_description: str = "") -> dict:
        """Write operation — gated by write_mode."""
        desc = action_description or f"PUT {endpoint}"
        if not self._check_write_permission(desc):
            raise CanvasPermissionError(f"User denied: {desc}")
        resp = self._request("PUT", endpoint, data=data, json_body=json_body)
        return resp.json()

    def delete(self, endpoint: str, action_description: str = "") -> dict:
        """Write operation — gated by write_mode."""
        desc = action_description or f"DELETE {endpoint}"
        if not self._check_write_permission(desc):
            raise CanvasPermissionError(f"User denied: {desc}")
        resp = self._request("DELETE", endpoint)
        return resp.json()

    def download_file(self, file_obj: dict, dest_path: Path) -> Path:
        """Download a Canvas file to local path.

        Handles Canvas's redirect chain (API URL -> file URL -> S3).
        Retries on transient network errors.

        Args:
            file_obj: File object from Canvas API (must have 'url' key)
            dest_path: Local destination path
        Returns:
            The dest_path on success
        """
        file_url = file_obj.get("url")
        if not file_url:
            raise CanvasAPIError(0, "File object has no 'url' field", "download")

        dest_path.parent.mkdir(parents=True, exist_ok=True)

        for attempt in range(BACKOFF_MAX_RETRIES + 1):
            try:
                # Canvas file URLs redirect to the actual file (often S3)
                # Use a fresh request (no auth header needed for the redirect target)
                resp = requests.get(file_url, stream=True, timeout=120)
                resp.raise_for_status()

                with open(dest_path, "wb") as f:
                    for chunk in resp.iter_content(chunk_size=8192):
                        f.write(chunk)

                return dest_path
            except (requests.Timeout, requests.ConnectionError) as e:
                if attempt < BACKOFF_MAX_RETRIES:
                    wait = BACKOFF_BASE_SECONDS * (2 ** attempt)
                    print(f"[canvas_api] Download failed ({type(e).__name__}), "
                          f"retrying in {wait}s...", file=sys.stderr)
                    time.sleep(wait)
                    continue
                raise CanvasAPIError(
                    0, f"Download failed after retries: {e}",
                    endpoint=file_url)

    # -- convenience methods ------------------------------------------------

    def get_courses(self, enrollment_state: str = "active",
                    include: list = None) -> list:
        """List all courses for the current user."""
        params = {"enrollment_state": enrollment_state}
        if include:
            for i, val in enumerate(include):
                params[f"include[{i}]"] = val
        return list(self.get_paginated("/api/v1/courses", params=params))

    def get_course(self, course_id: int, include: list = None) -> dict:
        """Get a single course."""
        params = {}
        if include:
            params["include[]"] = include
        return self.get(f"/api/v1/courses/{course_id}", params=params)

    def get_modules(self, course_id: int, include_items: bool = True) -> list:
        """List modules for a course."""
        params = {}
        if include_items:
            params["include[]"] = "items"
        return list(self.get_paginated(
            f"/api/v1/courses/{course_id}/modules", params=params))

    def get_pages(self, course_id: int) -> list:
        """List all pages for a course."""
        return list(self.get_paginated(
            f"/api/v1/courses/{course_id}/pages"))

    def get_page(self, course_id: int, page_url: str) -> dict:
        """Get a single page with body."""
        return self.get(f"/api/v1/courses/{course_id}/pages/{page_url}")

    def get_assignments(self, course_id: int) -> list:
        """List all assignments for a course."""
        return list(self.get_paginated(
            f"/api/v1/courses/{course_id}/assignments"))

    def get_announcements(self, course_id: int) -> list:
        """List announcements for a course."""
        return list(self.get_paginated(
            "/api/v1/announcements",
            params={"context_codes[]": f"course_{course_id}"}))

    def get_discussion_topics(self, course_id: int) -> list:
        """List discussion topics for a course."""
        return list(self.get_paginated(
            f"/api/v1/courses/{course_id}/discussion_topics"))

    def get_discussion_entries(self, course_id: int, topic_id: int) -> list:
        """Get entries for a discussion topic."""
        return list(self.get_paginated(
            f"/api/v1/courses/{course_id}/discussion_topics/{topic_id}/entries"))

    def get_files(self, course_id: int) -> list:
        """List all files for a course."""
        return list(self.get_paginated(
            f"/api/v1/courses/{course_id}/files"))

    def get_quizzes(self, course_id: int) -> list:
        """List quizzes for a course."""
        return list(self.get_paginated(
            f"/api/v1/courses/{course_id}/quizzes"))

    def get_enrollments(self, course_id: int) -> list:
        """List enrollments for a course."""
        return list(self.get_paginated(
            f"/api/v1/courses/{course_id}/enrollments"))

    def get_syllabus(self, course_id: int) -> dict:
        """Get course with syllabus body."""
        return self.get(
            f"/api/v1/courses/{course_id}",
            params={"include[]": "syllabus_body"})

    def get_submissions(self, course_id: int, assignment_id: int) -> list:
        """List submissions for an assignment."""
        return list(self.get_paginated(
            f"/api/v1/courses/{course_id}/assignments/{assignment_id}/submissions"))

    def get_calendar_events(self, course_id: int) -> list:
        """List calendar events for a course."""
        return list(self.get_paginated(
            "/api/v1/calendar_events",
            params={"context_codes[]": f"course_{course_id}", "type": "event"}))


# ---------------------------------------------------------------------------
# CLI interface
# ---------------------------------------------------------------------------

def _cli_courses(client: CanvasClient, args: list):
    """List all courses."""
    active_only = "--active" in args
    courses = client.get_courses()
    for c in courses:
        if active_only and c.get("workflow_state") != "available":
            continue
        roles = [e["type"] for e in c.get("enrollments", [])]
        state = c.get("workflow_state", "?")
        print(f"  {c['id']:>8}  {c.get('course_code', c['name'])[:60]:<62}"
              f"  {','.join(roles):<20}  {state}")


def _cli_query(client: CanvasClient, args: list):
    """Query a specific course."""
    if len(args) < 2:
        print("Usage: canvas_api.py query <course_id> <resource> "
              "[--search <term>] [--upcoming]", file=sys.stderr)
        sys.exit(1)

    try:
        course_id = int(args[0])
    except ValueError:
        print(f"Invalid course_id: {args[0]}", file=sys.stderr)
        sys.exit(1)
    resource = args[1]
    search_term = None
    if "--search" in args:
        idx = args.index("--search")
        if idx + 1 < len(args):
            search_term = args[idx + 1].lower()

    try:
        _cli_query_resource(client, course_id, resource, search_term,
                            "--upcoming" in args)
    except CanvasAPIError as e:
        print(f"  Error: {e}", file=sys.stderr)
        sys.exit(1)


def _cli_query_resource(client: CanvasClient, course_id: int,
                        resource: str, search_term: str = None,
                        upcoming: bool = False):
    """Execute a resource query."""
    if resource == "pages":
        pages = client.get_pages(course_id)
        for p in pages:
            title = p.get("title", "?")
            if search_term and search_term not in title.lower():
                continue
            print(f"  {p.get('url', '?'):<50}  {title}")

    elif resource == "assignments":
        assignments = client.get_assignments(course_id)
        from datetime import datetime, timezone
        now = datetime.now(timezone.utc)
        for a in assignments:
            due = a.get("due_at")
            if upcoming and due:
                due_dt = datetime.fromisoformat(due.replace("Z", "+00:00"))
                if due_dt < now:
                    continue
            if search_term and search_term not in a["name"].lower():
                continue
            pts = a.get("points_possible", "?")
            print(f"  {a['id']:>8}  {a['name'][:50]:<52}  "
                  f"due:{due or 'none':<26}  pts:{pts}")

    elif resource == "announcements":
        anns = client.get_announcements(course_id)
        for a in anns:
            posted = a.get('posted_at') or 'N/A'
            print(f"  {posted:<26}  {a['title'][:70]}")

    elif resource == "files":
        files = client.get_files(course_id)
        for f in files:
            size_mb = f.get("size", 0) / (1024 * 1024)
            print(f"  {f['id']:>8}  {f['display_name'][:50]:<52}  {size_mb:.1f}MB")

    elif resource == "modules":
        modules = client.get_modules(course_id)
        for m in modules:
            items = m.get("items", [])
            print(f"  Module {m.get('position', '?')}: {m['name']} "
                  f"({len(items)} items)")
            for item in items:
                print(f"    - [{item.get('type', '?')}] "
                      f"{item.get('title', '?')}")

    elif resource == "discussions":
        topics = client.get_discussion_topics(course_id)
        for d in topics:
            print(f"  {d['id']:>8}  {d['title'][:60]:<62}  "
                  f"posts:{d.get('discussion_subentry_count', '?')}")

    elif resource == "quizzes":
        quizzes = client.get_quizzes(course_id)
        for q in quizzes:
            print(f"  {q['id']:>8}  {q['title'][:50]:<52}  "
                  f"pts:{q.get('points_possible', '?')}  "
                  f"due:{q.get('due_at', 'none')}")

    elif resource == "enrollments":
        enrollments = client.get_enrollments(course_id)
        from collections import Counter
        counts = Counter(e["type"] for e in enrollments)
        print(f"  Total: {len(enrollments)}")
        for k, v in sorted(counts.items()):
            print(f"    {k}: {v}")

    elif resource == "syllabus":
        course = client.get_syllabus(course_id)
        body = course.get("syllabus_body", "")
        if body:
            print(f"  Syllabus body ({len(body)} chars):")
            print(f"  {body[:500]}...")
        else:
            print("  No syllabus body found.")

    else:
        print(f"Unknown resource: {resource}", file=sys.stderr)
        print("Available: pages, assignments, announcements, files, modules, "
              "discussions, quizzes, enrollments, syllabus", file=sys.stderr)
        sys.exit(1)


def _cli_validate():
    """Validate Canvas credentials."""
    base_url, [REDACTED_SECRET])
    print(f"  Base URL: {base_url}")
    print(f"  [REDACTED_SECRET]")
    print(f"  Validating...", end=" ", flush=True)
    if validate_credentials(base_url, token):
        print("OK")
    else:
        print("FAILED")
        sys.exit(1)


def main():
    if len(sys.argv) < 2:
        print(__doc__)
        sys.exit(1)

    command = sys.argv[1]

    if command == "validate":
        _cli_validate()
        return

    base_url, [REDACTED_SECRET])
    client = CanvasClient(base_url, token, write_mode="deny")

    if command == "courses":
        _cli_courses(client, sys.argv[2:])
    elif command == "query":
        _cli_query(client, sys.argv[2:])
    else:
        print(f"Unknown command: {command}", file=sys.stderr)
        print("Available: courses, query, validate", file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()


path: lib/canvas_content.py
content: #!/usr/bin/env python3
"""Canvas HTML-to-markdown converter and content formatters for RAG indexing.

Converts Canvas API responses into clean, RAG-optimized markdown files
following the existing memory/ naming conventions.

Usage as library:
    from canvas_content import html_to_markdown, format_page_markdown
    md = html_to_markdown(page["body"], base_url="https://your-canvas-instance.instructure.com")
    full_md = format_page_markdown(page, "ai-essentials-spring2026")
"""

import re
from datetime import datetime
from typing import Optional
from bs4 import BeautifulSoup, NavigableString, Tag


# ---------------------------------------------------------------------------
# HTML to Markdown conversion
# ---------------------------------------------------------------------------

def html_to_markdown(html: str, base_url: str = None) -> str:
    """Convert Canvas HTML to clean markdown.

    Handles standard HTML elements and strips Canvas-specific cruft
    (CSS classes, iframe embeds, LTI tool links, media_comment elements).
    """
    if not html or not html.strip():
        return ""

    soup = BeautifulSoup(html, "html.parser")

    # Remove Canvas-specific elements
    for tag in soup.find_all(["iframe", "script", "style"]):
        tag.decompose()
    for tag in soup.find_all(class_=re.compile(r"media_comment|lti-|instructure_")):
        tag.decompose()
    for tag in soup.find_all("a", class_="instructure_file_link"):
        # Keep the link text, just convert normally
        pass

    result = _convert_element(soup, base_url)

    # Clean up whitespace
    result = re.sub(r"\n{3,}", "\n\n", result)
    result = re.sub(r"[ \t]+\n", "\n", result)
    result = result.strip()

    return result


def _convert_element(element, base_url: str = None) -> str:
    """Recursively convert an HTML element to markdown."""
    if isinstance(element, NavigableString):
        text = str(element)
        # Collapse internal whitespace but preserve newlines
        text = re.sub(r"[ \t]+", " ", text)
        return text

    if not isinstance(element, Tag):
        return ""

    tag = element.name
    children_md = "".join(_convert_element(c, base_url) for c in element.children)
    children_md_stripped = children_md.strip()

    # Skip empty elements
    if not children_md_stripped and tag not in ("br", "hr", "img"):
        return ""

    # Headings
    if tag in ("h1", "h2", "h3", "h4", "h5", "h6"):
        level = int(tag[1])
        # Bump all headings down by one since we use h1 for the file title
        level = min(level + 1, 6)
        return f"\n\n{'#' * level} {children_md_stripped}\n\n"

    # Paragraphs
    if tag == "p":
        return f"\n\n{children_md_stripped}\n\n"

    # Line breaks
    if tag == "br":
        return "\n"

    # Horizontal rules
    if tag == "hr":
        return "\n\n---\n\n"

    # Bold
    if tag in ("strong", "b"):
        return f"**{children_md_stripped}**"

    # Italic
    if tag in ("em", "i"):
        return f"*{children_md_stripped}*"

    # Code (inline)
    if tag == "code":
        return f"`{children_md_stripped}`"

    # Code blocks
    if tag == "pre":
        code = element.find("code")
        if code:
            return f"\n\n```\n{code.get_text()}\n```\n\n"
        return f"\n\n```\n{element.get_text()}\n```\n\n"

    # Blockquotes
    if tag == "blockquote":
        lines = children_md_stripped.split("\n")
        quoted = "\n".join(f"> {line}" for line in lines)
        return f"\n\n{quoted}\n\n"

    # Links
    if tag == "a":
        href = element.get("href", "")
        if base_url and href.startswith("/"):
            href = base_url.rstrip("/") + href
        if children_md_stripped:
            return f"[{children_md_stripped}]({href})"
        return href

    # Images
    if tag == "img":
        src = element.get("src", "")
        alt = element.get("alt", "image")
        if base_url and src.startswith("/"):
            src = base_url.rstrip("/") + src
        return f"![{alt}]({src})"

    # Unordered lists
    if tag == "ul":
        items = []
        for li in element.find_all("li", recursive=False):
            li_md = _convert_element(li, base_url).strip()
            # Handle nested lists — indent continuation lines
            lines = li_md.split("\n")
            items.append(f"- {lines[0]}")
            for line in lines[1:]:
                if line.strip():
                    items.append(f"  {line}")
        return "\n\n" + "\n".join(items) + "\n\n"

    # Ordered lists
    if tag == "ol":
        items = []
        start = int(element.get("start", 1))
        for i, li in enumerate(element.find_all("li", recursive=False)):
            li_md = _convert_element(li, base_url).strip()
            lines = li_md.split("\n")
            items.append(f"{start + i}. {lines[0]}")
            for line in lines[1:]:
                if line.strip():
                    items.append(f"   {line}")
        return "\n\n" + "\n".join(items) + "\n\n"

    # List items (handled by parent ul/ol, but in case they appear standalone)
    if tag == "li":
        return children_md_stripped

    # Tables
    if tag == "table":
        return _convert_table(element, base_url)

    # Table sub-elements (handled by _convert_table)
    if tag in ("thead", "tbody", "tfoot", "tr", "th", "td", "caption"):
        return children_md

    # Div, span, section — just pass through children
    if tag in ("div", "span", "section", "article", "header", "footer",
               "main", "nav", "figure", "figcaption", "details", "summary"):
        return children_md

    # Default: pass through children
    return children_md


def _convert_table(table: Tag, base_url: str = None) -> str:
    """Convert an HTML table to markdown table."""
    rows = []
    for tr in table.find_all("tr"):
        cells = []
        for cell in tr.find_all(["th", "td"]):
            cell_md = _convert_element(cell, base_url).strip()
            # Remove newlines within cells
            cell_md = re.sub(r"\n+", " ", cell_md)
            cells.append(cell_md)
        if cells:
            rows.append(cells)

    if not rows:
        return ""

    # Normalize column count
    max_cols = max(len(r) for r in rows)
    for r in rows:
        while len(r) < max_cols:
            r.append("")

    # Build markdown table
    lines = []
    # Header row
    lines.append("| " + " | ".join(rows[0]) + " |")
    lines.append("| " + " | ".join("---" for _ in rows[0]) + " |")
    # Data rows
    for row in rows[1:]:
        lines.append("| " + " | ".join(row) + " |")

    return "\n\n" + "\n".join(lines) + "\n\n"


# ---------------------------------------------------------------------------
# Content formatters — produce RAG-optimized markdown with frontmatter
# ---------------------------------------------------------------------------

def _slugify(text: str) -> str:
    """Convert text to a filename-safe slug."""
    slug = text.lower()
    slug = re.sub(r"[^a-z0-9]+", "-", slug)
    return slug.strip("-")[:80]


def _format_date(iso_str: Optional[str]) -> str:
    """Format ISO date string to human-readable."""
    if not iso_str:
        return "N/A"
    try:
        dt = datetime.fromisoformat(iso_str.replace("Z", "+00:00"))
        return dt.strftime("%Y-%m-%d %H:%M UTC")
    except (ValueError, TypeError):
        return iso_str


def format_page_markdown(page_data: dict, course_slug: str,
                         base_url: str = None) -> str:
    """Format a Canvas page API response into a RAG-optimized markdown file."""
    title = page_data.get("title", "Untitled Page")
    updated = _format_date(page_data.get("updated_at"))
    body_html = page_data.get("body", "")
    body_md = html_to_markdown(body_html, base_url)

    return f"""# {title}
> **Course:** {course_slug}
> **Type:** Canvas Page
> **Last Updated:** {updated}
> **URL:** {page_data.get('html_url', 'N/A')}

{body_md}
"""


def format_assignment_markdown(assignment: dict, course_slug: str,
                               base_url: str = None) -> str:
    """Format assignment for RAG indexing."""
    name = assignment.get("name", "Untitled Assignment")
    due = _format_date(assignment.get("due_at"))
    pts = assignment.get("points_possible", "N/A")
    sub_types = ", ".join(assignment.get("submission_types", []))
    desc_html = assignment.get("description", "") or ""
    desc_md = html_to_markdown(desc_html, base_url)

    lock_at = _format_date(assignment.get("lock_at"))
    unlock_at = _format_date(assignment.get("unlock_at"))

    meta = f"""# {name}
> **Course:** {course_slug}
> **Type:** Canvas Assignment
> **Due Date:** {due}
> **Points:** {pts}
> **Submission Types:** {sub_types}
> **Available:** {unlock_at} to {lock_at}
> **URL:** {assignment.get('html_url', 'N/A')}
"""

    if assignment.get("allowed_extensions"):
        meta += f"> **Allowed Extensions:** {', '.join(assignment['allowed_extensions'])}\n"

    if assignment.get("group_category_id"):
        meta += "> **Group Assignment:** Yes\n"

    meta += f"\n{desc_md}\n"
    return meta


def format_announcement_markdown(announcement: dict, course_slug: str,
                                 base_url: str = None) -> str:
    """Format announcement for RAG indexing."""
    title = announcement.get("title", "Untitled Announcement")
    posted = _format_date(announcement.get("posted_at"))
    author = announcement.get("user_name", announcement.get("author", {}).get("display_name", "Unknown"))
    body_html = announcement.get("message", "") or ""
    body_md = html_to_markdown(body_html, base_url)

    return f"""# {title}
> **Course:** {course_slug}
> **Type:** Canvas Announcement
> **Posted:** {posted}
> **Author:** {author}
> **URL:** {announcement.get('html_url', 'N/A')}

{body_md}
"""


def format_discussion_markdown(topic: dict, replies: list,
                               course_slug: str, base_url: str = None) -> str:
    """Format discussion topic with top replies for RAG indexing."""
    title = topic.get("title", "Untitled Discussion")
    posted = _format_date(topic.get("posted_at"))
    body_html = topic.get("message", "") or ""
    body_md = html_to_markdown(body_html, base_url)
    reply_count = topic.get("discussion_subentry_count", 0)

    result = f"""# {title}
> **Course:** {course_slug}
> **Type:** Canvas Discussion
> **Posted:** {posted}
> **Replies:** {reply_count}
> **URL:** {topic.get('html_url', 'N/A')}

{body_md}
"""

    if replies:
        result += "\n## Replies\n\n"
        for reply in replies[:20]:  # Cap at 20 replies
            author = reply.get("user_name", "Anonymous")
            msg = html_to_markdown(reply.get("message", ""), base_url)
            created = _format_date(reply.get("created_at"))
            result += f"**{author}** ({created}):\n{msg}\n\n---\n\n"

    return result


def format_quiz_markdown(quiz: dict, course_slug: str) -> str:
    """Format quiz metadata for RAG (no questions for academic integrity)."""
    title = quiz.get("title", "Untitled Quiz")
    due = _format_date(quiz.get("due_at"))
    pts = quiz.get("points_possible", "N/A")
    time_limit = quiz.get("time_limit")
    attempts = quiz.get("allowed_attempts", -1)
    q_count = quiz.get("question_count", "?")
    quiz_type = quiz.get("quiz_type", "?")

    time_str = f"{time_limit} minutes" if time_limit else "Unlimited"
    attempts_str = "Unlimited" if attempts == -1 else str(attempts)

    desc_html = quiz.get("description", "") or ""
    desc_md = html_to_markdown(desc_html) if desc_html else ""

    return f"""# {title}
> **Course:** {course_slug}
> **Type:** Canvas Quiz ({quiz_type})
> **Due Date:** {due}
> **Points:** {pts}
> **Questions:** {q_count}
> **Time Limit:** {time_str}
> **Allowed Attempts:** {attempts_str}
> **URL:** {quiz.get('html_url', 'N/A')}

{desc_md}
"""


def format_module_summary_markdown(modules: list, course_slug: str) -> str:
    """Format all modules as a course table of contents."""
    result = f"""# Course Modules — Table of Contents
> **Course:** {course_slug}
> **Type:** Canvas Module Structure
> **Modules:** {len(modules)}

"""
    for m in modules:
        pos = m.get("position", "?")
        name = m.get("name", "Untitled Module")
        items = m.get("items", [])
        result += f"## Module {pos}: {name}\n\n"

        if not items:
            result += "*No items*\n\n"
            continue

        for item in items:
            item_type = item.get("type", "?")
            item_title = item.get("title", "?")
            indent = ""
            if item.get("indent", 0) > 0:
                indent = "  " * item["indent"]

            type_icon = {
                "Assignment": "[Assignment]",
                "Quiz": "[Quiz]",
                "Discussion": "[Discussion]",
                "File": "[File]",
                "Page": "[Page]",
                "ExternalUrl": "[Link]",
                "ExternalTool": "[Tool]",
                "SubHeader": "**",
            }.get(item_type, f"[{item_type}]")

            if item_type == "SubHeader":
                result += f"{indent}- **{item_title}**\n"
            else:
                result += f"{indent}- {type_icon} {item_title}\n"

        result += "\n"

    return result


def format_syllabus_markdown(course_data: dict, course_slug: str,
                             base_url: str = None) -> str:
    """Format course syllabus body."""
    name = course_data.get("name", "Course")
    body_html = course_data.get("syllabus_body", "") or ""
    body_md = html_to_markdown(body_html, base_url)

    return f"""# Syllabus — {name}
> **Course:** {course_slug}
> **Type:** Canvas Syllabus

{body_md}
"""


def format_file_index_markdown(files: list, course_slug: str) -> str:
    """Format a file index for RAG (lists all files with metadata)."""
    result = f"""# Course Files Index
> **Course:** {course_slug}
> **Type:** Canvas File Index
> **Total Files:** {len(files)}

"""
    # Group by folder
    by_folder = {}
    for f in files:
        folder = f.get("folder_id", "root")
        by_folder.setdefault(folder, []).append(f)

    for folder_id, folder_files in by_folder.items():
        for f in sorted(folder_files, key=lambda x: x.get("display_name", "")):
            name = f.get("display_name", "?")
            size_mb = f.get("size", 0) / (1024 * 1024)
            updated = _format_date(f.get("updated_at"))
            result += f"- **{name}** ({size_mb:.1f} MB) — updated {updated}\n"

    return result


# ---------------------------------------------------------------------------
# Utility: memory file naming
# ---------------------------------------------------------------------------

def memory_filename(course_slug: str, content_type: str, name: str) -> str:
    """Generate a memory-compatible filename.

    Examples:
        memory_filename("ai-essentials-spring2026", "canvas-page", "homepage")
        -> "ai-essentials-spring2026__canvas-page__homepage.md"
    """
    safe_name = _slugify(name)
    return f"{course_slug}__{content_type}__{safe_name}.md"


path: lib/canvas_courses.py
content: #!/usr/bin/env python3
"""Canvas course discovery, activation, and multi-course management.

Usage:
    python3 canvas_courses.py list [--active-only]
    python3 canvas_courses.py activate <course_id> [--slug <slug>]
    python3 canvas_courses.py deactivate <course_id>
    python3 canvas_courses.py map <course_id> <existing_slug>
    python3 canvas_courses.py status
"""

import json
import sys
from datetime import datetime, timezone
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))

from canvas_api import CanvasClient, load_canvas_credentials, slugify_course_name
from paths import CANVAS_CONFIG, COURSES_DIR, COURSE_CONFIGS_DIR


# ---------------------------------------------------------------------------
# Paths
# ---------------------------------------------------------------------------

CANVAS_CONFIG_PATH = CANVAS_CONFIG


# ---------------------------------------------------------------------------
# Config management
# ---------------------------------------------------------------------------

DEFAULT_CONFIG = {
    "version": 1,
    "default_sync_interval_hours": 6,
    "active_courses": [],
    "ignored_courses": [],
    "file_download_rules": {
        "max_file_size_mb": 50,
        "allowed_extensions": [".pdf", ".docx", ".pptx", ".xlsx", ".txt", ".csv", ".ipynb"],
        "skip_extensions": [".mp4", ".mov", ".zip", ".tar"],
    },
}


def load_config() -> dict:
    """Load canvas-config.json, creating default if missing."""
    if CANVAS_CONFIG_PATH.exists():
        return json.loads(CANVAS_CONFIG_PATH.read_text())
    return dict(DEFAULT_CONFIG)


def save_config(config: dict):
    """Save canvas-config.json."""
    CANVAS_CONFIG_PATH.write_text(json.dumps(config, indent=2) + "\n")


def find_active_course(config: dict, canvas_id: int) -> dict:
    """Find an active course entry by canvas_id, or None."""
    for c in config["active_courses"]:
        if c["canvas_id"] == canvas_id:
            return c
    return None


# ---------------------------------------------------------------------------
# Commands
# ---------------------------------------------------------------------------

def cmd_list(client: CanvasClient, active_only: bool = False):
    """List all Canvas courses for the current user."""
    config = load_config()
    active_ids = {c["canvas_id"] for c in config["active_courses"]}

    courses = client.get_courses()

    # Group by role
    by_role = {}
    for c in courses:
        if active_only and c.get("workflow_state") != "available":
            continue
        roles = [e["type"] for e in c.get("enrollments", [])]
        primary_role = roles[0] if roles else "unknown"
        by_role.setdefault(primary_role, []).append(c)

    role_order = ["teacher", "ta", "designer", "observer", "student"]
    for role in role_order:
        group = by_role.pop(role, [])
        if not group:
            continue
        print(f"\n  === {role.upper()} ({len(group)}) ===")
        for c in sorted(group, key=lambda x: x.get("name", "")):
            state = c.get("workflow_state", "?")
            code = c.get("course_code", c["name"])[:55]
            active_marker = " [ACTIVE]" if c["id"] in active_ids else ""
            print(f"    {c['id']:>8}  {code:<57}  {state:<12}{active_marker}")

    # Any remaining roles
    for role, group in by_role.items():
        print(f"\n  === {role.upper()} ({len(group)}) ===")
        for c in sorted(group, key=lambda x: x.get("name", "")):
            print(f"    {c['id']:>8}  {c.get('course_code', c['name'])[:55]:<57}  "
                  f"{c.get('workflow_state', '?')}")

    print(f"\n  Total: {len(courses)} courses"
          f" ({len(active_ids)} activated)")


def cmd_activate(client: CanvasClient, course_id: int, slug: str = None):
    """Activate a Canvas course for local sync."""
    config = load_config()

    # Check if already active
    existing = find_active_course(config, course_id)
    if existing:
        print(f"  Course {course_id} is already active as '{existing['slug']}'")
        return

    # Fetch course info
    try:
        course = client.get_course(course_id, include=["term", "total_students"])
    except Exception as e:
        print(f"  Error fetching course {course_id}: {e}", file=sys.stderr)
        sys.exit(1)

    # Generate slug
    if not slug:
        slug = slugify_course_name(course.get("course_code", str(course_id)))

    # Check if this slug is already used by a different course
    for c in config["active_courses"]:
        if c["slug"] == slug:
            print(f"  Slug '{slug}' is already used by "
                  f"canvas_id {c['canvas_id']}. Use --slug to specify "
                  f"a different slug.", file=sys.stderr)
            sys.exit(1)

    # Determine role
    roles = [e["type"] for e in course.get("enrollments", [])]
    role = roles[0] if roles else "unknown"

    # Term info
    term = course.get("term", {})
    term_name = term.get("name", "Unknown Term") if term else "Unknown Term"

    # Create directory structure
    canvas_dir = COURSES_DIR / slug / "canvas"
    for subdir in ["pages", "assignments", "announcements", "discussions",
                   "quizzes", "modules", "files", "syllabus", "enrollments"]:
        (canvas_dir / subdir).mkdir(parents=True, exist_ok=True)

    # Create course-info.json
    course_info = {
        "canvas_id": course_id,
        "slug": slug,
        "name": course.get("name", ""),
        "course_code": course.get("course_code", ""),
        "role": role,
        "term": term_name,
        "canvas_url": f"{client.base_url}/courses/{course_id}",
        "activated_at": datetime.now(timezone.utc).isoformat(),
        "total_students": course.get("total_students"),
    }
    (COURSES_DIR / slug / "course-info.json").write_text(
        json.dumps(course_info, indent=2) + "\n")

    # Initialize sync state
    sync_state = {
        "last_full_sync": None,
        "last_incremental_sync": None,
        "content_hashes": {},
    }
    (canvas_dir / "sync_state.json").write_text(
        json.dumps(sync_state, indent=2) + "\n")

    # Add to config
    entry = {
        "canvas_id": course_id,
        "slug": slug,
        "name": f"{course.get('course_code', '')} - {course.get('name', '')}",
        "role": role,
        "term": term_name,
        "discord_channels": [],
        "sync_enabled": True,
        "sync_interval_hours": config.get("default_sync_interval_hours", 6),
        "sync_content": {
            "pages": True,
            "assignments": True,
            "announcements": True,
            "discussions": True,
            "modules": True,
            "quizzes": True,
            "files": True,
            "syllabus": True,
            "enrollments": True,
        },
        "mapped_at": datetime.now(timezone.utc).isoformat(),
        "last_sync": None,
    }
    config["active_courses"].append(entry)
    save_config(config)

    # Create per-course config in course-configs/
    COURSE_CONFIGS_DIR.mkdir(parents=True, exist_ok=True)
    course_config = {
        "canvas_id": course_id,
        "slug": slug,
        "course_name": course.get("name", ""),
        "course_code": course.get("course_code", ""),
        "role": role,
        "term": term_name,
        "canvas_url": f"{client.base_url}/courses/{course_id}",
        "allowed_channels": [],
        "blocked_channels": [],
        "admin_users": [],
        "privileged_users": {},
        "editable_files": {},
    }
    (COURSE_CONFIGS_DIR / f"{slug}.json").write_text(
        json.dumps(course_config, indent=2) + "\n")

    print(f"  Activated course {course_id} as '{slug}'")
    print(f"    Name: {course.get('name', '?')}")
    print(f"    Role: {role}")
    print(f"    Term: {term_name}")
    print(f"    Dir:  {COURSES_DIR / slug}/")
    print(f"\n  Next: run 'python3 canvas_sync.py full {course_id}' to sync content")


def cmd_map(client: CanvasClient, course_id: int, existing_slug: str):
    """Map a Canvas course to an existing local directory."""
    config = load_config()

    # Check if this canvas_id is already active
    existing = find_active_course(config, course_id)
    if existing:
        print(f"  Course {course_id} is already active as '{existing['slug']}'")
        return

    # Check if this slug is already used by a different course
    for c in config["active_courses"]:
        if c["slug"] == existing_slug:
            print(f"  Slug '{existing_slug}' is already used by "
                  f"canvas_id {c['canvas_id']}", file=sys.stderr)
            sys.exit(1)

    # Verify the directory exists
    course_dir = COURSES_DIR / existing_slug
    if not course_dir.exists():
        print(f"  Error: directory {course_dir} does not exist", file=sys.stderr)
        sys.exit(1)

    # Fetch course info
    try:
        course = client.get_course(course_id, include=["term", "total_students"])
    except Exception as e:
        print(f"  Error fetching course {course_id}: {e}", file=sys.stderr)
        sys.exit(1)

    roles = [e["type"] for e in course.get("enrollments", [])]
    role = roles[0] if roles else "unknown"
    term = course.get("term", {})
    term_name = term.get("name", "Unknown Term") if term else "Unknown Term"

    # Create canvas/ subdirectory (alongside existing module dirs)
    canvas_dir = course_dir / "canvas"
    for subdir in ["pages", "assignments", "announcements", "discussions",
                   "quizzes", "modules", "files", "syllabus", "enrollments"]:
        (canvas_dir / subdir).mkdir(parents=True, exist_ok=True)

    # Initialize sync state
    sync_state = {
        "last_full_sync": None,
        "last_incremental_sync": None,
        "content_hashes": {},
    }
    (canvas_dir / "sync_state.json").write_text(
        json.dumps(sync_state, indent=2) + "\n")

    # Save course info
    (course_dir / "course-info.json").write_text(json.dumps({
        "canvas_id": course_id,
        "slug": existing_slug,
        "name": course.get("name", ""),
        "course_code": course.get("course_code", ""),
        "role": role,
        "term": term_name,
        "canvas_url": f"{client.base_url}/courses/{course_id}",
        "mapped_at": datetime.now(timezone.utc).isoformat(),
        "total_students": course.get("total_students"),
    }, indent=2) + "\n")

    # Add to config
    entry = {
        "canvas_id": course_id,
        "slug": existing_slug,
        "name": f"{course.get('course_code', '')} - {course.get('name', '')}",
        "role": role,
        "term": term_name,
        "discord_channels": [],
        "sync_enabled": True,
        "sync_interval_hours": config.get("default_sync_interval_hours", 6),
        "sync_content": {
            "pages": True,
            "assignments": True,
            "announcements": True,
            "discussions": True,
            "modules": True,
            "quizzes": True,
            "files": True,
            "syllabus": True,
            "enrollments": True,
        },
        "mapped_at": datetime.now(timezone.utc).isoformat(),
        "last_sync": None,
    }
    config["active_courses"].append(entry)
    save_config(config)

    # Create per-course config
    COURSE_CONFIGS_DIR.mkdir(parents=True, exist_ok=True)
    if not (COURSE_CONFIGS_DIR / f"{existing_slug}.json").exists():
        (COURSE_CONFIGS_DIR / f"{existing_slug}.json").write_text(json.dumps({
            "canvas_id": course_id,
            "slug": existing_slug,
            "course_name": course.get("name", ""),
            "course_code": course.get("course_code", ""),
            "role": role,
            "term": term_name,
            "canvas_url": f"{client.base_url}/courses/{course_id}",
            "allowed_channels": [],
            "blocked_channels": [],
            "admin_users": [],
            "privileged_users": {},
            "editable_files": {},
        }, indent=2) + "\n")

    print(f"  Mapped Canvas course {course_id} -> existing '{existing_slug}'")
    print(f"    Name: {course.get('name', '?')}")
    print(f"    Role: {role}")
    print(f"    Canvas dir created: {canvas_dir}/")
    print(f"    Existing content preserved (module dirs, slides, etc.)")
    print(f"\n  Next: run 'python3 canvas_sync.py full {course_id}' to sync content")


def cmd_deactivate(course_id: int):
    """Deactivate a course (remove from config, keep data)."""
    config = load_config()
    entry = find_active_course(config, course_id)

    if not entry:
        print(f"  Course {course_id} is not currently active", file=sys.stderr)
        sys.exit(1)

    slug = entry["slug"]
    config["active_courses"] = [
        c for c in config["active_courses"] if c["canvas_id"] != course_id
    ]
    save_config(config)

    print(f"  Deactivated course {course_id} ('{slug}')")
    print(f"  Data preserved in: {COURSES_DIR / slug}/")
    print(f"  To reactivate: python3 canvas_courses.py activate {course_id} --slug {slug}")


def cmd_status():
    """Show status of all activated courses."""
    config = load_config()
    active = config.get("active_courses", [])

    if not active:
        print("  No courses activated yet.")
        print("  Run 'python3 canvas_courses.py list' to see available courses.")
        return

    print(f"\n  === Activated Courses ({len(active)}) ===\n")

    for c in active:
        slug = c["slug"]
        canvas_dir = COURSES_DIR / slug / "canvas"
        sync_state_path = canvas_dir / "sync_state.json"

        last_sync = c.get("last_sync", "never")

        # Count local files
        content_counts = {}
        for content_type in ["pages", "assignments", "announcements",
                             "discussions", "quizzes", "files"]:
            type_dir = canvas_dir / content_type
            if type_dir.exists():
                content_counts[content_type] = len(list(type_dir.iterdir()))
            else:
                content_counts[content_type] = 0

        # Check sync state
        if sync_state_path.exists():
            state = json.loads(sync_state_path.read_text())
            last_full = state.get("last_full_sync", "never")
            last_inc = state.get("last_incremental_sync", "never")
        else:
            last_full = "never"
            last_inc = "never"

        print(f"  [{c['canvas_id']}] {c['name']}")
        print(f"    Slug: {slug}")
        print(f"    Role: {c.get('role', '?')}")
        print(f"    Sync enabled: {c.get('sync_enabled', False)}")
        print(f"    Last full sync: {last_full}")
        print(f"    Last incremental: {last_inc}")
        print(f"    Local content: " + ", ".join(
            f"{k}:{v}" for k, v in content_counts.items() if v > 0) or "empty")
        print()


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    if len(sys.argv) < 2:
        print(__doc__)
        sys.exit(1)

    command = sys.argv[1]

    if command == "status":
        cmd_status()
        return

    if command == "deactivate":
        if len(sys.argv) < 3:
            print("Usage: canvas_courses.py deactivate <course_id>", file=sys.stderr)
            sys.exit(1)
        cmd_deactivate(int(sys.argv[2]))
        return

    # Commands that need the API client
    base_url, [REDACTED_SECRET])
    client = CanvasClient(base_url, token, write_mode="deny")

    if command == "list":
        active_only = "--active-only" in sys.argv
        cmd_list(client, active_only)

    elif command == "activate":
        if len(sys.argv) < 3:
            print("Usage: canvas_courses.py activate <course_id> [--slug <slug>]",
                  file=sys.stderr)
            sys.exit(1)
        course_id = int(sys.argv[2])
        slug = None
        if "--slug" in sys.argv:
            idx = sys.argv.index("--slug")
            if idx + 1 < len(sys.argv):
                slug = sys.argv[idx + 1]
        cmd_activate(client, course_id, slug)

    elif command == "map":
        if len(sys.argv) < 4:
            print("Usage: canvas_courses.py map <course_id> <existing_slug>",
                  file=sys.stderr)
            sys.exit(1)
        cmd_map(client, int(sys.argv[2]), sys.argv[3])

    else:
        print(f"Unknown command: {command}", file=sys.stderr)
        print("Available: list, activate, deactivate, map, status", file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()


path: lib/canvas_dashboard.py
content: #!/usr/bin/env python3
"""Canvas dashboard — aggregated views across all active courses.

Provides deadline dashboards, submission tracking, engagement monitoring,
quiz statistics, and cross-section broadcasting.

Usage:
    python3 canvas_dashboard.py deadlines [--days 7]         # Upcoming deadlines
    python3 canvas_dashboard.py submissions <course_id> <assignment_name>  # Missing submissions
    python3 canvas_dashboard.py engagement <course_id>       # Discussion engagement
    python3 canvas_dashboard.py grades <course_id>           # Grade overview per assignment
    python3 canvas_dashboard.py roster <course_id>           # Student roster
    python3 canvas_dashboard.py broadcast <message>          # Post announcement to all sections
"""

import json
import sys
from collections import Counter
from datetime import datetime, timedelta, timezone
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))

from canvas_api import CanvasClient, CanvasAPIError, load_canvas_credentials
from paths import CANVAS_CONFIG as CANVAS_CONFIG_PATH


def _load_config() -> dict:
    if CANVAS_CONFIG_PATH.exists():
        return json.loads(CANVAS_CONFIG_PATH.read_text())
    return {"active_courses": []}


def _parse_dt(iso_str: str) -> datetime:
    if not iso_str:
        return None
    return datetime.fromisoformat(iso_str.replace("Z", "+00:00"))


# ---------------------------------------------------------------------------
# Deadline Dashboard
# ---------------------------------------------------------------------------

def cmd_deadlines(client: CanvasClient, days: int = 7):
    """Show upcoming deadlines across all active courses."""
    config = _load_config()
    now = datetime.now(timezone.utc)
    cutoff = now + timedelta(days=days)

    all_deadlines = []

    for course_entry in config.get("active_courses", []):
        canvas_id = course_entry["canvas_id"]
        slug = course_entry["slug"]
        name = course_entry.get("name", slug)

        try:
            assignments = client.get_assignments(canvas_id)
        except CanvasAPIError:
            continue

        for a in assignments:
            due = a.get("due_at")
            if not due:
                continue
            due_dt = _parse_dt(due)
            if due_dt and now <= due_dt <= cutoff:
                all_deadlines.append({
                    "course": name,
                    "slug": slug,
                    "assignment": a["name"],
                    "due": due_dt,
                    "points": a.get("points_possible", 0),
                    "type": ", ".join(a.get("submission_types", [])),
                    "canvas_id": canvas_id,
                    "assignment_id": a["id"],
                })

    # Sort by due date
    all_deadlines.sort(key=lambda x: x["due"])

    print(f"\n  === Upcoming Deadlines (next {days} days) ===\n")

    if not all_deadlines:
        print("  No deadlines in this window.")
        return

    current_date = None
    for d in all_deadlines:
        date_str = d["due"].strftime("%A, %B %d")
        if date_str != current_date:
            current_date = date_str
            print(f"\n  {date_str}")
            print(f"  {'─' * 60}")

        time_str = d["due"].strftime("%I:%M %p UTC")
        delta = d["due"] - now
        hours_left = delta.total_seconds() / 3600

        if hours_left < 24:
            urgency = " ⚠️"
        elif hours_left < 48:
            urgency = " ⏰"
        else:
            urgency = ""

        print(f"    {time_str}  {d['assignment'][:45]:<47}  "
              f"({d['points']} pts){urgency}")
        print(f"             {d['course'][:55]}")

    print(f"\n  Total: {len(all_deadlines)} upcoming deadlines")


# ---------------------------------------------------------------------------
# Submission Tracker
# ---------------------------------------------------------------------------

def cmd_submissions(client: CanvasClient, course_id: int, assignment_search: str):
    """Show who hasn't submitted a specific assignment."""
    assignments = client.get_assignments(course_id)

    # Try ID-based lookup first
    try:
        search_id = int(assignment_search)
        matches = [a for a in assignments if a["id"] == search_id]
    except ValueError:
        # Fall back to substring search
        search_lower = assignment_search.lower()
        matches = [a for a in assignments if search_lower in a["name"].lower()]

    if not matches:
        print(f"  No assignment matching '{assignment_search}' found.")
        print(f"\n  Available assignments:")
        for a in assignments:
            print(f"    {a['id']:>8}  {a['name'][:60]}")
        return

    if len(matches) > 1:
        print(f"  Multiple matches found:")
        for a in matches:
            print(f"    {a['id']}: {a['name']}")
        print(f"  Please be more specific.")
        return

    assignment = matches[0]
    print(f"\n  === Submission Tracker: {assignment['name']} ===")
    print(f"  Due: {assignment.get('due_at', 'N/A')}")
    print(f"  Points: {assignment.get('points_possible', 'N/A')}")

    # Get submissions
    submissions = client.get_submissions(course_id, assignment["id"])

    # Get enrollments for student names
    enrollments = client.get_enrollments(course_id)
    students = {}
    for e in enrollments:
        if e["type"] == "StudentEnrollment" and e.get("enrollment_state") == "active":
            user = e.get("user", {})
            students[e["user_id"]] = user.get("name", user.get("sortable_name", "?"))

    submitted = set()
    graded = set()
    not_submitted = []

    for s in submissions:
        uid = s.get("user_id")
        if s.get("workflow_state") in ("submitted", "graded", "pending_review"):
            submitted.add(uid)
        if s.get("workflow_state") == "graded":
            graded.add(uid)

    for uid, name in sorted(students.items(), key=lambda x: x[1]):
        if uid not in submitted:
            not_submitted.append(name)

    print(f"\n  Submitted: {len(submitted)}/{len(students)}")
    print(f"  Graded:    {len(graded)}/{len(students)}")

    if not_submitted:
        print(f"\n  Missing submissions ({len(not_submitted)}):")
        for name in not_submitted:
            print(f"    - {name}")
    else:
        print(f"\n  All students have submitted!")


# ---------------------------------------------------------------------------
# Engagement Monitor
# ---------------------------------------------------------------------------

def cmd_engagement(client: CanvasClient, course_id: int):
    """Show discussion engagement rates."""
    topics = client.get_discussion_topics(course_id)
    enrollments = client.get_enrollments(course_id)

    student_count = sum(1 for e in enrollments
                        if e["type"] == "StudentEnrollment"
                        and e.get("enrollment_state") == "active")

    print(f"\n  === Discussion Engagement (course {course_id}) ===")
    print(f"  Active students: {student_count}\n")

    for topic in topics:
        if topic.get("is_announcement"):
            continue

        title = topic.get("title", "?")
        reply_count = topic.get("discussion_subentry_count", 0)
        pct = (reply_count / student_count * 100) if student_count > 0 else 0

        # Engagement indicator
        if pct >= 80:
            indicator = "🟢"
        elif pct >= 50:
            indicator = "🟡"
        elif pct >= 20:
            indicator = "🟠"
        else:
            indicator = "🔴"

        print(f"  {indicator} {title[:50]:<52}  "
              f"{reply_count:>3}/{student_count} replies ({pct:.0f}%)")

    print()


# ---------------------------------------------------------------------------
# Grade Overview
# ---------------------------------------------------------------------------

def cmd_grades(client: CanvasClient, course_id: int):
    """Show grade statistics per assignment."""
    assignments = client.get_assignments(course_id)

    print(f"\n  === Grade Overview (course {course_id}) ===\n")
    print(f"  {'Assignment':<45}  {'Max':>5}  {'Mean':>6}  {'Med':>5}  "
          f"{'Min':>5}  {'Max':>5}  {'Sub':>4}")
    print(f"  {'─' * 45}  {'─' * 5}  {'─' * 6}  {'─' * 5}  "
          f"{'─' * 5}  {'─' * 5}  {'─' * 4}")

    for a in sorted(assignments, key=lambda x: x.get("due_at") or "9999"):
        pts = a.get("points_possible", 0)
        if not pts or pts == 0:
            continue

        try:
            subs = client.get_submissions(course_id, a["id"])
        except CanvasAPIError:
            continue

        scores = [s.get("score") for s in subs
                  if s.get("score") is not None and s.get("workflow_state") == "graded"]

        if not scores:
            print(f"  {a['name'][:45]:<45}  {pts:>5.0f}  {'—':>6}  {'—':>5}  "
                  f"{'—':>5}  {'—':>5}  {0:>4}")
            continue

        scores.sort()
        mean = sum(scores) / len(scores)
        median = scores[len(scores) // 2]
        low = min(scores)
        high = max(scores)

        print(f"  {a['name'][:45]:<45}  {pts:>5.0f}  {mean:>6.1f}  "
              f"{median:>5.1f}  {low:>5.1f}  {high:>5.1f}  {len(scores):>4}")

    print()


# ---------------------------------------------------------------------------
# Roster
# ---------------------------------------------------------------------------

def cmd_roster(client: CanvasClient, course_id: int):
    """Show student roster for a course."""
    enrollments = client.get_enrollments(course_id)

    by_role = {}
    for e in enrollments:
        role = e.get("type", "Unknown")
        user = e.get("user", {})
        name = user.get("name", user.get("sortable_name", "?"))
        state = e.get("enrollment_state", "?")
        by_role.setdefault(role, []).append((name, state))

    print(f"\n  === Roster (course {course_id}) ===\n")

    for role in ["TeacherEnrollment", "TaEnrollment", "StudentEnrollment",
                 "ObserverEnrollment", "DesignerEnrollment"]:
        members = by_role.pop(role, [])
        if not members:
            continue
        label = role.replace("Enrollment", "").upper() + "S"
        print(f"  {label} ({len(members)})")
        for name, state in sorted(members):
            state_indicator = "" if state == "active" else f" [{state}]"
            print(f"    - {name}{state_indicator}")
        print()

    # Any remaining roles
    for role, members in by_role.items():
        print(f"  {role} ({len(members)})")
        for name, state in sorted(members):
            print(f"    - {name}")
        print()


# ---------------------------------------------------------------------------
# Cross-Section Broadcast
# ---------------------------------------------------------------------------

def cmd_broadcast(client: CanvasClient, message: str):
    """Post an announcement to all active courses (with confirmation).

    Groups courses by base course code to identify sections of the same course.
    """
    config = _load_config()
    active = config.get("active_courses", [])

    if not active:
        print("  No active courses.")
        return

    # Show what will be posted
    print(f"\n  === Cross-Section Broadcast ===\n")
    print(f"  Message:\n    {message}\n")
    print(f"  Will post to {len(active)} course(s):")
    for c in active:
        print(f"    - [{c['canvas_id']}] {c['name']}")

    # Confirmation gate
    try:
        confirm = input(f"\n  Post announcement to all {len(active)} courses? [y/N] ")
    except (EOFError, KeyboardInterrupt):
        print("\n  Cancelled.")
        return

    if confirm.strip().lower() not in ("y", "yes"):
        print("  Cancelled.")
        return

    # Use write_mode="allow" here because we already prompted above
    write_client = CanvasClient(
        client.base_url, client.access_token, write_mode="allow")

    posted = 0
    for c in active:
        try:
            write_client.post(
                f"/api/v1/courses/{c['canvas_id']}/discussion_topics",
                json_body={
                    "title": message[:80],
                    "message": f"<p>{message}</p>",
                    "is_announcement": True,
                    "published": True,
                },
                action_description=f"Post announcement to {c['name']}",
            )
            print(f"    ✅ Posted to {c['name']}")
            posted += 1
        except Exception as e:
            print(f"    ❌ Failed for {c['name']}: {e}")

    print(f"\n  Posted to {posted}/{len(active)} courses.")


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    if len(sys.argv) < 2:
        print(__doc__)
        sys.exit(1)

    command = sys.argv[1]

    base_url, [REDACTED_SECRET])
    client = CanvasClient(base_url, token, write_mode="deny")

    if command == "deadlines":
        days = 7
        if "--days" in sys.argv:
            idx = sys.argv.index("--days")
            if idx + 1 < len(sys.argv):
                days = int(sys.argv[idx + 1])
        cmd_deadlines(client, days)

    elif command == "submissions":
        if len(sys.argv) < 4:
            print("Usage: canvas_dashboard.py submissions <course_id> <assignment_name>",
                  file=sys.stderr)
            sys.exit(1)
        cmd_submissions(client, int(sys.argv[2]), " ".join(sys.argv[3:]))

    elif command == "engagement":
        if len(sys.argv) < 3:
            print("Usage: canvas_dashboard.py engagement <course_id>",
                  file=sys.stderr)
            sys.exit(1)
        cmd_engagement(client, int(sys.argv[2]))

    elif command == "grades":
        if len(sys.argv) < 3:
            print("Usage: canvas_dashboard.py grades <course_id>",
                  file=sys.stderr)
            sys.exit(1)
        cmd_grades(client, int(sys.argv[2]))

    elif command == "roster":
        if len(sys.argv) < 3:
            print("Usage: canvas_dashboard.py roster <course_id>",
                  file=sys.stderr)
            sys.exit(1)
        cmd_roster(client, int(sys.argv[2]))

    elif command == "broadcast":
        if len(sys.argv) < 3:
            print("Usage: canvas_dashboard.py broadcast '<message>'",
                  file=sys.stderr)
            sys.exit(1)
        cmd_broadcast(client, " ".join(sys.argv[2:]))

    else:
        print(f"Unknown command: {command}", file=sys.stderr)
        print("Available: deadlines, submissions, engagement, grades, roster, broadcast",
              file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()


path: lib/canvas_sync.py
content: #!/usr/bin/env python3
"""Canvas content sync engine — full and incremental sync with change detection.

Downloads all Canvas content for activated courses, converts to markdown,
and generates memory files for RAG indexing. Uses SHA-256 hashing to detect
changes and only write files that actually differ.

Usage:
    python3 canvas_sync.py full <course_id>           # Full sync
    python3 canvas_sync.py incremental <course_id>    # Only changes since last sync
    python3 canvas_sync.py all [--incremental]        # Sync all active courses
    python3 canvas_sync.py status <course_id>         # Show sync state
"""

import hashlib
import json
import os
import sys
import traceback
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional

sys.path.insert(0, str(Path(__file__).parent))

from canvas_api import (CanvasClient, CanvasAPIError, load_canvas_credentials,
                        slugify_course_name)
from canvas_content import (html_to_markdown, format_page_markdown,
                            format_assignment_markdown, format_announcement_markdown,
                            format_discussion_markdown, format_quiz_markdown,
                            format_module_summary_markdown, format_syllabus_markdown,
                            format_file_index_markdown, memory_filename, _slugify)
from paths import (COURSES_DIR, MEMORY_DIR, CANVAS_CONFIG as CANVAS_CONFIG_PATH)

# Default sync_content — all enabled
ALL_CONTENT_TYPES = {
    "pages": True, "assignments": True, "announcements": True,
    "discussions": True, "modules": True, "quizzes": True,
    "files": True, "syllabus": True, "enrollments": True,
}


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

def _content_hash(data) -> str:
    """SHA-256 hash of content for change detection."""
    if isinstance(data, str):
        raw = data.encode("utf-8")
    else:
        raw = json.dumps(data, sort_keys=True).encode("utf-8")
    return hashlib.sha256(raw).hexdigest()


def _now_iso() -> str:
    return datetime.now(timezone.utc).isoformat()


# ---------------------------------------------------------------------------
# Course Syncer
# ---------------------------------------------------------------------------

class CourseSyncer:
    """Syncs all Canvas content for a single course."""

    def __init__(self, client: CanvasClient, canvas_id: int, slug: str):
        self.client = client
        self.canvas_id = canvas_id
        self.slug = slug

        self.course_dir = COURSES_DIR / slug
        self.canvas_dir = self.course_dir / "canvas"
        self.sync_state_path = self.canvas_dir / "sync_state.json"

        self.changes = []  # Track what changed
        self.errors = []   # Track errors per content type

        # Ensure directories exist
        MEMORY_DIR.mkdir(parents=True, exist_ok=True)
        for subdir in ["pages", "assignments", "announcements", "discussions",
                        "quizzes", "modules", "files", "syllabus", "enrollments"]:
            (self.canvas_dir / subdir).mkdir(parents=True, exist_ok=True)

    def load_sync_state(self) -> dict:
        if self.sync_state_path.exists():
            return json.loads(self.sync_state_path.read_text())
        return {"last_full_sync": None, "last_incremental_sync": None,
                "content_hashes": {}}

    def save_sync_state(self, state: dict):
        self.sync_state_path.write_text(json.dumps(state, indent=2) + "\n")

    def _write_if_changed(self, state: dict, key: str,
                          content: str, local_path: Path,
                          memory_path: Optional[Path] = None) -> bool:
        """Write file only if content hash changed. Returns True if written."""
        new_hash = _content_hash(content)
        old_hash = state["content_hashes"].get(key)

        if new_hash == old_hash:
            return False

        local_path.parent.mkdir(parents=True, exist_ok=True)
        local_path.write_text(content)

        if memory_path:
            memory_path.parent.mkdir(parents=True, exist_ok=True)
            memory_path.write_text(content)

        state["content_hashes"][key] = new_hash
        return True

    # -- Sync methods -------------------------------------------------------

    def sync_pages(self, state: dict) -> int:
        """Sync all Canvas pages. Returns count of changed items."""
        print(f"    Syncing pages...", end=" ", flush=True)
        changed = 0

        try:
            pages_list = self.client.get_pages(self.canvas_id)
        except CanvasAPIError as e:
            msg = f"pages: {e}"
            self.errors.append(msg)
            print(f"ERROR: {e}")
            return 0

        for page_summary in pages_list:
            page_url = page_summary.get("url", "")
            if not page_url:
                continue

            try:
                page = self.client.get_page(self.canvas_id, page_url)
            except CanvasAPIError:
                continue

            md = format_page_markdown(page, self.slug, self.client.base_url)
            key = f"pages/{page_url}"
            local_path = self.canvas_dir / "pages" / f"{page_url}.md"
            mem_path = MEMORY_DIR / memory_filename(
                self.slug, "canvas-page", page.get("title", page_url))

            if self._write_if_changed(state, key, md, local_path, mem_path):
                changed += 1
                self.changes.append(f"page: {page.get('title', page_url)}")

        print(f"{changed} changed out of {len(pages_list)}")
        return changed

    def sync_assignments(self, state: dict) -> int:
        """Sync all assignments."""
        print(f"    Syncing assignments...", end=" ", flush=True)
        changed = 0

        try:
            assignments = self.client.get_assignments(self.canvas_id)
        except CanvasAPIError as e:
            self.errors.append(f"assignments: {e}")
            print(f"ERROR: {e}")
            return 0

        for a in assignments:
            name = a.get("name", "untitled")
            md = format_assignment_markdown(a, self.slug, self.client.base_url)
            key = f"assignments/{a['id']}"
            safe_name = _slugify(name)
            local_path = self.canvas_dir / "assignments" / f"{a['id']}-{safe_name}.md"
            mem_path = MEMORY_DIR / memory_filename(
                self.slug, "canvas-assignment", name)

            if self._write_if_changed(state, key, md, local_path, mem_path):
                changed += 1
                self.changes.append(f"assignment: {name}")

        print(f"{changed} changed out of {len(assignments)}")
        return changed

    def sync_announcements(self, state: dict) -> int:
        """Sync all announcements."""
        print(f"    Syncing announcements...", end=" ", flush=True)
        changed = 0

        try:
            announcements = self.client.get_announcements(self.canvas_id)
        except CanvasAPIError as e:
            self.errors.append(f"announcements: {e}")
            print(f"ERROR: {e}")
            return 0

        for ann in announcements:
            title = ann.get("title", "untitled")
            posted = ann.get("posted_at", "")
            date_prefix = posted[:10] if posted else "undated"
            md = format_announcement_markdown(ann, self.slug, self.client.base_url)
            key = f"announcements/{ann['id']}"
            safe_title = _slugify(title)
            local_path = self.canvas_dir / "announcements" / f"{date_prefix}-{safe_title}.md"
            mem_path = MEMORY_DIR / memory_filename(
                self.slug, "canvas-announcement", f"{date_prefix}-{title}")

            if self._write_if_changed(state, key, md, local_path, mem_path):
                changed += 1
                self.changes.append(f"announcement: {title}")

        print(f"{changed} changed out of {len(announcements)}")
        return changed

    def sync_discussions(self, state: dict) -> int:
        """Sync discussion topics with replies."""
        print(f"    Syncing discussions...", end=" ", flush=True)
        changed = 0

        try:
            topics = self.client.get_discussion_topics(self.canvas_id)
        except CanvasAPIError as e:
            self.errors.append(f"discussions: {e}")
            print(f"ERROR: {e}")
            return 0

        non_announcement_topics = [t for t in topics if not t.get("is_announcement")]

        for topic in non_announcement_topics:
            title = topic.get("title", "untitled")
            replies = []
            try:
                entries = self.client.get_discussion_entries(
                    self.canvas_id, topic["id"])
                replies = entries[:20]  # Cap replies
            except CanvasAPIError:
                pass

            md = format_discussion_markdown(
                topic, replies, self.slug, self.client.base_url)
            key = f"discussions/{topic['id']}"
            safe_title = _slugify(title)
            local_path = self.canvas_dir / "discussions" / f"{topic['id']}-{safe_title}.md"
            mem_path = MEMORY_DIR / memory_filename(
                self.slug, "canvas-discussion", title)

            if self._write_if_changed(state, key, md, local_path, mem_path):
                changed += 1
                self.changes.append(f"discussion: {title}")

        print(f"{changed} changed out of {len(non_announcement_topics)}")
        return changed

    def sync_quizzes(self, state: dict) -> int:
        """Sync quiz metadata (no questions for academic integrity)."""
        print(f"    Syncing quizzes...", end=" ", flush=True)
        changed = 0

        try:
            quizzes = self.client.get_quizzes(self.canvas_id)
        except CanvasAPIError as e:
            self.errors.append(f"quizzes: {e}")
            print(f"ERROR: {e}")
            return 0

        for q in quizzes:
            title = q.get("title", "untitled")
            md = format_quiz_markdown(q, self.slug)
            key = f"quizzes/{q['id']}"
            safe_title = _slugify(title)
            local_path = self.canvas_dir / "quizzes" / f"{q['id']}-{safe_title}.md"
            mem_path = MEMORY_DIR / memory_filename(
                self.slug, "canvas-quiz", title)

            if self._write_if_changed(state, key, md, local_path, mem_path):
                changed += 1
                self.changes.append(f"quiz: {title}")

        print(f"{changed} changed out of {len(quizzes)}")
        return changed

    def sync_modules(self, state: dict) -> int:
        """Sync module structure as a course table of contents."""
        print(f"    Syncing modules...", end=" ", flush=True)

        try:
            modules = self.client.get_modules(self.canvas_id, include_items=True)
        except CanvasAPIError as e:
            self.errors.append(f"modules: {e}")
            print(f"ERROR: {e}")
            return 0

        # Save raw structure as JSON
        struct_path = self.canvas_dir / "modules" / "structure.json"
        struct_path.parent.mkdir(parents=True, exist_ok=True)
        struct_path.write_text(json.dumps(modules, indent=2, default=str) + "\n")

        # Generate summary markdown
        md = format_module_summary_markdown(modules, self.slug)
        key = "modules/summary"
        local_path = self.canvas_dir / "modules" / "summary.md"
        mem_path = MEMORY_DIR / f"{self.slug}__canvas-modules.md"

        changed = 0
        if self._write_if_changed(state, key, md, local_path, mem_path):
            changed = 1
            self.changes.append("modules: course structure")

        print(f"{changed} changed ({len(modules)} modules)")
        return changed

    def sync_files(self, state: dict, config: dict = None) -> int:
        """Sync course files (download based on rules)."""
        print(f"    Syncing files...", end=" ", flush=True)

        # Load download rules
        rules = (config or {}).get("file_download_rules", {})
        max_size = rules.get("max_file_size_mb", 50) * 1024 * 1024
        allowed_ext = set(rules.get("allowed_extensions",
                                    [".pdf", ".docx", ".pptx", ".xlsx",
                                     ".txt", ".csv", ".ipynb"]))
        skip_ext = set(rules.get("skip_extensions",
                                 [".mp4", ".mov", ".zip", ".tar"]))

        try:
            files = self.client.get_files(self.canvas_id)
        except CanvasAPIError as e:
            self.errors.append(f"files: {e}")
            print(f"ERROR: {e}")
            return 0

        downloaded = 0
        skipped = 0
        failed = 0

        for f in files:
            name = f.get("display_name", "unknown")
            size = f.get("size", 0)
            ext = os.path.splitext(name)[1].lower()

            # Skip based on rules
            if size > max_size:
                skipped += 1
                continue
            if ext in skip_ext:
                skipped += 1
                continue
            if allowed_ext and ext not in allowed_ext:
                skipped += 1
                continue

            # Validate download URL
            if not f.get("url"):
                skipped += 1
                continue

            # Check if file changed (using updated_at + size as proxy)
            key = f"files/{f['id']}"
            file_hash = f"{f.get('updated_at', '')}_{size}"
            old_hash = state["content_hashes"].get(key)
            if file_hash == old_hash:
                continue

            # Download
            dest = self.canvas_dir / "files" / name
            try:
                self.client.download_file(f, dest)
                state["content_hashes"][key] = file_hash
                downloaded += 1
                self.changes.append(f"file: {name}")
            except Exception as e:
                failed += 1
                self.errors.append(f"file download '{name}': {e}")

        # Generate file index for memory
        idx_md = format_file_index_markdown(files, self.slug)
        idx_key = "files/index"
        idx_path = self.canvas_dir / "files" / "_index.md"
        mem_path = MEMORY_DIR / f"{self.slug}__canvas-files.md"
        self._write_if_changed(state, idx_key, idx_md, idx_path, mem_path)

        parts = [f"{downloaded} downloaded"]
        if skipped:
            parts.append(f"{skipped} skipped")
        if failed:
            parts.append(f"{failed} FAILED")
        print(f"{', '.join(parts)} (of {len(files)} total)")
        return downloaded

    def sync_syllabus(self, state: dict) -> int:
        """Sync course syllabus body."""
        print(f"    Syncing syllabus...", end=" ", flush=True)

        try:
            course = self.client.get_syllabus(self.canvas_id)
        except CanvasAPIError as e:
            self.errors.append(f"syllabus: {e}")
            print(f"ERROR: {e}")
            return 0

        body = course.get("syllabus_body", "")
        if not body:
            print("no syllabus body")
            return 0

        md = format_syllabus_markdown(course, self.slug, self.client.base_url)
        key = "syllabus"
        local_path = self.canvas_dir / "syllabus" / "syllabus.md"
        mem_path = MEMORY_DIR / f"{self.slug}__canvas-syllabus.md"

        changed = 0
        if self._write_if_changed(state, key, md, local_path, mem_path):
            changed = 1
            self.changes.append("syllabus")

        print(f"{changed} changed")
        return changed

    def sync_enrollments(self, state: dict) -> int:
        """Sync enrollment roster (JSON only, NOT indexed in memory for privacy)."""
        print(f"    Syncing enrollments...", end=" ", flush=True)

        try:
            enrollments = self.client.get_enrollments(self.canvas_id)
        except CanvasAPIError as e:
            self.errors.append(f"enrollments: {e}")
            print(f"ERROR: {e}")
            return 0

        # Build roster (sanitized — just names, roles, no emails)
        roster = []
        for e in enrollments:
            user = e.get("user", {})
            roster.append({
                "user_id": e.get("user_id"),
                "name": user.get("name", user.get("sortable_name", "?")),
                "role": e.get("type", "?"),
                "state": e.get("enrollment_state", "?"),
            })

        key = "enrollments"
        content = json.dumps(roster, indent=2)
        new_hash = _content_hash(content)
        old_hash = state["content_hashes"].get(key)

        changed = 0
        if new_hash != old_hash:
            roster_path = self.canvas_dir / "enrollments" / "roster.json"
            roster_path.parent.mkdir(parents=True, exist_ok=True)
            roster_path.write_text(content + "\n")
            state["content_hashes"][key] = new_hash
            changed = 1
            self.changes.append(f"enrollments: {len(roster)} users")

        from collections import Counter
        counts = Counter(e["role"] for e in roster)
        roles_str = ", ".join(f"{k}:{v}" for k, v in sorted(counts.items()))
        print(f"{changed} changed ({len(roster)} users: {roles_str})")
        return changed

    # -- Orchestration ------------------------------------------------------

    def _run_sync(self, state: dict, sync_content: dict,
                  config: dict = None) -> dict:
        """Run sync methods based on sync_content flags. Returns summary."""
        summary = {}

        sync_methods = {
            "pages": lambda: self.sync_pages(state),
            "assignments": lambda: self.sync_assignments(state),
            "announcements": lambda: self.sync_announcements(state),
            "discussions": lambda: self.sync_discussions(state),
            "quizzes": lambda: self.sync_quizzes(state),
            "modules": lambda: self.sync_modules(state),
            "files": lambda: self.sync_files(state, config),
            "syllabus": lambda: self.sync_syllabus(state),
            "enrollments": lambda: self.sync_enrollments(state),
        }

        for content_type, sync_fn in sync_methods.items():
            if sync_content.get(content_type, True):
                summary[content_type] = sync_fn()
            else:
                summary[content_type] = 0

        return summary

    def full_sync(self, config: dict = None,
                  sync_content: dict = None) -> dict:
        """Run all enabled sync methods. Returns summary."""
        state = self.load_sync_state()
        self.changes = []
        self.errors = []

        content_flags = sync_content or ALL_CONTENT_TYPES

        print(f"\n  === Full sync: {self.slug} "
              f"(canvas_id: {self.canvas_id}) ===\n")

        summary = self._run_sync(state, content_flags, config)

        state["last_full_sync"] = _now_iso()
        state["last_incremental_sync"] = _now_iso()
        self.save_sync_state(state)

        _update_config_last_sync(self.canvas_id)

        total_changed = sum(summary.values())
        print(f"\n    Total: {total_changed} items changed")

        if self.errors:
            print(f"\n    Errors ({len(self.errors)}):")
            for err in self.errors:
                print(f"      ! {err}")

        if total_changed > 0:
            print(f"\n    Changed items:")
            for c in self.changes:
                print(f"      - {c}")
            print(f"\n    >> Run 'openclaw memory index --force' "
                  f"to update RAG index")

        return summary

    def incremental_sync(self, config: dict = None,
                         sync_content: dict = None) -> dict:
        """Same as full sync but relies on hash-based change detection.

        Canvas API doesn't reliably support filtering by updated_at for
        all endpoints, so we fetch everything but only write changed files.
        """
        state = self.load_sync_state()
        self.changes = []
        self.errors = []

        content_flags = sync_content or ALL_CONTENT_TYPES

        print(f"\n  === Incremental sync: {self.slug} "
              f"(canvas_id: {self.canvas_id}) ===\n")

        summary = self._run_sync(state, content_flags, config)

        state["last_incremental_sync"] = _now_iso()
        self.save_sync_state(state)

        _update_config_last_sync(self.canvas_id)

        total_changed = sum(summary.values())
        print(f"\n    Total: {total_changed} items changed")

        if self.errors:
            print(f"\n    Errors ({len(self.errors)}):")
            for err in self.errors:
                print(f"      ! {err}")

        if total_changed > 0:
            print(f"\n    Changed items:")
            for c in self.changes:
                print(f"      - {c}")
            print(f"\n    >> Run 'openclaw memory index --force' "
                  f"to update RAG index")
        else:
            print("    Everything up to date.")

        return summary


# ---------------------------------------------------------------------------
# Config helpers
# ---------------------------------------------------------------------------

def _load_canvas_config() -> dict:
    if CANVAS_CONFIG_PATH.exists():
        return json.loads(CANVAS_CONFIG_PATH.read_text())
    return {"active_courses": []}


def _update_config_last_sync(canvas_id: int):
    """Update last_sync in canvas-config.json."""
    config = _load_canvas_config()
    for c in config["active_courses"]:
        if c["canvas_id"] == canvas_id:
            c["last_sync"] = _now_iso()
            break
    CANVAS_CONFIG_PATH.write_text(json.dumps(config, indent=2) + "\n")


def _find_course_in_config(canvas_id: int) -> dict:
    """Find a course entry in canvas-config.json."""
    config = _load_canvas_config()
    for c in config["active_courses"]:
        if c["canvas_id"] == canvas_id:
            return c
    return None


def _get_sync_content_for_course(canvas_id: int) -> dict:
    """Get the sync_content flags for a specific course."""
    entry = _find_course_in_config(canvas_id)
    if entry:
        return entry.get("sync_content", ALL_CONTENT_TYPES)
    return dict(ALL_CONTENT_TYPES)


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def cmd_full(client: CanvasClient, course_id: int):
    config = _load_canvas_config()
    entry = _find_course_in_config(course_id)
    if not entry:
        print(f"  Course {course_id} is not activated. "
              f"Run 'canvas_courses.py activate {course_id}' first.",
              file=sys.stderr)
        sys.exit(1)

    sync_content = entry.get("sync_content", ALL_CONTENT_TYPES)
    syncer = CourseSyncer(client, course_id, entry["slug"])
    syncer.full_sync(config, sync_content)


def cmd_incremental(client: CanvasClient, course_id: int):
    config = _load_canvas_config()
    entry = _find_course_in_config(course_id)
    if not entry:
        print(f"  Course {course_id} is not activated.", file=sys.stderr)
        sys.exit(1)

    sync_content = entry.get("sync_content", ALL_CONTENT_TYPES)
    syncer = CourseSyncer(client, course_id, entry["slug"])
    syncer.incremental_sync(config, sync_content)


def cmd_all(client: CanvasClient, incremental: bool = False):
    config = _load_canvas_config()
    active = config.get("active_courses", [])

    if not active:
        print("  No active courses. Run 'canvas_courses.py activate' first.")
        return

    print(f"\n  Syncing {len(active)} active course(s)...")

    succeeded = []
    failed = {}

    for entry in active:
        if not entry.get("sync_enabled", True):
            print(f"\n  Skipping {entry['slug']} (sync disabled)")
            continue

        sync_content = entry.get("sync_content", ALL_CONTENT_TYPES)
        syncer = CourseSyncer(client, entry["canvas_id"], entry["slug"])
        try:
            if incremental:
                syncer.incremental_sync(config, sync_content)
            else:
                syncer.full_sync(config, sync_content)
            succeeded.append(entry["slug"])
        except Exception as e:
            failed[entry["slug"]] = str(e)
            print(f"\n  ERROR syncing {entry['slug']}: {e}", file=sys.stderr)
            traceback.print_exc()

    # Final summary
    print(f"\n  {'=' * 50}")
    print(f"  Sync complete: {len(succeeded)} succeeded, "
          f"{len(failed)} failed")
    if failed:
        for slug, error in failed.items():
            print(f"    FAILED: {slug} -- {error[:100]}")


def cmd_status(course_id: int):
    entry = _find_course_in_config(course_id)
    if not entry:
        print(f"  Course {course_id} is not activated.", file=sys.stderr)
        sys.exit(1)

    slug = entry["slug"]
    canvas_dir = COURSES_DIR / slug / "canvas"
    sync_state_path = canvas_dir / "sync_state.json"

    print(f"\n  === Sync status: {slug} ===\n")

    if sync_state_path.exists():
        state = json.loads(sync_state_path.read_text())
        print(f"    Last full sync:        "
              f"{state.get('last_full_sync', 'never')}")
        print(f"    Last incremental sync: "
              f"{state.get('last_incremental_sync', 'never')}")
        print(f"    Tracked items:         "
              f"{len(state.get('content_hashes', {}))}")
    else:
        print("    No sync state found. Run a full sync first.")

    # Sync content flags
    sync_content = entry.get("sync_content", ALL_CONTENT_TYPES)
    disabled = [k for k, v in sync_content.items() if not v]
    if disabled:
        print(f"    Disabled content types: {', '.join(disabled)}")

    print(f"\n    Local content:")
    for content_type in ["pages", "assignments", "announcements",
                         "discussions", "quizzes", "modules", "files",
                         "syllabus", "enrollments"]:
        type_dir = canvas_dir / content_type
        if type_dir.exists():
            count = len([f for f in type_dir.iterdir()
                         if not f.name.startswith("_")])
            print(f"      {content_type:<16} {count} files")

    # Count memory files
    prefix = f"{slug}__canvas-"
    if MEMORY_DIR.exists():
        mem_count = len([f for f in MEMORY_DIR.iterdir()
                         if f.name.startswith(prefix)])
    else:
        mem_count = 0
    print(f"\n    Memory files (RAG):    {mem_count}")


def main():
    if len(sys.argv) < 2:
        print(__doc__)
        sys.exit(1)

    command = sys.argv[1]

    if command == "status":
        if len(sys.argv) < 3:
            print("Usage: canvas_sync.py status <course_id>", file=sys.stderr)
            sys.exit(1)
        cmd_status(int(sys.argv[2]))
        return

    # Commands that need the API client
    base_url, [REDACTED_SECRET])
    client = CanvasClient(base_url, token, write_mode="deny")

    if command == "full":
        if len(sys.argv) < 3:
            print("Usage: canvas_sync.py full <course_id>", file=sys.stderr)
            sys.exit(1)
        cmd_full(client, int(sys.argv[2]))

    elif command == "incremental":
        if len(sys.argv) < 3:
            print("Usage: canvas_sync.py incremental <course_id>",
                  file=sys.stderr)
            sys.exit(1)
        cmd_incremental(client, int(sys.argv[2]))

    elif command == "all":
        incremental = "--incremental" in sys.argv
        cmd_all(client, incremental)

    else:
        print(f"Unknown command: {command}", file=sys.stderr)
        print("Available: full, incremental, all, status", file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()


path: lib/extract_slides.py
content: #!/usr/bin/env python3
"""
extract_slides.py — PPTX → Compact Markdown index generator for course-ta skill.

Produces lean, summaritive memory files optimized for fast RAG lookup:
- One file per PPTX (not per slide)
- Frontmatter: module, file, slide count, topic list
- Per-slide: title + key bullets only (strips filler, URLs, single words)
- Speaker notes: first 2 sentences only (high-signal context)
- Skips image-only, blank, and boilerplate slides

Usage:
    python3 extract_slides.py <course_dir> <memory_dir>
"""

import os
import sys
import re
import json
import datetime
from pathlib import Path

try:
    from pptx import Presentation
    from pptx.enum.text import PP_ALIGN
except ImportError:
    print("ERROR: python-pptx not installed. Run: pip3 install python-pptx")
    sys.exit(1)


# ── Text utilities ──────────────────────────────────────────────────────────

BOILERPLATE = re.compile(
    r'^(http[s]?://|www\.|©|copyright|confidential|proprietary'
    r'|all rights reserved|slide \d+|\d+$)',
    re.IGNORECASE
)

def clean_text(t: str) -> str:
    t = t.strip()
    t = re.sub(r'\s+', ' ', t)
    return t

def is_meaningful(line: str) -> bool:
    line = line.strip()
    if len(line) < 4:
        return False
    if BOILERPLATE.search(line):
        return False
    return True

def truncate_notes(notes_text: str, max_sentences: int = 2) -> str:
    """Return first N sentences of speaker notes."""
    if not notes_text:
        return ""
    # Split on sentence boundaries
    sentences = re.split(r'(?<=[.!?])\s+', notes_text.strip())
    chosen = [s.strip() for s in sentences[:max_sentences] if len(s.strip()) > 10]
    return " ".join(chosen)

def slugify(name: str) -> str:
    name = re.sub(r'\.pptx$', '', name, flags=re.IGNORECASE)
    name = name.lower()
    name = re.sub(r'[^a-z0-9]+', '-', name)
    return name.strip('-')


# ── Slide extraction ─────────────────────────────────────────────────────────

def extract_slide(slide, slide_num: int):
    """
    Extract title + key content bullets from one slide.
    Returns None if slide has no useful text (image-only, blank, etc.)
    """
    title = None
    bullets = []
    notes_text = ""

    # Speaker notes
    try:
        if slide.has_notes_slide:
            raw_notes = slide.notes_slide.notes_text_frame.text.strip()
            notes_text = truncate_notes(raw_notes, max_sentences=2)
    except Exception:
        pass

    for shape in slide.shapes:
        if not shape.has_text_frame:
            continue
        try:
            text = shape.text_frame.text.strip()
        except Exception:
            continue
        if not text:
            continue

        # Determine if this is a title placeholder
        is_title = False
        try:
            if shape.is_placeholder and shape.placeholder_format.idx == 0:
                is_title = True
        except Exception:
            pass

        if is_title:
            title = clean_text(text.splitlines()[0]) if text else None
        else:
            # Collect meaningful lines as bullets
            for line in text.splitlines():
                line = clean_text(line)
                if is_meaningful(line):
                    bullets.append(line)

    if not title and bullets:
        title = bullets.pop(0)

    # Deduplicate bullets while preserving order
    seen = set()
    unique_bullets = []
    for b in bullets:
        key = b.lower()
        if key not in seen:
            seen.add(key)
            unique_bullets.append(b)

    # Skip essentially empty slides
    if not title and not unique_bullets:
        return None

    return {
        "num": slide_num,
        "title": title or f"Slide {slide_num}",
        "bullets": unique_bullets[:12],   # cap at 12 bullets per slide
        "notes": notes_text,
    }


# ── PPTX → Markdown ──────────────────────────────────────────────────────────

def pptx_to_markdown(pptx_path: Path, module_label: str, course_meta: dict) -> str:
    """
    Convert a PPTX to a compact, search-optimised markdown file.

    Structure:
      # <Course> | <Module> | <Lecture title>
      ## Quick Reference  ← topic list for fast triage
      ## Slides           ← per-slide title + bullets + 2-sentence note
    """
    prs = Presentation(str(pptx_path))
    slides_data = []
    for i, slide in enumerate(prs.slides, 1):
        result = extract_slide(slide, i)
        if result:
            slides_data.append(result)

    course_name = course_meta.get("course_name", "Course")
    semester    = course_meta.get("semester", "")
    lecture_id  = pptx_path.stem

    lines = []

    # ── Frontmatter ──
    lines.append(f"# {course_name} | {module_label} | {lecture_id}")
    lines.append(f"> **File:** `{pptx_path.name}`")
    lines.append(f"> **Course:** {course_name} ({semester})")
    lines.append(f"> **Module:** {module_label}")
    lines.append(f"> **Slides:** {len(prs.slides)} total, {len(slides_data)} indexed")
    lines.append(f"> **Indexed:** {datetime.date.today().isoformat()}")
    lines.append("")

    # ── Quick Reference (topic list) ──
    lines.append("## Quick Reference")
    lines.append("")
    lines.append("Key topics covered in this lecture:")
    lines.append("")
    for s in slides_data:
        lines.append(f"- **Slide {s['num']}:** {s['title']}")
    lines.append("")

    # ── Slide Content ──
    lines.append("## Slide Content")
    lines.append("")

    for s in slides_data:
        lines.append(f"### Slide {s['num']}: {s['title']}")
        if s["bullets"]:
            for b in s["bullets"]:
                lines.append(f"- {b}")
        if s["notes"]:
            lines.append(f"> 📝 *{s['notes']}*")
        lines.append("")

    return "\n".join(lines)


# ── Course processing ─────────────────────────────────────────────────────────

def find_course_meta(course_dir: Path) -> dict:
    search = course_dir
    for _ in range(6):
        candidate = search / "course-ta.json"
        if candidate.exists():
            try:
                with open(candidate) as f:
                    return json.load(f)
            except Exception:
                pass
        search = search.parent
    return {}

def process_course(course_dir: Path, memory_dir: Path):
    course_meta = find_course_meta(course_dir)
    course_name = course_dir.name  # e.g. ai-essentials-spring2026

    slides_dirs = sorted(course_dir.rglob("slides"))
    if not slides_dirs:
        print(f"  No slides/ directories found under {course_dir}")
        return []

    generated = []
    for slides_dir in slides_dirs:
        module_name = slides_dir.parent.name   # e.g. module1
        module_label = module_name.replace("module", "Module ")  # "Module 1"
        course_meta["module"] = module_label

        pptx_files = sorted(slides_dir.glob("*.pptx"))
        if not pptx_files:
            print(f"  No .pptx files in {slides_dir}")
            continue

        for pptx_path in pptx_files:
            real_path = pptx_path.resolve()
            slug = slugify(pptx_path.name)
            md_name = f"{course_name}__{module_name}__{slug}.md"
            md_path = memory_dir / md_name

            print(f"  [{module_name}] {pptx_path.name} → {md_name}")
            try:
                md_content = pptx_to_markdown(real_path, module_label, course_meta)
                md_path.write_text(md_content, encoding="utf-8")
                generated.append(md_name)
            except Exception as e:
                print(f"    ERROR: {e}")

    print(f"\nGenerated {len(generated)} index files in {memory_dir}")
    return generated


def main():
    if len(sys.argv) < 3:
        print(__doc__)
        sys.exit(1)

    course_dir = Path(sys.argv[1]).resolve()
    memory_dir = Path(sys.argv[2]).resolve()

    if not course_dir.exists():
        print(f"ERROR: course_dir not found: {course_dir}")
        sys.exit(1)
    memory_dir.mkdir(parents=True, exist_ok=True)

    print(f"Course dir:  {course_dir}")
    print(f"Memory dir:  {memory_dir}")
    print("")

    # Auto-detect: if course_dir itself has slides/ subdirs, process it directly;
    # otherwise treat it as a root containing multiple course folders.
    has_slides = any(True for _ in course_dir.rglob("slides"))

    if has_slides:
        process_course(course_dir, memory_dir)
    else:
        for subdir in sorted(d for d in course_dir.iterdir() if d.is_dir()):
            print(f"\n=== Course: {subdir.name} ===")
            process_course(subdir, memory_dir)


if __name__ == "__main__":
    main()


path: lib/log_interaction.py
content: #!/usr/bin/env python3
"""
log_interaction.py — Append a Q&A interaction to the daily audit log.

Usage:
    python3 log_interaction.py \
        --log-dir <path>     \
        --user-id <id>       \
        --channel <id>       \
        --thread <id|"">     \
        --question <text>    \
        --answer <text>      \
        --status <ok|rate_limited|out_of_scope|no_material>

Output: JSON entry appended to <log-dir>/YYYY-MM-DD.jsonl
Prints the log entry to stdout on success.
"""

import argparse
import json
import sys
from datetime import datetime, timezone
from pathlib import Path


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--log-dir", type=Path, required=True)
    parser.add_argument("--user-id", required=True)
    parser.add_argument("--channel", default="")
    parser.add_argument("--thread", default="")
    parser.add_argument("--question", required=True)
    parser.add_argument("--answer", required=True)
    parser.add_argument("--status", default="ok",
                        choices=["ok", "rate_limited", "out_of_scope",
                                 "no_material", "admin_edit", "canvas_write",
                                 "forward_failed"])
    args = parser.parse_args()

    args.log_dir.mkdir(parents=True, exist_ok=True)

    now = datetime.now(timezone.utc)
    log_file = args.log_dir / f"{now.strftime('%Y-%m-%d')}.jsonl"

    entry = {
        "timestamp": now.isoformat(),
        "userId": args.user_id,
        "channelId": args.channel,
        "threadId": args.thread,
        "status": args.status,
        "question": args.question[:300],
        "answer": args.answer[:500],
    }

    with open(log_file, "a", encoding="utf-8") as f:
        f.write(json.dumps(entry, ensure_ascii=False) + "\n")

    print(json.dumps({"logged": True, "file": str(log_file), "status": args.status}))


if __name__ == "__main__":
    main()


path: lib/paths.py
content: """Centralized path resolution for the Course TA skill module.

All paths are relative to the skill root directory:
    ~/.openclaw/skills/course-ta/

Layout:
    lib/              Python modules (this file lives here)
    config/           Runtime configs (course-ta.json, canvas-config.json, etc.)
    data/
      courses/        Course content (Canvas sync cache, GDrive slides)
      memory/         Generated markdown files for RAG indexing
      logs/           Interaction audit logs (JSONL per day)
      credentials/    API tokens (gitignored)
      tests/          Test checklists
"""

from pathlib import Path

# Skill root: two levels up from lib/paths.py
SKILL_DIR = Path(__file__).resolve().parent.parent

# Configuration
CONFIG_DIR = SKILL_DIR / "config"
COURSE_TA_CONFIG = CONFIG_DIR / "course-ta.json"
CANVAS_CONFIG = CONFIG_DIR / "canvas-config.json"
COURSE_CONFIGS_DIR = CONFIG_DIR / "course-configs"
RATE_LIMIT_STATE = CONFIG_DIR / "ta-rate-limit.json"

# Data
DATA_DIR = SKILL_DIR / "data"
COURSES_DIR = DATA_DIR / "courses"
MEMORY_DIR = DATA_DIR / "memory"
LOGS_DIR = DATA_DIR / "logs"
CREDENTIALS_DIR = DATA_DIR / "credentials"
CANVAS_CREDENTIALS = CREDENTIALS_DIR / "canvas.json"

# Also expose the workspace path for OpenClaw memory indexing
WORKSPACE_DIR = Path.home() / ".openclaw" / "workspace"


path: lib/rate_limit.py
content: #!/usr/bin/env python3
"""
rate_limit.py — Per-user rate limit checker/updater for course-ta skill.

Usage:
    python3 rate_limit.py check <userId> [--state <path>] [--max-hour N] [--max-day N] [--config <path>]
    python3 rate_limit.py record <userId> [--state <path>]

Privileged users defined in course-ta.json under "privileged_users" get their
own max_hour and max_day limits, overriding the defaults.

Exit codes:
    0 = allowed
    1 = rate limited (prints reason to stdout as JSON)
    2 = error

Stdout (JSON):
    {"allowed": true/false, "reason": "...", "count_hour": N, "count_day": N}
"""

import json
import sys
import time
import os
import argparse
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from paths import RATE_LIMIT_STATE, COURSE_TA_CONFIG

DEFAULT_STATE = RATE_LIMIT_STATE
DEFAULT_CONFIG = COURSE_TA_CONFIG
DEFAULT_MAX_HOUR = 20
DEFAULT_MAX_DAY = 60


def get_user_limits(user_id: str, config_path: Path, default_max_hour: int, default_max_day: int):
    """Return (max_hour, max_day) for a user, respecting privileged_users in config."""
    try:
        config = json.loads(config_path.read_text())
        privileged = config.get("privileged_users", {})
        if user_id in privileged:
            entry = privileged[user_id]
            return entry.get("max_hour", default_max_hour), entry.get("max_day", default_max_day)
    except Exception:
        pass
    return default_max_hour, default_max_day

def load_state(path: Path) -> dict:
    if path.exists():
        try:
            return json.loads(path.read_text())
        except Exception:
            return {}
    return {}

def save_state(path: Path, state: dict):
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(state, indent=2))

def clean_old(timestamps: list, now: float) -> list:
    """Keep only last 24h of timestamps."""
    cutoff = now - 86400
    return [t for t in timestamps if t > cutoff]

def count_in_window(timestamps: list, now: float, window: int) -> int:
    cutoff = now - window
    return sum(1 for t in timestamps if t > cutoff)

def check(user_id: str, state_path: Path, max_hour: int, max_day: int, config_path: Path = DEFAULT_CONFIG):
    # Check privileged_users for custom limits
    max_hour, max_day = get_user_limits(user_id, config_path, max_hour, max_day)

    state = load_state(state_path)
    now = time.time()
    user_data = state.get(user_id, {"messages": []})
    msgs = clean_old(user_data.get("messages", []), now)

    count_hour = count_in_window(msgs, now, 3600)
    count_day  = count_in_window(msgs, now, 86400)

    if count_hour >= max_hour:
        result = {
            "allowed": False,
            "reason": f"hourly limit ({max_hour} messages/hour)",
            "count_hour": count_hour,
            "count_day": count_day
        }
    elif count_day >= max_day:
        result = {
            "allowed": False,
            "reason": f"daily limit ({max_day} messages/day)",
            "count_hour": count_hour,
            "count_day": count_day
        }
    else:
        result = {
            "allowed": True,
            "reason": "ok",
            "count_hour": count_hour,
            "count_day": count_day
        }

    print(json.dumps(result))
    sys.exit(0 if result["allowed"] else 1)

def record(user_id: str, state_path: Path):
    state = load_state(state_path)
    now = time.time()
    user_data = state.get(user_id, {"messages": []})
    msgs = clean_old(user_data.get("messages", []), now)
    msgs.append(now)
    state[user_id] = {"messages": msgs}
    save_state(state_path, state)
    print(json.dumps({"recorded": True, "count_hour": count_in_window(msgs, now, 3600)}))

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("action", choices=["check", "record"])
    parser.add_argument("user_id")
    parser.add_argument("--state", type=Path, default=DEFAULT_STATE)
    parser.add_argument("--config", type=Path, default=DEFAULT_CONFIG)
    parser.add_argument("--max-hour", type=int, default=DEFAULT_MAX_HOUR)
    parser.add_argument("--max-day", type=int, default=DEFAULT_MAX_DAY)
    args = parser.parse_args()

    if args.action == "check":
        check(args.user_id, args.state, args.max_hour, args.max_day, args.config)
    elif args.action == "record":
        record(args.user_id, args.state)

if __name__ == "__main__":
    main()


path: README.md
content: # Course TA — OpenClaw Skill

An AI-powered virtual teaching assistant skill for [OpenClaw](https://github.com/nichochar/openclaw) that integrates with Canvas LMS and Discord to answer student questions using RAG over course materials.

## Features

- **Canvas LMS integration** — Auto-discovers courses, syncs all content (pages, assignments, announcements, discussions, quizzes, files, syllabus, enrollments)
- **Multi-course support** — Manages multiple courses/sections with full data isolation
- **Discord channel routing** — Maps channels to specific courses with security boundaries
- **RAG-powered Q&A** — Converts Canvas content and PPTX slides to markdown for retrieval-augmented answering
- **Rate limiting** — Per-user hourly/daily limits with privileged user tiers
- **Admin commands** — Inline content editing, Canvas sync triggers, submission tracking
- **Dashboard tools** — Deadlines, grades, engagement, roster, cross-section broadcast
- **Audit logging** — JSONL logs of all interactions
- **Write gating** — All Canvas write operations require explicit user confirmation; reads are unrestricted
- **Incremental sync** — SHA-256 change detection, only writes files that actually differ
- **Cron scheduling** — Periodic background sync (configurable interval)

## Directory Structure

```
course-ta/
├── SKILL.md                    # Agent behavior specification
├── .gitignore                  # Protects credentials, data, logs
├── README.md                   # This file
│
├── lib/                        # Python modules
│   ├── paths.py                # Central path resolver
│   ├── canvas_api.py           # Canvas REST API client
│   ├── canvas_content.py       # HTML → markdown converter
│   ├── canvas_courses.py       # Multi-course management CLI
│   ├── canvas_sync.py          # Content sync engine
│   ├── canvas_dashboard.py     # Admin dashboard tools
│   ├── extract_slides.py       # PPTX → markdown extractor
│   ├── rate_limit.py           # Per-user rate limiter
│   └── log_interaction.py      # Audit logger
│
├── config/                     # Runtime configs (created during setup)
│   ├── course-ta.json.template
│   ├── canvas-config.json.template
│   └── course-configs/         # Per-course configs
│
├── data/                       # All gitignored
│   ├── credentials/            # Canvas API token
│   ├── courses/                # Synced course content
│   ├── memory/                 # Generated RAG markdown files
│   └── logs/                   # Interaction audit logs
│
├── references/                 # Setup documentation
└── scripts/                    # Setup automation
```

## Quick Start

### Prerequisites

- [OpenClaw](https://github.com/nichochar/openclaw) installed and running
- Python 3.9+ with `requests` and `beautifulsoup4`:
  ```bash
  pip3 install requests beautifulsoup4
  ```
- A Canvas LMS API token (generate from Canvas → Account → Settings → New Access Token)

### 1. Install the skill

Copy this directory to `~/.openclaw/skills/course-ta/`.

### 2. Configure Canvas credentials

```bash
cp data/credentials/canvas.json.template data/credentials/canvas.json
# Edit with your Canvas base URL and API token
```

### 3. Initialize configs

```bash
cp config/course-ta.json.template config/course-ta.json
cp config/canvas-config.json.template config/canvas-config.json
# Edit course-ta.json with your Discord channel IDs, admin users, etc.
```

### 4. Discover and activate courses

```bash
cd ~/.openclaw/skills/course-ta/lib

# List available Canvas courses
python3 canvas_courses.py list

# Activate a course
python3 canvas_courses.py activate <canvas_course_id>

# Or map to an existing local directory
python3 canvas_courses.py map <canvas_course_id> <slug>
```

### 5. Sync course content

```bash
# Full initial sync
python3 canvas_sync.py full <canvas_course_id>

# Incremental sync (for periodic updates)
python3 canvas_sync.py all --incremental
```

### 6. Index memory for OpenClaw RAG

```bash
# Symlink skill memory to workspace (if not already done)
ln -sf ~/.openclaw/skills/course-ta/data/memory ~/.openclaw/workspace/memory

# Rebuild index
openclaw memory index --force
```

### 7. Verify

```bash
python3 canvas_api.py validate
python3 canvas_courses.py status
python3 canvas_dashboard.py deadlines
```

## Dashboard Commands

```bash
python3 canvas_dashboard.py deadlines [--days 7]
python3 canvas_dashboard.py submissions <course_id> <assignment_name>
python3 canvas_dashboard.py engagement <course_id>
python3 canvas_dashboard.py grades <course_id>
python3 canvas_dashboard.py roster <course_id>
python3 canvas_dashboard.py broadcast "<message>"
```

## Security Model

- **Read operations** (Canvas API GET, file reads): unrestricted
- **Write operations** (Canvas API POST/PUT, file edits): require explicit user confirmation
- **Channel isolation**: each Discord channel maps to exactly one course; the agent only reads memory files matching that course's slug
- **Enrollment privacy**: student rosters are stored as JSON but never indexed in RAG memory
- **Credentials**: stored in `data/credentials/` (gitignored)
- **Audit trail**: all interactions logged to `data/logs/` with timestamps, user IDs, and status codes

## License

MIT


path: references/setup-guide.md
content: # Course TA Setup Guide

Complete walkthrough for setting up the Course TA on the professor's Mac Mini.

## Prerequisites

- OpenClaw installed and configured with a model (e.g., `openclaw models status` shows a default model with auth)
- Discord bot created and token added (`openclaw channels add --channel discord --token "BOT_TOKEN"`)
- Gateway running (`openclaw gateway`)

## Step 1: Run the Setup Script

```bash
# From the skill directory:
bash scripts/setup_workspace.sh
```

This creates:
- `<workspace>/memory/` — drop course files here (flat, no subdirectories)
- `<workspace>/course-ta.json` — channel config (edit after setup)
- `<workspace>/AGENTS.md` — agent persona (auto-generated, editable)

## Step 2: Add Course Materials

Place files **directly** in `<workspace>/memory/`. OpenClaw memory only indexes files at the root of this directory — subdirectories are NOT scanned.

Supported formats: PDF, Markdown, TXT, PPTX.

Use descriptive filenames to organize:

```
memory/
├── lecture01-intro-to-ai.pdf
├── lecture02-ml-basics.pdf
├── lecture03-deep-learning.md
├── reading-turing-1950.pdf
└── syllabus.pdf
```

Then index:

```bash
openclaw memory index --force
```

Verify with:

```bash
openclaw memory status
# Should show indexed files > 0 and chunks > 0
```

Run `openclaw memory index --force` each time new materials are added.

## Step 3: Configure Channel Restrictions

### Option A: Discord-side (recommended, hard boundary)

In the Discord server:
1. Go to **Server Settings → Roles** — ensure the bot role exists
2. Go to each channel's **Edit Channel → Permissions**
3. For channels where the bot should NOT respond: deny the bot role **View Channel** and **Send Messages**
4. For the course Q&A channel(s): allow **View Channel**, **Send Messages**, **Read Message History**

This is the strongest restriction — the bot physically cannot see or post in unauthorized channels.

### Option B: Skill-level config (soft boundary)

Edit `<workspace>/course-ta.json`:

```json
{
  "allowed_channels": ["channel:1234567890", "channel:0987654321"],
  "course_name": "AI Essentials",
  "professor_name": "Prof. Smith",
  "semester": "Spring 2026"
}
```

To find channel IDs:
1. In Discord: **Settings → Advanced → enable Developer Mode**
2. Right-click the channel → **Copy Channel ID**
3. Prefix with `channel:` in the config

**Best practice**: use both Option A and B together for defense in depth.

## Step 4: Customize Agent Identity (Optional)

```bash
openclaw agents set-identity main \
  --name "Course TA" \
  --emoji "🎓"
```

## Step 5: Bind Discord to the Agent

If using a dedicated agent (not `main`):

```bash
openclaw agents add course-ta --bind discord --workspace <workspace-dir>
```

If using the default `main` agent, Discord is already routed there.

## Step 6: Verify

```bash
# Check Discord connection
openclaw channels status --probe

# Check memory index
openclaw memory status

# Test a question
openclaw agent --message "What topics does lecture 1 cover?"
```

## Updating Materials Mid-Semester

1. Add/replace files in `memory/`
2. Reindex: `openclaw memory index --force`
3. No restart needed — the next query picks up new content

## Removing the TA

```bash
openclaw skills uninstall course-ta
rm <workspace>/course-ta.json
# Optionally clean memory/ if no longer needed
```


path: scripts/setup_workspace.sh
content: #!/usr/bin/env bash
# Setup script for Course TA skill
# Creates workspace directories, config, and AGENTS.md on macOS (Mac Mini)

set -euo pipefail

# Resolve workspace directory
if [ -n "${OPENCLAW_WORKSPACE_DIR:-}" ]; then
  WORKSPACE="$OPENCLAW_WORKSPACE_DIR"
else
  WORKSPACE="$HOME/.openclaw/workspace"
fi

echo "==> Using workspace: $WORKSPACE"

# Create memory directory (course materials go here directly, not in subdirectories)
MEMORY_DIR="$WORKSPACE/memory"
mkdir -p "$MEMORY_DIR"
echo "==> Created $MEMORY_DIR"

# Create course-ta.json config if not present
CONFIG_FILE="$WORKSPACE/course-ta.json"
if [ ! -f "$CONFIG_FILE" ]; then
  cat > "$CONFIG_FILE" << 'JSONEOF'
{
  "allowed_channels": [],
  "course_name": "AI Essentials",
  "professor_name": "Professor",
  "semester": "Spring 2026"
}
JSONEOF
  echo "==> Created $CONFIG_FILE (edit with your channel IDs and course info)"
else
  echo "==> $CONFIG_FILE already exists, skipping"
fi

# Create or update AGENTS.md
AGENTS_FILE="$WORKSPACE/AGENTS.md"
if [ ! -f "$AGENTS_FILE" ]; then
  cat > "$AGENTS_FILE" << 'MDEOF'
# Agent Instructions

## Role

You are a course teaching assistant. Your primary function is to help students understand course material by answering questions based on indexed course documents.

## Behavior

- Always search memory before answering course-related questions.
- Only provide information grounded in course materials.
- Be helpful, patient, and pedagogical.
- Do not provide direct answers to homework or exams.
- Do not discuss grades or evaluation criteria.
- Refuse off-topic requests politely.
- Respond in the student's language (default English, Chinese if asked in Chinese).

## Channel Policy

Check `course-ta.json` in the workspace root for allowed Discord channel IDs. If the file lists specific channels, only respond in those channels.
MDEOF
  echo "==> Created $AGENTS_FILE"
else
  echo "==> $AGENTS_FILE already exists, skipping (check it includes TA instructions)"
fi

echo ""
echo "Setup complete. Next steps:"
echo "  1. Place course materials (PDF, PPTX, MD) directly in: $MEMORY_DIR"
echo "  2. Run: openclaw memory index --force"
echo "  3. Edit $CONFIG_FILE with your Discord channel IDs and course info"
echo "  4. Ensure your Discord bot permissions are set (see references/setup-guide.md)"


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)