Skill: pangfan/gitlab-batch-cloner (version 1.0.1)

--- name: pf-clone-gitlab description: Batch clone GitLab group projects with branch checkout and Excel indexing. Use when user needs to clone all projects from a GitLab group, organize by group structure, checkout multiple branches (master/default + latest + release/prod), and generate an Excel index file with project metadata and branch information. --- # Clone GitLab Skill Batch clone all projects from GitLab group(s), maintain folder hierarchy, checkout key branches, and generate an Excel index. ## When to Use - User wants to batch clone all projects from one or more GitLab top-level groups - Need to maintain original group/subgroup folder hierarchy locally - Need to checkout multiple branches (default + latest active + release/prod) - Need an Excel index file (`01.Index.xlsx`) with per-group sheets ## Requirements Before starting, collect from user: | Parameter | Required | Default | Notes | |-----------|----------|---------|-------| | GitLab URL | ✅ | — | e.g. `https://gitlab.company.com` | | Personal Access Token | ✅ | — | Needs `read_api` + `read_repository` scopes | | Target Group(s) | ✅ | — | Group/sub-group/project paths, comma-separated. Supports: top-level group (`myGroup`), sub-group path (`myGroup/mySubGroup`), or direct project path (`myGroup/mySubGroup/my-project`) | | Local Storage Path | ❌ | `~/Desktop/Code` | Where repos are stored | | Auth Method | ❌ | HTTPS+Token | Or SSH if key is configured | | Mode | ❌ | clone | `clone` (default), `update` (pull only), or `sync` (clone+pull+cleanup stale Excel rows) | | Workers | ❌ | 4 | Parallel clone workers (`GITLAB_WORKERS`) | | Total Timeout | ❌ | 0 (none) | Global timeout in seconds (`GITLAB_TOTAL_TIMEOUT`). 0=no limit | ## Workflow ### Step 1: Collect Input Ask user for the 4 parameters above. Pass them as environment variables to the script: ``` GITLAB_URL, GITLAB_TOKEN, GITLAB_GROUPS (comma-separated), GITLAB_BASE_DIR ``` ### Step 2: Run the Script ```bash cd <skill-dir>/scripts python3 clone_and_index.py ``` The script handles everything: 1. Resolves each input path — auto-detects if it's a top-level group, sub-group, or direct project 2. For top-level groups: fetches all subgroups recursively (skips if access denied) 3. For sub-group paths (e.g. `myGroup/mySubGroup`): directly resolves and syncs that sub-group and its descendants only 4. For direct project paths: syncs only that specific project 5. Clones new projects / pulls existing ones (with 5-min timeout per project) 6. Uses multiprocessing for parallel clone/pull (process-group kill on timeout to prevent orphan git processes) 7. Checks out branches: default + latest active + release/prod 8. **Incrementally** updates `01.Index.xlsx` every 50 projects (so partial results survive crashes) 9. On SIGTERM/SIGINT, emergency-flushes pending results to Excel before exit 10. Prints per-group project counts during discovery and real-time progress with elapsed time 11. In `sync` mode, the final Excel write removes rows for deleted/archived projects and handles cross-group migrations ### Step 3: Report Results After the script finishes, report: - Total projects cloned/updated - Any failed/timed-out projects (the script prints a summary table) ## Modes ### Update Only If projects already exist and user just wants to update: ```bash GITLAB_MODE=update python3 clone_and_index.py ``` This skips clone, only does `git fetch --all && git pull` on existing repos, re-checkouts branches, and refreshes the Excel. ### Sync (Full Sync with Cleanup) For a complete sync that also cleans up stale data in the Excel: ```bash GITLAB_MODE=sync python3 clone_and_index.py ``` This behaves like `clone` mode (new repos are cloned, existing ones are pulled), but additionally: - Removes Excel rows for projects that no longer exist on GitLab (deleted/archived) - Handles projects that moved between groups (updates path, removes old row) - Only cleans up sheets belonging to the groups specified in `GITLAB_GROUPS` (won't touch other groups' data) ## Excel Specification **File:** `01.Index.xlsx` inside `<local-path>/` (e.g. `~/Desktop/Code/01.Index.xlsx`) **Sheets:** One sheet per top-level group (sheet name = group name, e.g. "myGroup") **Columns:** | Col | Field | Content | |-----|-------|---------| | A | 主Group名称 | Top-level group (e.g. myGroup) | | B | 子Group路径 | Full group path without project name | | C | Project路径 | Full path (e.g. myGroup/mySubGroup/myProject) | | D | Project名称 | Project name | | E | Project描述 | GitLab description | | F | 已checkout分支 | All local branches, one per line | | G | 分支最新提交时间 | Corresponding commit times, one per line | | H | SSH Git链接 | ssh_url_to_repo | | I | 下载时间 | Clone/update timestamp | | J | Project ID | GitLab project ID (hidden column, used for matching) | **Sort:** A (asc) → B (asc) → D (asc) **Formatting:** Frozen header row, thin borders on all cells, F/G columns left-aligned with wrap, other columns center-aligned, UTF-8 encoding. ## Security - Token is passed via environment variable, never logged - After clone, remote URL is reset to remove embedded token - If clone times out or crashes, a cleanup step removes token from `.git/config` ## Output Structure ``` ~/Desktop/Code/ ├── 01.Index.xlsx ├── myGroup/ │ ├── SubGroup1/ │ │ ├── project-a/ │ │ └── project-b/ │ └── SubGroup2/ │ └── project-c/ └── AnotherGroup/ └── ... ```

path: scripts/clone_and_index.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
GitLab Batch Clone & Index Generator

Clones all projects from GitLab group(s), checks out key branches,
and generates an Excel index with per-group sheets.

Supports:
  - Top-level groups: "myGroup" → enumerate all subgroups & projects under myGroup
  - Sub-group paths:  "myGroup/mySubGroup" → directly sync mySubGroup (and its
    descendants) without enumerating the entire myGroup tree
  - Direct project paths: "myGroup/mySubGroup/my-project" → sync only that project
  - Graceful skip when a top-level group's subgroups/projects can't be fetched

Environment variables:
    GITLAB_URL       - GitLab instance URL (required)
    GITLAB_TOKEN     - Personal access token (required)
    GITLAB_GROUPS    - Comma-separated group/sub-group/project paths (required)
    GITLAB_BASE_DIR  - Local storage path (default: ~/Desktop/Code)
    GITLAB_MODE      - "clone" (default), "update" (skip clone, only pull),
                       or "sync" (clone+pull+cleanup stale Excel rows)
    GITLAB_WORKERS   - Number of parallel workers (default: 4)
"""

from __future__ import annotations

import os
import sys
import json
import signal
import subprocess
import urllib.request
import urllib.parse
import urllib.error
import ssl
import time
import threading
from datetime import datetime
from concurrent.futures import ProcessPoolExecutor, as_completed

# ---------------------------------------------------------------------------
# Dependency bootstrap
# ---------------------------------------------------------------------------
try:
    import openpyxl
    from openpyxl.styles import Font, Alignment, Border, Side
except ImportError:
    print("[setup] Installing openpyxl ...")
    subprocess.check_call([sys.executable, "-m", "pip", "install", "openpyxl", "-q"])
    import openpyxl
    from openpyxl.styles import Font, Alignment, Border, Side

# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
CLONE_TIMEOUT = 300       # 5 minutes per project
FETCH_TIMEOUT = 300       # 5 minutes for fetch/pull
GIT_CMD_TIMEOUT = 60      # 1 minute for lightweight git commands
API_PER_PAGE = 100
INCREMENTAL_EXCEL_BATCH = 50  # Write Excel every N completed projects
DEFAULT_TOTAL_TIMEOUT = 0     # 0 = no global timeout; override via GITLAB_TOTAL_TIMEOUT

HEADERS_CN = [
    "主Group名称", "子Group路径", "Project路径", "Project名称",
    "Project描述", "已checkout分支", "分支最新提交时间", "SSH Git链接", "下载时间",
    "Project ID",
]

NUM_COLUMNS = len(HEADERS_CN)  # 10

COL_WIDTHS = {"A": 15, "B": 35, "C": 45, "D": 30, "E": 40, "F": 35, "G": 25, "H": 60, "I": 20, "J": 12}


# ===========================================================================
# GitLab API helpers (using urllib – no extra dependency)
# ===========================================================================

def _make_ssl_ctx():
    """Create a permissive SSL context (many internal GitLabs use self-signed certs)."""
    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    return ctx

_SSL_CTX = _make_ssl_ctx()


def api_get(gitlab_url: str, token: str, path: str, params: dict | None = None) -> list | dict:
    """GET request to GitLab API. Returns parsed JSON."""
    url = f"{gitlab_url}/api/v4{path}"
    if params:
        url += "?" + urllib.parse.urlencode(params)
    req = urllib.request.Request(url, headers={"PRIVATE-TOKEN": token})
    try:
        with urllib.request.urlopen(req, context=_SSL_CTX, timeout=30) as resp:
            status = resp.status
            body = resp.read().decode("utf-8")
    except urllib.error.HTTPError as e:
        print(f"  [API error] {e.code} for {path}: {e.reason}")
        return []
    except Exception as e:
        print(f"  [API error] {path}: {e}")
        return []

    try:
        return json.loads(body)
    except json.JSONDecodeError:
        print(f"  [API error] Invalid JSON from {path}")
        return []


def api_get_paginated(gitlab_url: str, token: str, path: str, params: dict | None = None) -> list:
    """GET with automatic pagination. Returns flat list."""
    results = []
    page = 1
    base_params = dict(params or {})
    base_params["per_page"] = API_PER_PAGE
    while True:
        base_params["page"] = page
        data = api_get(gitlab_url, token, path, base_params)
        if not isinstance(data, list) or len(data) == 0:
            break
        results.extend(data)
        if len(data) < API_PER_PAGE:
            break
        page += 1
    return results


# ===========================================================================
# GitLab data fetching
# ===========================================================================

def fetch_subgroups_recursive(gitlab_url: str, token: str, group_id: int) -> list[dict]:
    """Recursively fetch all descendant subgroups of a group."""
    # Use the /groups/:id/descendant_groups endpoint if available (GitLab 13.5+)
    descendants = api_get_paginated(
        gitlab_url, token, f"/groups/{group_id}/descendant_groups",
        {"all_available": "false"},
    )
    if isinstance(descendants, list) and len(descendants) > 0:
        return descendants

    # Fallback: manual BFS via /groups/:id/subgroups
    all_subs = []
    queue = [group_id]
    visited = {group_id}
    while queue:
        gid = queue.pop(0)
        subs = api_get_paginated(gitlab_url, token, f"/groups/{gid}/subgroups")
        for s in subs:
            if s["id"] not in visited:
                visited.add(s["id"])
                all_subs.append(s)
                queue.append(s["id"])
    return all_subs


def resolve_group_id(gitlab_url: str, token: str, group_path: str) -> int | None:
    """Resolve a group path (e.g. 'myGroup') to its numeric ID."""
    encoded = urllib.parse.quote(group_path, safe="")
    data = api_get(gitlab_url, token, f"/groups/{encoded}")
    if isinstance(data, dict) and "id" in data:
        return data["id"]
    return None


def resolve_project(gitlab_url: str, token: str, project_path: str) -> dict | None:
    """Try to resolve a path as a project. Returns project dict or None."""
    encoded = urllib.parse.quote(project_path, safe="")
    data = api_get(gitlab_url, token, f"/projects/{encoded}")
    if isinstance(data, dict) and "id" in data:
        return data
    return None


def is_subpath(path: str) -> bool:
    """Check if a group input contains '/' — meaning the user specified a sub-path."""
    return "/" in path


def resolve_path_type(gitlab_url: str, token: str, path: str) -> tuple[str, dict | int | None]:
    """Resolve a user-provided path to either a project, group, or nothing.

    Returns:
        ("project", project_dict)  — path is a project
        ("group", group_id)        — path is a group/sub-group
        ("none", None)             — path could not be resolved

    Strategy: try group first (more common for paths with '/'), then project.
    This avoids a wasted 404 on /projects/ for sub-group paths like 'myGroup/mySubGroup'.
    """
    # Try as group first (sub-paths are more likely to be groups)
    gid = resolve_group_id(gitlab_url, token, path)
    if gid is not None:
        return ("group", gid)

    # Try as project
    proj = resolve_project(gitlab_url, token, path)
    if proj is not None:
        return ("project", proj)

    return ("none", None)


def fetch_all_projects(gitlab_url: str, token: str, group_ids: list[int]) -> list[dict]:
    """Fetch projects for a list of group IDs, deduplicated by project ID."""
    seen = set()
    projects = []
    for gid in group_ids:
        items = api_get_paginated(
            gitlab_url, token, f"/groups/{gid}/projects",
            {"include_subgroups": "false", "archived": "false"},
        )
        for p in items:
            if isinstance(p, dict) and "id" in p and p["id"] not in seen:
                seen.add(p["id"])
                projects.append(p)
    return projects


# ===========================================================================
# Git operations
# ===========================================================================

def _run_git(args: list[str], cwd: str, timeout: int = GIT_CMD_TIMEOUT) -> subprocess.CompletedProcess:
    """Run a git command with timeout.  Uses a new session so the entire
    process group can be killed on timeout (prevents orphan git-remote-https)."""
    proc = subprocess.Popen(
        ["git"] + args,
        cwd=cwd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True,
        start_new_session=True,
    )
    try:
        stdout, stderr = proc.communicate(timeout=timeout)
        return subprocess.CompletedProcess(proc.args, proc.returncode, stdout, stderr)
    except subprocess.TimeoutExpired:
        # Kill entire process group to avoid orphan children
        try:
            os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
        except OSError:
            proc.kill()
        proc.wait()
        raise


def clone_project(http_url: str, token: str, target_dir: str) -> None:
    """Clone a project via HTTPS with embedded token, then strip token from remote.
    Uses start_new_session + process-group kill to avoid orphan git processes."""
    url_with_token = http_url.replace("https://", f"https://oauth2:{token}@")
    proc = subprocess.Popen(
        ["git", "clone", "--quiet", url_with_token, target_dir],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True,
        start_new_session=True,
    )
    try:
        proc.communicate(timeout=CLONE_TIMEOUT)
    except subprocess.TimeoutExpired:
        try:
            os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
        except OSError:
            proc.kill()
        proc.wait()
        # Always try to strip token from remote (even if clone partially failed)
        _sanitize_remote(target_dir, http_url)
        raise
    # Always try to strip token from remote (even if clone partially failed)
    _sanitize_remote(target_dir, http_url)


def pull_project(target_dir: str) -> None:
    """Fetch all remotes and pull current branch."""
    _run_git(["fetch", "--all", "--quiet"], cwd=target_dir, timeout=FETCH_TIMEOUT)
    _run_git(["pull", "--quiet"], cwd=target_dir, timeout=FETCH_TIMEOUT)


def _sanitize_remote(target_dir: str, clean_url: str) -> None:
    """Reset origin URL to token-free version."""
    if os.path.isdir(os.path.join(target_dir, ".git")):
        _run_git(["remote", "set-url", "origin", clean_url], cwd=target_dir)


def get_default_branch(repo_path: str) -> str:
    """Detect the default branch from origin/HEAD or fall back heuristically."""
    # Try symbolic ref
    r = _run_git(["symbolic-ref", "refs/remotes/origin/HEAD", "--short"], cwd=repo_path)
    if r.returncode == 0 and r.stdout.strip():
        return r.stdout.strip().replace("origin/", "")

    # Fallback: current HEAD
    r = _run_git(["rev-parse", "--abbrev-ref", "HEAD"], cwd=repo_path)
    head = r.stdout.strip() if r.returncode == 0 else ""
    if head and head != "HEAD":
        return head

    # Last resort: check for master/main in remotes
    r = _run_git(["branch", "-r", "--list", "origin/master", "origin/main"], cwd=repo_path)
    for line in r.stdout.strip().splitlines():
        name = line.strip().replace("origin/", "")
        if name in ("master", "main"):
            return name
    return "master"


def get_remote_branches_sorted(repo_path: str) -> list[tuple[str, str]]:
    """Return remote branches sorted by committer date (newest first).
    Each item: (branch_name, 'YYYY-MM-DD HH:MM:SS')
    """
    r = _run_git(
        ["for-each-ref", "--sort=-committerdate", "refs/remotes/origin/",
         "--format=%(refname:short)\t%(committerdate:iso8601)"],
        cwd=repo_path,
    )
    if r.returncode != 0:
        return []

    result = []
    for line in r.stdout.strip().splitlines():
        line = line.strip()
        if not line:
            continue
        parts = line.split("\t", 1)
        if len(parts) < 2:
            continue
        ref = parts[0]
        ts = parts[1]
        if "HEAD" in ref:
            continue
        branch = ref.replace("origin/", "", 1)
        # Normalize timestamp: strip timezone offset
        ts_clean = ts.split(" +")[0].split(" -")[0]
        result.append((branch, ts_clean))
    return result


def _is_release_branch(name: str) -> bool:
    low = name.lower()
    return "release" in low or "prod" in low


def checkout_branches(repo_path: str) -> None:
    """Checkout: default branch + latest active + latest release/prod."""
    default = get_default_branch(repo_path)
    remotes = get_remote_branches_sorted(repo_path)
    if not remotes:
        return

    to_checkout = [default]

    # Latest active branch (first non-default in time-sorted list)
    for name, _ in remotes:
        if name != default:
            to_checkout.append(name)
            break

    # Latest release/prod branch
    for name, _ in remotes:
        if name in to_checkout:
            continue
        if _is_release_branch(name):
            to_checkout.append(name)
            break

    for branch in to_checkout:
        # Already exists locally?
        r = _run_git(["branch", "--list", branch], cwd=repo_path)
        if r.stdout.strip():
            # Fast-forward existing local branch to match remote (no checkout needed)
            _run_git(["fetch", "origin", f"{branch}:{branch}"], cwd=repo_path)
        else:
            # Create local tracking branch from remote
            _run_git(["checkout", "-b", branch, f"origin/{branch}"], cwd=repo_path)

    # Return to default
    _run_git(["checkout", default], cwd=repo_path)


def get_local_branches_info(repo_path: str) -> list[tuple[str, str]]:
    """Return (branch_name, commit_time) for every local branch."""
    r = _run_git(
        ["for-each-ref", "--format=%(refname:short)\t%(committerdate:iso8601)", "refs/heads/"],
        cwd=repo_path,
    )
    if r.returncode != 0:
        return []
    result = []
    for line in r.stdout.strip().splitlines():
        parts = line.strip().split("\t", 1)
        if len(parts) < 2:
            continue
        name = parts[0]
        ts = parts[1].split(" +")[0].split(" -")[0]
        result.append((name, ts))
    return result


# ===========================================================================
# Per-project worker (runs in subprocess via ProcessPoolExecutor)
# ===========================================================================

def process_one_project(
    project_info: dict,
    base_dir: str,
    token: str,
    mode: str,
) -> dict:
    """Clone/pull a single project and return its metadata dict.

    Returns a dict with project data or an 'error' key on failure.
    """
    project_path = project_info["path_with_namespace"]
    http_url = project_info["http_url_to_repo"]
    ssh_url = project_info.get("ssh_url_to_repo", "")
    description = project_info.get("description", "") or ""
    target_dir = os.path.join(base_dir, project_path)
    now_str = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

    is_existing = os.path.isdir(os.path.join(target_dir, ".git"))

    try:
        if is_existing:
            # Update mode or already cloned
            pull_project(target_dir)
        elif mode == "update":
            # Update-only mode but repo doesn't exist locally — skip
            return {"error": "skip", "project_path": project_path, "reason": "not cloned yet (update mode)"}
        else:
            os.makedirs(os.path.dirname(target_dir), exist_ok=True)
            clone_project(http_url, token, target_dir)

        if not os.path.isdir(os.path.join(target_dir, ".git")):
            return {"error": "clone_failed", "project_path": project_path, "reason": "no .git after clone"}

        # Ensure token is stripped
        _sanitize_remote(target_dir, http_url)

        # Checkout branches
        checkout_branches(target_dir)

        # Gather branch info
        branches = get_local_branches_info(target_dir)
        branches_str = "\n".join(b[0] for b in branches)
        times_str = "\n".join(b[1] for b in branches)

    except subprocess.TimeoutExpired:
        # Clean up token if partially cloned
        _sanitize_remote(target_dir, http_url)
        return {"error": "timeout", "project_path": project_path, "reason": "exceeded 5-min timeout"}
    except Exception as e:
        _sanitize_remote(target_dir, http_url)
        return {"error": "exception", "project_path": project_path, "reason": str(e)}

    # Compute group columns
    parts = project_path.split("/")
    main_group = parts[0] if parts else ""
    sub_group = "/".join(parts[:-1]) if len(parts) > 1 else main_group

    return {
        "main_group": main_group,
        "sub_group": sub_group,
        "project_path": project_path,
        "project_name": project_info["name"],
        "description": description,
        "branches": branches_str,
        "times": times_str,
        "ssh_url": ssh_url,
        "download_time": now_str,
        "project_id": project_info["id"],
    }


# ===========================================================================
# Excel generation
# ===========================================================================

def _item_to_row(item: dict) -> list:
    """Convert a project data dict to a row list matching HEADERS_CN."""
    return [
        item["main_group"],
        item["sub_group"],
        item["project_path"],
        item["project_name"],
        item["description"],
        item["branches"],
        item["times"],
        item["ssh_url"],
        item["download_time"],
        item.get("project_id", ""),
    ]


def build_excel(excel_path: str, projects_data: list[dict]) -> None:
    """Create or overwrite the Excel index file with per-group sheets."""
    if not projects_data:
        print("[excel] No project data to write, skipping Excel creation.")
        return

    wb = openpyxl.Workbook()
    # Remove default sheet
    if "Sheet" in wb.sheetnames:
        wb.remove(wb["Sheet"])

    # Group projects by main_group
    groups: dict[str, list[dict]] = {}
    for p in projects_data:
        mg = p["main_group"]
        groups.setdefault(mg, []).append(p)

    for group_name in sorted(groups.keys()):
        items = groups[group_name]
        # Sort: sub_group asc, project_name asc
        items.sort(key=lambda x: (x["sub_group"], x["project_name"]))

        ws = wb.create_sheet(title=group_name[:31])  # sheet name max 31 chars
        ws.append(HEADERS_CN)

        for item in items:
            ws.append(_item_to_row(item))

        _format_sheet(ws)

    wb.save(excel_path)
    print(f"\n[excel] Saved: {excel_path}")


def _find_sheet_for_group(wb, group_name: str):
    """Find the sheet belonging to *group_name* by scanning Column A data.

    This avoids relying solely on sheet title (which may be truncated to 31 chars).
    Returns the worksheet or None.
    """
    # Fast path: exact title match
    truncated = group_name[:31]
    if truncated in wb.sheetnames:
        return wb[truncated]

    # Slow path: scan Column A (主Group名称) of every sheet
    for ws in wb.worksheets:
        if ws.max_row >= 2:
            val = ws.cell(row=2, column=1).value
            if val == group_name:
                return ws
    return None


def _build_global_id_lookup(wb) -> dict[int, tuple[str, int]]:
    """Build a global lookup: project_id (int) → (sheet_title, row_idx).

    Scans Column J (Project ID) of every sheet.  Rows without a valid ID are
    skipped — they will be matched by path as a fallback.
    """
    lookup: dict[int, tuple[str, int]] = {}
    for ws in wb.worksheets:
        for row_idx in range(2, ws.max_row + 1):
            raw = ws.cell(row=row_idx, column=10).value  # Column J
            if raw is not None and raw != "":
                try:
                    pid = int(raw)
                    lookup[pid] = (ws.title, row_idx)
                except (ValueError, TypeError):
                    pass
    return lookup


def _build_path_lookup(ws) -> dict[str, int]:
    """Build path → row_idx lookup for a single sheet (Column C)."""
    lookup: dict[str, int] = {}
    for row_idx in range(2, ws.max_row + 1):
        pp = ws.cell(row=row_idx, column=3).value
        if pp:
            lookup[pp] = row_idx
    return lookup


def update_excel(
    excel_path: str,
    projects_data: list[dict],
    *,
    all_project_ids: set[int] | None = None,
    synced_groups: set[str] | None = None,
) -> None:
    """Update an existing Excel file with project data.

    Matching priority:
      1. Project ID (Column J) — handles renames and path changes
      2. Project path (Column C) — fallback for legacy rows without ID

    When *all_project_ids* is provided (sync mode, final write), rows in sheets
    belonging to *synced_groups* whose Project ID is not in the set are deleted.
    This cleans up projects that were deleted/archived on GitLab.
    """
    if not os.path.exists(excel_path):
        build_excel(excel_path, projects_data)
        return

    wb = openpyxl.load_workbook(excel_path)

    # --- Phase 1: Build global ID lookup (for cross-sheet migration) ---
    global_id_lookup = _build_global_id_lookup(wb)

    # Collect rows to delete from other sheets (cross-group migration)
    # Key: sheet_title → list of row indices
    rows_to_delete: dict[str, list[int]] = {}

    # --- Phase 2: Upsert incoming project data ---
    groups: dict[str, list[dict]] = {}
    for p in projects_data:
        mg = p["main_group"]
        groups.setdefault(mg, []).append(p)

    for group_name, items in groups.items():
        ws = _find_sheet_for_group(wb, group_name)

        if ws is not None:
            path_lookup = _build_path_lookup(ws)

            for item in items:
                row_data = _item_to_row(item)
                pid = item.get("project_id")
                matched_row = None

                # Try ID match first (global — may be in a different sheet)
                if pid is not None and int(pid) in global_id_lookup:
                    old_sheet_title, old_row = global_id_lookup[int(pid)]
                    if old_sheet_title == ws.title:
                        # Same sheet — update in place
                        matched_row = old_row
                    else:
                        # Cross-sheet migration: mark old row for deletion
                        rows_to_delete.setdefault(old_sheet_title, []).append(old_row)
                        # Will append as new row below

                # Fallback: path match within current sheet (for legacy rows without ID)
                if matched_row is None and item["project_path"] in path_lookup:
                    matched_row = path_lookup[item["project_path"]]

                if matched_row is not None:
                    for col_i, val in enumerate(row_data, 1):
                        ws.cell(row=matched_row, column=col_i, value=val)
                else:
                    ws.append(row_data)

            _resort_sheet(ws)
            _format_sheet(ws)
        else:
            # New group — create sheet, but still check for cross-sheet migrations
            for item in items:
                pid = item.get("project_id")
                if pid is not None and int(pid) in global_id_lookup:
                    old_sheet_title, old_row = global_id_lookup[int(pid)]
                    rows_to_delete.setdefault(old_sheet_title, []).append(old_row)

            items.sort(key=lambda x: (x["sub_group"], x["project_name"]))
            sheet_name = _unique_sheet_name(wb, group_name)
            ws = wb.create_sheet(title=sheet_name)
            ws.append(HEADERS_CN)
            for item in items:
                ws.append(_item_to_row(item))
            _format_sheet(ws)

    # --- Phase 3: Delete cross-sheet migrated rows (from back to front) ---
    for sheet_title, row_indices in rows_to_delete.items():
        if sheet_title in wb.sheetnames:
            target_ws = wb[sheet_title]
            for ridx in sorted(row_indices, reverse=True):
                if ridx <= target_ws.max_row:
                    target_ws.delete_rows(ridx)
            _resort_sheet(target_ws)
            _format_sheet(target_ws)

    # --- Phase 4: Cleanup stale rows (sync mode only) ---
    if all_project_ids is not None and synced_groups is not None:
        for ws in wb.worksheets:
            # Only clean sheets belonging to synced groups
            group_in_sheet = ws.cell(row=2, column=1).value if ws.max_row >= 2 else None
            if group_in_sheet not in synced_groups:
                continue

            stale_rows = []
            for row_idx in range(2, ws.max_row + 1):
                raw_id = ws.cell(row=row_idx, column=10).value  # Column J
                if raw_id is not None and raw_id != "":
                    try:
                        if int(raw_id) not in all_project_ids:
                            stale_rows.append(row_idx)
                    except (ValueError, TypeError):
                        pass
                else:
                    # Legacy row without ID — check if path is still in current data
                    pp = ws.cell(row=row_idx, column=3).value
                    if pp:
                        path_still_exists = any(
                            p.get("project_path") == pp for p in projects_data
                        )
                        if not path_still_exists:
                            stale_rows.append(row_idx)

            for ridx in sorted(stale_rows, reverse=True):
                ws.delete_rows(ridx)

            if stale_rows:
                print(f"  [excel] Removed {len(stale_rows)} stale rows from sheet '{ws.title}'")
                _resort_sheet(ws)
                _format_sheet(ws)

    wb.save(excel_path)
    print(f"\n[excel] Updated: {excel_path}")


def _unique_sheet_name(wb, group_name: str) -> str:
    """Generate a unique sheet name (max 31 chars) for a group."""
    name = group_name[:31]
    if name not in wb.sheetnames:
        return name
    for i in range(2, 100):
        suffix = f"_{i}"
        candidate = group_name[:31 - len(suffix)] + suffix
        if candidate not in wb.sheetnames:
            return candidate
    return name  # fallback (should never happen)


def _resort_sheet(ws) -> None:
    """Re-sort data rows (2..max_row) by sub_group(B), project_name(D)."""
    if ws.max_row < 3:
        return
    data = []
    for row in ws.iter_rows(min_row=2, max_row=ws.max_row, values_only=True):
        data.append(list(row))

    data.sort(key=lambda r: (r[1] or "", r[3] or ""))

    # Determine column count from data (handles both 9-col legacy and 10-col new)
    max_cols = max((len(r) for r in data), default=NUM_COLUMNS)
    col_count = max(max_cols, NUM_COLUMNS)

    # Clear data rows
    for row_idx in range(2, ws.max_row + 1):
        for col_idx in range(1, col_count + 1):
            ws.cell(row=row_idx, column=col_idx, value=None)

    # Write back
    for i, row_data in enumerate(data, 2):
        for ci, val in enumerate(row_data, 1):
            ws.cell(row=i, column=ci, value=val)


def _format_sheet(ws) -> None:
    """Apply formatting: widths, borders, alignment, freeze."""
    for col_letter, width in COL_WIDTHS.items():
        ws.column_dimensions[col_letter].width = width

    # Header style
    hdr_font = Font(bold=True, size=11)
    hdr_align = Alignment(horizontal="center", vertical="center", wrap_text=True)
    for cell in ws[1]:
        cell.font = hdr_font
        cell.alignment = hdr_align

    thin = Border(
        left=Side(style="thin"), right=Side(style="thin"),
        top=Side(style="thin"), bottom=Side(style="thin"),
    )
    center_wrap = Alignment(horizontal="center", vertical="center", wrap_text=True)
    left_wrap = Alignment(horizontal="left", vertical="center", wrap_text=True)

    for row in ws.iter_rows(min_row=1, max_row=ws.max_row, min_col=1, max_col=NUM_COLUMNS):
        for cell in row:
            cell.border = thin
            if cell.row == 1:
                continue
            if cell.column in (6, 7):  # F, G
                cell.alignment = left_wrap
            else:
                cell.alignment = center_wrap

    # Hide Project ID column (J) — internal use, not for end users
    ws.column_dimensions["J"].hidden = True

    ws.freeze_panes = "A2"


# ===========================================================================
# Main
# ===========================================================================

# ---------------------------------------------------------------------------
# Incremental Excel helper (thread-safe via lock)
# ---------------------------------------------------------------------------
_excel_lock = threading.Lock()


def _incremental_excel_write(
    excel_path: str,
    batch: list[dict],
    *,
    all_project_ids: set[int] | None = None,
    synced_groups: set[str] | None = None,
) -> None:
    """Thread-safe incremental write: merge *batch* into the Excel file.

    Pass *all_project_ids* and *synced_groups* only on the **final** write
    (sync mode) to trigger stale-row cleanup.  During incremental writes
    these should be None to avoid premature deletion.
    """
    has_cleanup = all_project_ids is not None
    if not batch and not has_cleanup:
        return
    with _excel_lock:
        if os.path.exists(excel_path):
            update_excel(
                excel_path, batch,
                all_project_ids=all_project_ids,
                synced_groups=synced_groups,
            )
        elif batch:
            build_excel(excel_path, batch)


# ---------------------------------------------------------------------------
# Signal-safe emergency Excel dump
# ---------------------------------------------------------------------------
_emergency_state: dict = {
    "excel_path": "",
    "pending": [],      # successes not yet written
}


def _emergency_excel_handler(signum, frame):
    """On SIGTERM / SIGINT, attempt to flush pending results to Excel."""
    pending = _emergency_state["pending"]
    excel_path = _emergency_state["excel_path"]
    if pending and excel_path:
        print(f"\n[signal] Caught signal {signum}, flushing {len(pending)} results to Excel ...")
        try:
            _incremental_excel_write(excel_path, list(pending))
            print(f"[signal] Emergency Excel write done: {excel_path}")
        except Exception as e:
            print(f"[signal] Emergency Excel write failed: {e}")
    sys.exit(1)


def main() -> None:
    gitlab_url = os.environ.get("GITLAB_URL", "").strip().rstrip("/")
    [REDACTED_SECRET]"GITLAB_TOKEN", "").strip()
    groups_input = os.environ.get("GITLAB_GROUPS", "").strip()
    base_dir = os.path.expanduser(os.environ.get("GITLAB_BASE_DIR", "~/Desktop/Code").strip())
    mode = os.environ.get("GITLAB_MODE", "clone").strip().lower()
    workers = int(os.environ.get("GITLAB_WORKERS", "4"))
    total_timeout = int(os.environ.get("GITLAB_TOTAL_TIMEOUT", str(DEFAULT_TOTAL_TIMEOUT)))

    if not all([gitlab_url, token, groups_input]):
        print("Error: GITLAB_URL, GITLAB_TOKEN, and GITLAB_GROUPS must be set.")
        sys.exit(1)

    target_groups = [g.strip() for g in groups_input.split(",") if g.strip()]
    os.makedirs(base_dir, exist_ok=True)

    excel_path = os.path.join(base_dir, "01.Index.xlsx")
    _emergency_state["excel_path"] = excel_path

    # Install signal handlers for emergency Excel dump
    signal.signal(signal.SIGTERM, _emergency_excel_handler)
    signal.signal(signal.SIGINT, _emergency_excel_handler)

    # -----------------------------------------------------------------------
    # 1. Resolve target groups/sub-groups/projects
    # -----------------------------------------------------------------------
    mode_label = {"update": "Update", "sync": "Sync", "clone": "Clone"}.get(mode, "Clone")
    is_sync = mode == "sync"
    # In sync mode, clone/pull behaves like "clone" mode, but final Excel
    # write also cleans up stale rows for synced groups.
    effective_mode = "clone" if is_sync else mode

    print(f"=== GitLab Batch {mode_label} ===")
    print(f"URL: {gitlab_url}")
    print(f"Groups: {', '.join(target_groups)}")
    print(f"Storage: {base_dir}")
    print(f"Workers: {workers}")
    if total_timeout > 0:
        print(f"Global timeout: {total_timeout}s")
    print()

    all_group_ids: list[int] = []
    direct_projects: list[dict] = []   # projects resolved directly by path

    for gpath in target_groups:
        print(f"[resolve] {gpath}")

        # --- Case A: User specified a sub-path (contains '/') ---
        if is_subpath(gpath):
            path_type, resolved = resolve_path_type(gitlab_url, token, gpath)

            if path_type == "project":
                # Direct project path — skip group enumeration entirely
                print(f"  → Resolved as PROJECT (id={resolved['id']})")
                direct_projects.append(resolved)
                continue

            elif path_type == "group":
                # Sub-group path — enumerate only this sub-group and its children
                gid = resolved
                print(f"  → Resolved as SUB-GROUP (id={gid}), fetching descendants ...")
                all_group_ids.append(gid)
                subs = fetch_subgroups_recursive(gitlab_url, token, gid)
                sub_ids = [s["id"] for s in subs]
                all_group_ids.extend(sub_ids)
                print(f"  Found {len(subs)} child subgroups ({len(sub_ids) + 1} total)")
                continue

            else:
                print(f"  ⚠ Could not resolve '{gpath}' as group or project, skipping")
                continue

        # --- Case B: Top-level group (no '/') ---
        gid = resolve_group_id(gitlab_url, token, gpath)
        if gid is None:
            print(f"  ⚠ Could not resolve group '{gpath}', skipping")
            continue

        all_group_ids.append(gid)
        print(f"  ID={gid}, fetching subgroups ...")
        subs = fetch_subgroups_recursive(gitlab_url, token, gid)
        if isinstance(subs, list):
            sub_ids = [s["id"] for s in subs]
        else:
            sub_ids = []

        # Check if we can actually access this group's data
        # Try fetching direct projects of this group as a canary
        canary = api_get_paginated(
            gitlab_url, token, f"/groups/{gid}/projects",
            {"include_subgroups": "false", "per_page": "1"},
        )
        if not isinstance(canary, list):
            # API returned non-list (error dict, etc.) — can't enumerate this group
            print(f"  ⚠ Cannot fetch subgroups/projects for '{gpath}' (access denied?), skipping")
            # Remove the group id we just added
            all_group_ids.pop()
            continue

        all_group_ids.extend(sub_ids)
        print(f"  Found {len(subs)} subgroups ({len(sub_ids) + 1} total groups)")

    if not all_group_ids and not direct_projects:
        print("No groups or projects found. Exiting.")
        sys.exit(1)

    # -----------------------------------------------------------------------
    # 2. Fetch all projects from groups + merge direct projects
    # -----------------------------------------------------------------------
    all_projects: list[dict] = []
    seen_ids: set[int] = set()

    # Add directly-resolved projects first
    for p in direct_projects:
        if p["id"] not in seen_ids:
            seen_ids.add(p["id"])
            all_projects.append(p)

    if all_group_ids:
        print(f"\n[projects] Fetching projects from {len(all_group_ids)} groups ...")
        group_projects = fetch_all_projects(gitlab_url, token, all_group_ids)
        for p in group_projects:
            if p["id"] not in seen_ids:
                seen_ids.add(p["id"])
                all_projects.append(p)

    print(f"[projects] Total unique projects: {len(all_projects)}")

    # Per-group project count breakdown
    group_count: dict[str, int] = {}
    for p in all_projects:
        top = p["path_with_namespace"].split("/")[0]
        group_count[top] = group_count.get(top, 0) + 1
    for gname, cnt in sorted(group_count.items()):
        print(f"  {gname}: {cnt} projects")

    if not all_projects:
        print("No projects found. Exiting.")
        sys.exit(0)

    # -----------------------------------------------------------------------
    # 3. Process projects (clone/pull + checkout + metadata)
    # -----------------------------------------------------------------------
    successes: list[dict] = []
    failures: list[dict] = []
    pending_batch: list[dict] = []   # buffer for incremental Excel writes
    total = len(all_projects)
    start_time = time.monotonic()

    print(f"\n[process] Starting with {workers} workers ...\n")

    timed_out_globally = False

    with ProcessPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(process_one_project, p, base_dir, token, effective_mode): p for p in all_projects}
        done_count = 0
        for future in as_completed(futures):
            done_count += 1
            proj = futures[future]
            elapsed = time.monotonic() - start_time
            try:
                result = future.result()
            except Exception as e:
                result = {"error": "worker_exception", "project_path": proj["path_with_namespace"], "reason": str(e)}

            tag = "✓" if "error" not in result else "✗"
            elapsed_str = f"{elapsed:.0f}s"
            print(f"  [{done_count}/{total}] {tag} {proj['path_with_namespace']}  ({elapsed_str})")

            if "error" in result:
                failures.append(result)
            else:
                successes.append(result)
                pending_batch.append(result)
                _emergency_state["pending"].append(result)

            # Incremental Excel write every INCREMENTAL_EXCEL_BATCH projects
            if len(pending_batch) >= INCREMENTAL_EXCEL_BATCH:
                print(f"  [excel] Incremental write ({len(successes)} projects so far) ...")
                _incremental_excel_write(excel_path, pending_batch)
                # Clear pending from both buffers (slice — O(n), no dict equality issues)
                _emergency_state["pending"] = _emergency_state["pending"][len(pending_batch):]
                pending_batch = []

            # Check global timeout
            if total_timeout > 0 and elapsed > total_timeout:
                print(f"\n⚠ Global timeout ({total_timeout}s) exceeded after {done_count}/{total} projects.")
                timed_out_globally = True
                # Cancel remaining futures
                for f in futures:
                    f.cancel()
                break

    # -----------------------------------------------------------------------
    # 4. Flush remaining results to Excel
    # -----------------------------------------------------------------------
    # In sync mode, the final write also cleans up stale rows.
    # Compute the full set of project IDs we fetched from GitLab (not just successes).
    sync_all_ids: set[int] | None = None
    sync_groups: set[str] | None = None
    if is_sync:
        sync_all_ids = {p["id"] for p in all_projects}
        sync_groups = {p["path_with_namespace"].split("/")[0] for p in all_projects}

    if pending_batch or is_sync:
        print(f"  [excel] Final write ({len(successes)} total projects) ...")
        _incremental_excel_write(
            excel_path, pending_batch,
            all_project_ids=sync_all_ids,
            synced_groups=sync_groups,
        )
        _emergency_state["pending"] = []
        pending_batch = []

    # -----------------------------------------------------------------------
    # 5. Summary
    # -----------------------------------------------------------------------
    total_elapsed = time.monotonic() - start_time
    elapsed_min = total_elapsed / 60

    print(f"\n{'=' * 60}")
    print(f"  Done!  Success: {len(successes)}  |  Failed/Timeout: {len(failures)}")
    if timed_out_globally:
        remaining = total - len(successes) - len(failures)
        print(f"  ⚠ Cancelled (global timeout): ~{remaining} projects")
    print(f"  Time: {elapsed_min:.1f} min")
    print(f"  Projects: {base_dir}")
    print(f"  Excel:    {excel_path}")
    print(f"{'=' * 60}")

    if failures:
        print(f"\n⚠ Failed / Timed-out projects ({len(failures)}):\n")
        print(f"  {'Project Path':<55} {'Reason'}")
        print(f"  {'-'*55} {'-'*30}")
        for f in failures:
            pp = f.get("project_path", "?")
            reason = f.get("reason", "unknown")
            print(f"  {pp:<55} {reason}")
        print()


if __name__ == "__main__":
    main()


path: scripts/test_clone_and_index.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Tests for clone_and_index.py

Covers:
  1. Original functionality (top-level group resolution, subgroup enumeration, project fetch)
  2. Graceful skip when a main group can't be accessed
  3. Sub-path resolution: sub-group paths and direct project paths
  4. Excel generation and update logic
  5. Edge cases: mixed inputs, deduplication, empty results
"""

from __future__ import annotations
import os
import sys
import json
import tempfile
import shutil
import subprocess
import unittest
from unittest.mock import patch, MagicMock
from datetime import datetime

# Ensure the script directory is on the path
sys.path.insert(0, os.path.dirname(__file__))

import clone_and_index as ci


# ===========================================================================
# Mock data factories
# ===========================================================================

def make_group(gid: int, name: str, full_path: str) -> dict:
    return {"id": gid, "name": name, "full_path": full_path, "path": name}

def make_project(pid: int, name: str, namespace_path: str, description: str = "") -> dict:
    full_path = f"{namespace_path}/{name}"
    return {
        "id": pid,
        "name": name,
        "path_with_namespace": full_path,
        "http_url_to_repo": f"https://gitlab.test.com/{full_path}.git",
        "ssh_url_to_repo": f"[REDACTED_SECRET]:{full_path}.git",
        "description": description,
    }


# ===========================================================================
# Test: Helper functions
# ===========================================================================

class TestHelpers(unittest.TestCase):
    """Test is_subpath and resolve_path_type logic."""

    def test_is_subpath_simple(self):
        self.assertFalse(ci.is_subpath("myGroup"))
        self.assertFalse(ci.is_subpath("myGroup"))

    def test_is_subpath_with_slash(self):
        self.assertTrue(ci.is_subpath("myGroup/mySubGroup"))
        self.assertTrue(ci.is_subpath("a/b/c"))
        self.assertTrue(ci.is_subpath("myGroup/sub"))

    @patch("clone_and_index.resolve_project")
    @patch("clone_and_index.resolve_group_id")
    def test_resolve_path_type_project(self, mock_group, mock_project):
        """When path resolves as a project (not a group), return ('project', dict)."""
        proj = make_project(100, "my-proj", "myGroup/mySubGroup")
        mock_group.return_value = None  # not a group
        mock_project.return_value = proj

        kind, data = ci.resolve_path_type("https://gl.test", "tok", "myGroup/mySubGroup/my-proj")
        self.assertEqual(kind, "project")
        self.assertEqual(data["id"], 100)
        # Group is tried first, then project
        mock_group.assert_called_once()
        mock_project.assert_called_once()

    @patch("clone_and_index.resolve_project")
    @patch("clone_and_index.resolve_group_id")
    def test_resolve_path_type_group(self, mock_group, mock_project):
        """When path resolves as a group, return ('group', id) — project not tried."""
        mock_group.return_value = 42
        mock_project.return_value = None  # should not be called

        kind, data = ci.resolve_path_type("https://gl.test", "tok", "myGroup/mySubGroup")
        self.assertEqual(kind, "group")
        self.assertEqual(data, 42)
        # Project should NOT be tried since group resolved
        mock_project.assert_not_called()

    @patch("clone_and_index.resolve_project")
    @patch("clone_and_index.resolve_group_id")
    def test_resolve_path_type_none(self, mock_group, mock_project):
        """When path doesn't resolve at all, return ('none', None)."""
        mock_group.return_value = None
        mock_project.return_value = None

        kind, data = ci.resolve_path_type("https://gl.test", "tok", "nonexistent/path")
        self.assertEqual(kind, "none")
        self.assertIsNone(data)


# ===========================================================================
# Test: API error handling / graceful skip
# ===========================================================================

class TestGracefulSkip(unittest.TestCase):
    """Test that inaccessible groups are skipped gracefully."""

    @patch("clone_and_index.api_get")
    def test_resolve_group_id_returns_none_on_error(self, mock_api):
        """When API returns an error list, resolve_group_id should return None."""
        mock_api.return_value = []  # API error returns empty list
        result = ci.resolve_group_id("https://gl.test", "tok", "forbidden-group")
        self.assertIsNone(result)

    @patch("clone_and_index.api_get")
    def test_resolve_project_returns_none_on_error(self, mock_api):
        mock_api.return_value = []
        result = ci.resolve_project("https://gl.test", "tok", "nonexistent/proj")
        self.assertIsNone(result)

    @patch("clone_and_index.api_get_paginated")
    def test_fetch_subgroups_recursive_empty_on_failure(self, mock_pag):
        """When both descendant and subgroup APIs return empty, we get []."""
        mock_pag.return_value = []
        result = ci.fetch_subgroups_recursive("https://gl.test", "tok", 999)
        self.assertEqual(result, [])


# ===========================================================================
# Test: Excel generation
# ===========================================================================

class TestExcel(unittest.TestCase):
    """Test Excel build and update logic."""

    def setUp(self):
        self.tmpdir = tempfile.mkdtemp()
        self.excel_path = os.path.join(self.tmpdir, "01.索引.xlsx")

    def tearDown(self):
        shutil.rmtree(self.tmpdir, ignore_errors=True)

    def _make_project_data(self, main_group, sub_group, path, name, desc=""):
        return {
            "main_group": main_group,
            "sub_group": sub_group,
            "project_path": path,
            "project_name": name,
            "description": desc,
            "branches": "master\ndev",
            "times": "2026-01-01 10:00:00\n2026-01-02 11:00:00",
            "ssh_url": f"[REDACTED_SECRET]:{path}.git",
            "download_time": "2026-03-11 09:00:00",
        }

    def test_build_excel_creates_file(self):
        data = [
            self._make_project_data("myGroup", "myGroup/sub1", "myGroup/sub1/proj-a", "proj-a"),
            self._make_project_data("myGroup", "myGroup/sub2", "myGroup/sub2/proj-b", "proj-b"),
        ]
        ci.build_excel(self.excel_path, data)
        self.assertTrue(os.path.exists(self.excel_path))

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        self.assertIn("myGroup", wb.sheetnames)
        ws = wb["myGroup"]
        # Header + 2 data rows
        self.assertEqual(ws.max_row, 3)
        # Check project paths
        self.assertEqual(ws.cell(row=2, column=3).value, "myGroup/sub1/proj-a")
        self.assertEqual(ws.cell(row=3, column=3).value, "myGroup/sub2/proj-b")

    def test_build_excel_multiple_groups(self):
        data = [
            self._make_project_data("myGroup", "myGroup", "myGroup/proj-1", "proj-1"),
            self._make_project_data("myGroup", "myGroup/mySubGroup", "myGroup/mySubGroup/map-svc", "map-svc"),
        ]
        ci.build_excel(self.excel_path, data)

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        self.assertIn("myGroup", wb.sheetnames)
        self.assertIn("myGroup", wb.sheetnames)

    def test_update_excel_adds_new_rows(self):
        # Build initial
        data1 = [self._make_project_data("myGroup", "myGroup/sub1", "myGroup/sub1/proj-a", "proj-a")]
        ci.build_excel(self.excel_path, data1)

        # Update with new project
        data2 = [self._make_project_data("myGroup", "myGroup/sub2", "myGroup/sub2/proj-b", "proj-b")]
        ci.update_excel(self.excel_path, data2)

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        ws = wb["myGroup"]
        # Header + 2 data rows
        self.assertEqual(ws.max_row, 3)

    def test_update_excel_updates_existing_rows(self):
        data1 = [self._make_project_data("myGroup", "myGroup/sub1", "myGroup/sub1/proj-a", "proj-a", "old desc")]
        ci.build_excel(self.excel_path, data1)

        data2 = [self._make_project_data("myGroup", "myGroup/sub1", "myGroup/sub1/proj-a", "proj-a", "new desc")]
        ci.update_excel(self.excel_path, data2)

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        ws = wb["myGroup"]
        self.assertEqual(ws.max_row, 2)  # header + 1 row (updated, not duplicated)
        self.assertEqual(ws.cell(row=2, column=5).value, "new desc")

    def test_update_excel_creates_new_sheet_for_new_group(self):
        data1 = [self._make_project_data("myGroup", "myGroup", "myGroup/proj-1", "proj-1")]
        ci.build_excel(self.excel_path, data1)

        data2 = [self._make_project_data("myGroup", "myGroup/mySubGroup", "myGroup/mySubGroup/map-svc", "map-svc")]
        ci.update_excel(self.excel_path, data2)

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        self.assertIn("myGroup", wb.sheetnames)
        self.assertIn("myGroup", wb.sheetnames)

    def test_build_excel_empty_data(self):
        """Empty project list should not create a file (graceful skip)."""
        ci.build_excel(self.excel_path, [])
        # No file created since there's nothing to write
        self.assertFalse(os.path.exists(self.excel_path))


# ===========================================================================
# Test: fetch_all_projects deduplication
# ===========================================================================

class TestFetchAllProjects(unittest.TestCase):
    """Test that fetch_all_projects deduplicates by project ID."""

    @patch("clone_and_index.api_get_paginated")
    def test_deduplication(self, mock_pag):
        proj_a = make_project(1, "proj-a", "G/sub1")
        proj_b = make_project(2, "proj-b", "G/sub2")
        proj_a_dup = make_project(1, "proj-a", "G/sub1")  # same id

        # Group 10 returns [a, b], group 20 returns [a_dup]
        mock_pag.side_effect = [
            [proj_a, proj_b],
            [proj_a_dup],
        ]

        result = ci.fetch_all_projects("https://gl.test", "tok", [10, 20])
        self.assertEqual(len(result), 2)
        ids = {p["id"] for p in result}
        self.assertEqual(ids, {1, 2})


# ===========================================================================
# Test: End-to-end main() logic with mocks
# ===========================================================================

class TestMainLogic(unittest.TestCase):
    """Integration tests for main() using mocked API and git operations."""

    def setUp(self):
        self.tmpdir = tempfile.mkdtemp()

    def tearDown(self):
        shutil.rmtree(self.tmpdir, ignore_errors=True)

    @patch("clone_and_index.fetch_all_projects")
    @patch("clone_and_index.fetch_subgroups_recursive")
    @patch("clone_and_index.api_get_paginated")
    @patch("clone_and_index.resolve_group_id")
    @patch("clone_and_index.resolve_path_type")
    def test_mixed_inputs_resolution_phase(self, mock_resolve_path, mock_resolve_group,
                                            mock_pag, mock_subs, mock_fetch):
        """Test the resolution phase with mixed inputs: top-level group + sub-group + project.

        We verify that the correct group IDs and direct projects are collected.
        We don't test the clone/pull phase here (ProcessPoolExecutor + mocks don't mix).
        """
        gitlab_url = "https://gl.test"
        [REDACTED_SECRET]"

        proj_direct = make_project(99, "my-proj", "myGroup/mySubGroup")

        # Simulate the resolution loop from main()
        target_groups = ["myGroup", "myGroup/mySubGroup", "myGroup/mySubGroup/my-proj"]

        mock_resolve_group.return_value = 1
        mock_subs.return_value = [make_group(2, "sub1", "myGroup/sub1")]
        mock_pag.return_value = [make_project(10, "my-proj", "myGroup")]

        def resolve_side_effect(url, tok, path):
            if path == "myGroup/mySubGroup":
                return ("group", 5)
            elif path == "myGroup/mySubGroup/my-proj":
                return ("project", proj_direct)
            return ("none", None)
        mock_resolve_path.side_effect = resolve_side_effect

        # Reproduce the resolution loop
        all_group_ids = []
        direct_projects = []

        for gpath in target_groups:
            if ci.is_subpath(gpath):
                path_type, resolved = ci.resolve_path_type(gitlab_url, token, gpath)
                if path_type == "project":
                    direct_projects.append(resolved)
                elif path_type == "group":
                    all_group_ids.append(resolved)
                    subs = ci.fetch_subgroups_recursive(gitlab_url, token, resolved)
                    all_group_ids.extend([s["id"] for s in subs])
            else:
                gid = ci.resolve_group_id(gitlab_url, token, gpath)
                if gid is not None:
                    all_group_ids.append(gid)
                    subs = ci.fetch_subgroups_recursive(gitlab_url, token, gid)
                    all_group_ids.extend([s["id"] for s in subs])

        # Verify: myGroup (id=1) + sub1 (id=2) + myGroup/mySubGroup (id=5) + its sub (id=2 again, but from mock)
        self.assertIn(1, all_group_ids)   # myGroup
        self.assertIn(5, all_group_ids)   # myGroup/mySubGroup (sub-group)
        self.assertEqual(len(direct_projects), 1)
        self.assertEqual(direct_projects[0]["id"], 99)  # my-proj

        # Verify dedup when merging
        mock_fetch.return_value = [
            make_project(10, "myGroup-proj", "myGroup"),
            make_project(20, "mySubGroup-proj", "myGroup/mySubGroup"),
        ]

        all_projects = []
        seen_ids = set()
        for p in direct_projects:
            if p["id"] not in seen_ids:
                seen_ids.add(p["id"])
                all_projects.append(p)
        group_projects = ci.fetch_all_projects(gitlab_url, token, all_group_ids)
        for p in group_projects:
            if p["id"] not in seen_ids:
                seen_ids.add(p["id"])
                all_projects.append(p)

        # Should have 3 unique projects: 99, 10, 20
        self.assertEqual(len(all_projects), 3)
        self.assertEqual({p["id"] for p in all_projects}, {99, 10, 20})

    @patch("clone_and_index.resolve_group_id")
    def test_inaccessible_group_skipped(self, mock_resolve):
        """When a top-level group can't be resolved, it should be skipped."""
        mock_resolve.return_value = None

        os.environ["GITLAB_URL"] = "https://gl.test"
        os.environ["GITLAB_TOKEN"] = "test-token"
        os.environ["GITLAB_GROUPS"] = "forbidden-group"
        os.environ["GITLAB_BASE_DIR"] = self.tmpdir

        with self.assertRaises(SystemExit) as ctx:
            ci.main()
        # Should exit with code 1 (no groups found)
        self.assertEqual(ctx.exception.code, 1)

        for key in ["GITLAB_URL", "GITLAB_TOKEN", "GITLAB_GROUPS", "GITLAB_BASE_DIR"]:
            os.environ.pop(key, None)

    @patch("clone_and_index.resolve_path_type")
    def test_unresolvable_subpath_skipped(self, mock_resolve_path):
        """When a sub-path can't be resolved as group or project, skip it."""
        mock_resolve_path.return_value = ("none", None)

        os.environ["GITLAB_URL"] = "https://gl.test"
        os.environ["GITLAB_TOKEN"] = "test-token"
        os.environ["GITLAB_GROUPS"] = "nonexistent/subpath"
        os.environ["GITLAB_BASE_DIR"] = self.tmpdir

        with self.assertRaises(SystemExit) as ctx:
            ci.main()
        self.assertEqual(ctx.exception.code, 1)

        for key in ["GITLAB_URL", "GITLAB_TOKEN", "GITLAB_GROUPS", "GITLAB_BASE_DIR"]:
            os.environ.pop(key, None)


# ===========================================================================
# Test: _is_release_branch
# ===========================================================================

class TestBranchClassification(unittest.TestCase):

    def test_release_branch_detection(self):
        self.assertTrue(ci._is_release_branch("release/1.0"))
        self.assertTrue(ci._is_release_branch("Release-v2"))
        self.assertTrue(ci._is_release_branch("prod"))
        self.assertTrue(ci._is_release_branch("production"))
        self.assertFalse(ci._is_release_branch("master"))
        self.assertFalse(ci._is_release_branch("dev"))
        self.assertFalse(ci._is_release_branch("feature/new-thing"))


# ===========================================================================
# Test: resort_sheet
# ===========================================================================

# ===========================================================================
# Test: Incremental Excel write
# ===========================================================================

class TestIncrementalExcel(unittest.TestCase):
    """Test the _incremental_excel_write helper."""

    def setUp(self):
        self.tmpdir = tempfile.mkdtemp()
        self.excel_path = os.path.join(self.tmpdir, "01.索引.xlsx")

    def tearDown(self):
        shutil.rmtree(self.tmpdir, ignore_errors=True)

    def _make_project_data(self, main_group, sub_group, path, name, desc=""):
        return {
            "main_group": main_group,
            "sub_group": sub_group,
            "project_path": path,
            "project_name": name,
            "description": desc,
            "branches": "master",
            "times": "2026-01-01 10:00:00",
            "ssh_url": f"[REDACTED_SECRET]:{path}.git",
            "download_time": "2026-03-11 09:00:00",
        }

    def test_incremental_creates_then_updates(self):
        """First call creates Excel, second call updates it."""
        batch1 = [self._make_project_data("myGroup", "myGroup/sub1", "myGroup/sub1/proj-a", "proj-a")]
        ci._incremental_excel_write(self.excel_path, batch1)
        self.assertTrue(os.path.exists(self.excel_path))

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        self.assertEqual(wb["myGroup"].max_row, 2)  # header + 1

        batch2 = [self._make_project_data("myGroup", "myGroup/sub2", "myGroup/sub2/proj-b", "proj-b")]
        ci._incremental_excel_write(self.excel_path, batch2)

        wb = openpyxl.load_workbook(self.excel_path)
        self.assertEqual(wb["myGroup"].max_row, 3)  # header + 2

    def test_incremental_empty_batch_noop(self):
        """Empty batch should not create a file."""
        ci._incremental_excel_write(self.excel_path, [])
        self.assertFalse(os.path.exists(self.excel_path))


# ===========================================================================
# Test: _run_git uses start_new_session
# ===========================================================================

class TestRunGitTimeout(unittest.TestCase):
    """Test that _run_git properly raises TimeoutExpired and cleans up."""

    def test_run_git_timeout_raises(self):
        """A command that sleeps longer than timeout should raise TimeoutExpired."""
        tmpdir = tempfile.mkdtemp()
        try:
            # Use a git command that will hang: init a repo first, then run a slow command
            subprocess.run(["git", "init", tmpdir], capture_output=True)
            with self.assertRaises(subprocess.TimeoutExpired):
                # 'git log' on empty repo is fast, so use a sleep trick via GIT_TRACE
                # Instead, just call _run_git with a very short timeout on a command that takes time
                ci._run_git(
                    ["-c", "core.pager=sleep 10", "log"],
                    cwd=tmpdir,
                    timeout=1,
                )
        except Exception:
            pass  # Some environments may not have git; skip gracefully
        finally:
            shutil.rmtree(tmpdir, ignore_errors=True)


# ===========================================================================
# Test: _resort_sheet
# ===========================================================================

class TestResortSheet(unittest.TestCase):

    def test_resort_sorts_by_subgroup_then_name(self):
        import openpyxl
        wb = openpyxl.Workbook()
        ws = wb.active
        ws.append(ci.HEADERS_CN)
        # Insert out of order
        ws.append(["G", "G/z", "G/z/proj-z", "proj-z", "", "", "", "", ""])
        ws.append(["G", "G/a", "G/a/proj-a", "proj-a", "", "", "", "", ""])
        ws.append(["G", "G/a", "G/a/proj-b", "proj-b", "", "", "", "", ""])

        ci._resort_sheet(ws)

        # Row 2 should be G/a / proj-a
        self.assertEqual(ws.cell(row=2, column=2).value, "G/a")
        self.assertEqual(ws.cell(row=2, column=4).value, "proj-a")
        # Row 3 should be G/a / proj-b
        self.assertEqual(ws.cell(row=3, column=4).value, "proj-b")
        # Row 4 should be G/z / proj-z
        self.assertEqual(ws.cell(row=4, column=2).value, "G/z")


# ===========================================================================
# Test: Project ID matching, cross-sheet migration, stale row cleanup
# ===========================================================================

class TestExcelProjectID(unittest.TestCase):
    """Test new Project ID based matching and cleanup logic."""

    def setUp(self):
        self.tmpdir = tempfile.mkdtemp()
        self.excel_path = os.path.join(self.tmpdir, "01.索引.xlsx")

    def tearDown(self):
        shutil.rmtree(self.tmpdir, ignore_errors=True)

    def _make_project_data(self, main_group, sub_group, path, name, pid, desc=""):
        return {
            "main_group": main_group,
            "sub_group": sub_group,
            "project_path": path,
            "project_name": name,
            "description": desc,
            "branches": "master",
            "times": "2026-01-01 10:00:00",
            "ssh_url": f"[REDACTED_SECRET]:{path}.git",
            "download_time": "2026-03-11 09:00:00",
            "project_id": pid,
        }

    def test_project_id_column_written(self):
        """Project ID should be written to Column J."""
        data = [self._make_project_data("myGroup", "myGroup/sub", "myGroup/sub/proj", "proj", 100)]
        ci.build_excel(self.excel_path, data)

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        ws = wb["myGroup"]
        # Header
        self.assertEqual(ws.cell(row=1, column=10).value, "Project ID")
        # Data
        self.assertEqual(ws.cell(row=2, column=10).value, 100)

    def test_update_matches_by_id_when_path_changes(self):
        """When a project is renamed (path changes but ID stays), update the existing row."""
        data1 = [self._make_project_data("myGroup", "myGroup/old", "myGroup/old/proj", "proj", 100)]
        ci.build_excel(self.excel_path, data1)

        # Same ID, different path (project was moved within same group)
        data2 = [self._make_project_data("myGroup", "myGroup/new", "myGroup/new/proj", "proj", 100)]
        ci.update_excel(self.excel_path, data2)

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        ws = wb["myGroup"]
        # Should have 1 data row (updated), not 2
        self.assertEqual(ws.max_row, 2)
        self.assertEqual(ws.cell(row=2, column=3).value, "myGroup/new/proj")
        self.assertEqual(ws.cell(row=2, column=10).value, 100)

    def test_cross_sheet_migration(self):
        """When a project moves from one group to another, old row is deleted, new row created."""
        data1 = [
            self._make_project_data("myGroup", "myGroup", "myGroup/proj-a", "proj-a", 100),
            self._make_project_data("Platform", "Platform", "Platform/proj-b", "proj-b", 200),
        ]
        ci.build_excel(self.excel_path, data1)

        # proj-a (ID 100) moves from myGroup to Platform
        data2 = [self._make_project_data("Platform", "Platform", "Platform/proj-a", "proj-a", 100)]
        ci.update_excel(self.excel_path, data2)

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)

        # myGroup sheet should have 0 data rows (original proj-a removed)
        ws_myGroup = wb["myGroup"]
        self.assertEqual(ws_myGroup.max_row, 1)  # only header

        # Platform sheet should have 2 data rows: proj-b + migrated proj-a
        ws_plat = wb["Platform"]
        self.assertEqual(ws_plat.max_row, 3)  # header + 2 rows
        paths = {ws_plat.cell(row=r, column=3).value for r in range(2, 4)}
        self.assertIn("Platform/proj-a", paths)
        self.assertIn("Platform/proj-b", paths)

    def test_cross_sheet_migration_multiple(self):
        """Multiple projects migrating from same sheet — row deletion order is correct."""
        data1 = [
            self._make_project_data("Old", "Old", "Old/p1", "p1", 100),
            self._make_project_data("Old", "Old", "Old/p2", "p2", 200),
            self._make_project_data("Old", "Old", "Old/p3", "p3", 300),
        ]
        ci.build_excel(self.excel_path, data1)

        # All three move to New group
        data2 = [
            self._make_project_data("New", "New", "New/p1", "p1", 100),
            self._make_project_data("New", "New", "New/p2", "p2", 200),
            self._make_project_data("New", "New", "New/p3", "p3", 300),
        ]
        ci.update_excel(self.excel_path, data2)

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)

        ws_old = wb["Old"]
        self.assertEqual(ws_old.max_row, 1)  # only header

        ws_new = wb["New"]
        self.assertEqual(ws_new.max_row, 4)  # header + 3 rows

    def test_sync_mode_removes_stale_rows(self):
        """In sync mode, projects not in all_project_ids are removed."""
        data1 = [
            self._make_project_data("myGroup", "myGroup", "myGroup/alive", "alive", 100),
            self._make_project_data("myGroup", "myGroup", "myGroup/deleted", "deleted", 200),
        ]
        ci.build_excel(self.excel_path, data1)

        # Only project 100 exists now; project 200 was deleted on GitLab
        data2 = [self._make_project_data("myGroup", "myGroup", "myGroup/alive", "alive", 100)]
        ci.update_excel(
            self.excel_path, data2,
            all_project_ids={100},
            synced_groups={"myGroup"},
        )

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        ws = wb["myGroup"]
        self.assertEqual(ws.max_row, 2)  # header + 1 (alive)
        self.assertEqual(ws.cell(row=2, column=3).value, "myGroup/alive")

    def test_sync_mode_does_not_touch_unsynced_groups(self):
        """Sync cleanup only affects sheets belonging to synced groups."""
        data1 = [
            self._make_project_data("myGroup", "myGroup", "myGroup/proj", "proj", 100),
            self._make_project_data("Other", "Other", "Other/proj", "proj", 200),
        ]
        ci.build_excel(self.excel_path, data1)

        # Sync only myGroup; "Other" should remain untouched
        data2 = [self._make_project_data("myGroup", "myGroup", "myGroup/proj", "proj", 100)]
        ci.update_excel(
            self.excel_path, data2,
            all_project_ids={100},
            synced_groups={"myGroup"},
        )

        import openpyxl
        wb = openpyxl.load_workbook(self.excel_path)
        ws_other = wb["Other"]
        self.assertEqual(ws_other.max_row, 2)  # header + 1 row (untouched)
        self.assertEqual(ws_other.cell(row=2, column=10).value, 200)

    def test_legacy_excel_without_id_column(self):
        """Opening a legacy 9-column Excel should not crash; path fallback should work."""
        import openpyxl
        from openpyxl.styles import Font, Alignment

        # Build a legacy Excel with 9 columns (no Project ID)
        wb = openpyxl.Workbook()
        wb.remove(wb["Sheet"])
        ws = wb.create_sheet("myGroup")
        old_headers = ci.HEADERS_CN[:9]  # first 9 only
        ws.append(old_headers)
        ws.append(["myGroup", "myGroup/sub", "myGroup/sub/proj", "proj", "desc",
                    "master", "2026-01-01", "[REDACTED_SECRET]:myGroup/sub/proj.git", "2026-03-01"])
        wb.save(self.excel_path)

        # Now update with data that has a project_id
        data = [self._make_project_data("myGroup", "myGroup/sub", "myGroup/sub/proj", "proj", 100, "new desc")]
        ci.update_excel(self.excel_path, data)

        wb = openpyxl.load_workbook(self.excel_path)
        ws = wb["myGroup"]
        # Should update existing row (path match fallback) and write project_id
        self.assertEqual(ws.max_row, 2)
        self.assertEqual(ws.cell(row=2, column=5).value, "new desc")
        self.assertEqual(ws.cell(row=2, column=10).value, 100)

    def test_legacy_excel_sync_cleanup_by_path(self):
        """Legacy rows without ID get cleaned up by path match in sync mode."""
        import openpyxl

        # Build legacy Excel without Project ID
        wb = openpyxl.Workbook()
        wb.remove(wb["Sheet"])
        ws = wb.create_sheet("myGroup")
        ws.append(ci.HEADERS_CN[:9])
        ws.append(["myGroup", "myGroup", "myGroup/alive", "alive", "", "master", "2026-01-01", "git@x", "2026-03-01"])
        ws.append(["myGroup", "myGroup", "myGroup/dead", "dead", "", "master", "2026-01-01", "git@x", "2026-03-01"])
        wb.save(self.excel_path)

        # Sync: only "alive" exists
        data = [self._make_project_data("myGroup", "myGroup", "myGroup/alive", "alive", 100)]
        ci.update_excel(
            self.excel_path, data,
            all_project_ids={100},
            synced_groups={"myGroup"},
        )

        wb = openpyxl.load_workbook(self.excel_path)
        ws = wb["myGroup"]
        self.assertEqual(ws.max_row, 2)  # header + 1
        self.assertEqual(ws.cell(row=2, column=3).value, "myGroup/alive")


# ===========================================================================
# Test: _find_sheet_for_group
# ===========================================================================

class TestFindSheetForGroup(unittest.TestCase):
    """Test sheet lookup by Column A fallback."""

    def test_exact_title_match(self):
        import openpyxl
        wb = openpyxl.Workbook()
        wb.remove(wb["Sheet"])
        ws = wb.create_sheet("myGroup")
        ws.append(ci.HEADERS_CN)
        ws.append(["myGroup", "", "", "", "", "", "", "", "", ""])

        result = ci._find_sheet_for_group(wb, "myGroup")
        self.assertEqual(result.title, "myGroup")

    def test_column_a_fallback(self):
        import openpyxl
        wb = openpyxl.Workbook()
        wb.remove(wb["Sheet"])
        # Sheet title is truncated
        long_name = "a" * 40
        ws = wb.create_sheet(long_name[:31])
        ws.append(ci.HEADERS_CN)
        ws.append([long_name, "", "", "", "", "", "", "", "", ""])

        # Exact title match won't work since we search by the full name
        result = ci._find_sheet_for_group(wb, long_name)
        self.assertIsNotNone(result)
        self.assertEqual(result.title, long_name[:31])

    def test_no_match_returns_none(self):
        import openpyxl
        wb = openpyxl.Workbook()
        wb.remove(wb["Sheet"])
        ws = wb.create_sheet("Other")
        ws.append(ci.HEADERS_CN)
        ws.append(["Other", "", "", "", "", "", "", "", "", ""])

        result = ci._find_sheet_for_group(wb, "myGroup")
        self.assertIsNone(result)


# ===========================================================================
# Test: _unique_sheet_name
# ===========================================================================

class TestUniqueSheetName(unittest.TestCase):

    def test_no_conflict(self):
        import openpyxl
        wb = openpyxl.Workbook()
        self.assertEqual(ci._unique_sheet_name(wb, "myGroup"), "myGroup")

    def test_conflict_adds_suffix(self):
        import openpyxl
        wb = openpyxl.Workbook()
        wb.create_sheet("myGroup")
        result = ci._unique_sheet_name(wb, "myGroup")
        self.assertEqual(result, "myGroup_2")
        self.assertLessEqual(len(result), 31)

    def test_long_name_conflict(self):
        import openpyxl
        wb = openpyxl.Workbook()
        long_name = "a" * 40
        wb.create_sheet(long_name[:31])
        result = ci._unique_sheet_name(wb, long_name)
        self.assertNotEqual(result, long_name[:31])
        self.assertLessEqual(len(result), 31)
        self.assertTrue(result.endswith("_2"))


# ===========================================================================
# Test: _item_to_row
# ===========================================================================

class TestItemToRow(unittest.TestCase):

    def test_includes_project_id(self):
        item = {
            "main_group": "G",
            "sub_group": "G/s",
            "project_path": "G/s/p",
            "project_name": "p",
            "description": "d",
            "branches": "master",
            "times": "2026-01-01",
            "ssh_url": "git@x",
            "download_time": "2026-03-01",
            "project_id": 42,
        }
        row = ci._item_to_row(item)
        self.assertEqual(len(row), 10)
        self.assertEqual(row[9], 42)

    def test_missing_project_id_defaults_empty(self):
        item = {
            "main_group": "G",
            "sub_group": "G/s",
            "project_path": "G/s/p",
            "project_name": "p",
            "description": "d",
            "branches": "master",
            "times": "2026-01-01",
            "ssh_url": "git@x",
            "download_time": "2026-03-01",
        }
        row = ci._item_to_row(item)
        self.assertEqual(row[9], "")


# ===========================================================================
# Test: Sync mode in main() — environment variable handling
# ===========================================================================

class TestSyncModeSetup(unittest.TestCase):
    """Test that sync mode is parsed correctly and effective_mode is 'clone'."""

    @patch("clone_and_index.resolve_group_id")
    def test_sync_mode_effective_is_clone(self, mock_resolve):
        """Sync mode should resolve to effective_mode=clone internally."""
        mock_resolve.return_value = None

        os.environ["GITLAB_URL"] = "https://gl.test"
        os.environ["GITLAB_TOKEN"] = "test-token"
        os.environ["GITLAB_GROUPS"] = "nonexistent"
        os.environ["GITLAB_BASE_DIR"] = tempfile.mkdtemp()
        os.environ["GITLAB_MODE"] = "sync"

        # main() will exit(1) since group not found, but we can check
        # that it doesn't crash parsing "sync" mode
        with self.assertRaises(SystemExit):
            ci.main()

        for key in ["GITLAB_URL", "GITLAB_TOKEN", "GITLAB_GROUPS", "GITLAB_BASE_DIR", "GITLAB_MODE"]:
            os.environ.pop(key, None)


if __name__ == "__main__":
    unittest.main(verbosity=2)


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)