Skill: zhuoran-liu/autodl-train (version 1.0.0)

--- name: operating-autodl-training description: Operates remote model training jobs on AutoDL Linux servers over SSH. Use when starting a training run, checking whether training is still alive, reviewing GPU/CPU/memory/disk usage, reading recent logs, diagnosing abnormal interruptions, or summarizing the latest training outcome with next-step recommendations. --- # Operating AutoDL Training Use this skill for remote training operations on an AutoDL Linux server. It is designed for high-frequency workflows around "start training, watch progress, inspect resources, read logs, diagnose failures, and decide what to do next" while keeping execution constrained to one configured project directory. ## What This Skill Does - Starts a configured training command in the target project directory over SSH. - Activates the remote Python environment with Conda or virtualenv fallbacks. - Checks whether training is still running by combining process, GPU, and log freshness signals. - Summarizes GPU, CPU, memory, and disk pressure instead of dumping raw command output. - Reads recent logs and extracts likely metrics such as `epoch`, `step`, `loss`, `lr`, `grad_norm`, `val_loss`, `accuracy`, `mAP`, and `F1`. - Detects common training failures such as CUDA OOM, NCCL errors, NaN, disk full, timeout, and segmentation faults. - Produces a human-readable training summary and recommends whether to continue, tune, or resume from a checkpoint. ## Required Inputs Collect or confirm these values before running any script: - `host`: AutoDL server hostname or IP. - `port`: SSH port, usually `22`. - `username`: Remote Linux username. - `project_path`: Absolute project directory on the remote server, for example `/root/autodl-tmp/your-project`. - One environment option: `env_name`, `env_activate`, or `venv_path`. - `train_command`: The training launch command, such as `python train.py`, `python -m torch.distributed.run ...`, or `bash scripts/train.sh`. - Optional password mode: provide `AUTOCLAW_TRAIN_SSH_PASSWORD` as an environment variable or local `.env` file when SSH key login is not available. Prefer a config file at `config.example.json` copied to a real file such as `config.json`, or environment variables based on `.env.example`. ## Safety Rules - Only operate inside the configured `project_path`. - Do not invent missing SSH credentials or secrets. - Do not write plaintext passwords into files. - Prefer SSH keys or environment variables. - Refuse obviously destructive launch commands such as `rm -rf`, `reboot`, `shutdown`, `mkfs`, or fork bombs. - Do not kill unrelated processes or run global destructive recovery commands. ## Workflow ### 1. Confirm Configuration Read `config.example.json` and `references/usage.md` to understand the expected fields. Ask the user for any missing values instead of guessing. ### 2. Start Or Resume Training Run `scripts/remote_train.py` to start a background job or build a resume command: ```bash python scripts/remote_train.py --config config.json python scripts/remote_train.py --config config.json --resume-from outputs/checkpoints/last.ckpt ``` Use this when the user asks to launch training, re-launch after interruption, or resume from a checkpoint. ### 3. Check Live Status Run `scripts/check_status.py` when the user asks whether training is still running: ```bash python scripts/check_status.py --config config.json ``` This script combines process matching, `nvidia-smi`, and recent log updates to classify the run as `running`, `stopped`, `failed`, or `unknown`. ### 4. Inspect Resource Pressure Run `scripts/monitor_resources.py` to summarize GPU/CPU/memory/disk usage: ```bash python scripts/monitor_resources.py --config config.json ``` Use the human-readable bottleneck assessment in the output instead of pasting raw command output unless the user asks for raw data. ### 5. Read Logs And Summaries Run `scripts/summarize_log.py` in one of these modes: ```bash python scripts/summarize_log.py --config config.json --action read --tail 200 python scripts/summarize_log.py --config config.json --action detect-failure --tail 400 python scripts/summarize_log.py --config config.json --action summarize --tail 400 ``` Use `read` for recent excerpts and metrics, `detect-failure` for exception diagnosis, and `summarize` for a concise human-facing assessment with next steps. ## Script Map - `scripts/remote_train.py`: start training, optional resume templating, structured launch result. - `scripts/check_status.py`: process/GPU/log-based training status. - `scripts/monitor_resources.py`: GPU/CPU/memory/disk summary and bottleneck hints. - `scripts/summarize_log.py`: read logs, detect failures, summarize convergence and next actions. - `scripts/common.py`: shared config loading, SSH execution, safe path checks, remote helpers. - `scripts/log_utils.py`: reusable log parsing, failure detection, trend analysis, recommendation logic. ## References - Read `references/usage.md` for setup steps, example configs, and example commands. - Read `references/troubleshooting.md` when SSH, environment activation, logs, or training recovery fail. ## Agent Guidance - Start with the least invasive action that answers the user’s request. - When the user asks a yes/no status question, prefer `scripts/check_status.py` before reading a long log. - When the user asks why training stopped, run `scripts/check_status.py` and then `scripts/summarize_log.py --action detect-failure`. - When the user asks whether to continue training, run `scripts/summarize_log.py --action summarize` and include the recommendations from the script in the final response. - When a checkpoint path is provided, prefer `scripts/remote_train.py --resume-from ...` so the resume command is explicit and auditable.

path: .gitignore
content: .env
.env.local
config.local.json
config.*.local.json
config.private.json
config.*.private.json
*.zip


path: agents/openai.yaml
content: interface:
  display_name: AutoDL Train Operator
  short_description: 通过 SSH 启动、监控、读日志并总结 AutoDL 训练任务
  icon: server
  primary_color: "#2563eb"


path: config.example.json
content: {
  "host": "YOUR_AUTODL_HOST",
  "port": 22,
  "username": "root",
  "ssh_key_path": "~/.ssh/id_rsa",
  "strict_host_key_checking": "accept-new",
  "project_path": "/root/autodl-tmp/your-project",
  "allowed_project_roots": [
    "/root/autodl-tmp"
  ],
  "env_name": "your_env",
  "conda_sh_path": "/root/miniconda3/etc/profile.d/conda.sh",
  "venv_path": "venv",
  "env_activate": "",
  "train_command": "python train.py --config configs/train.yaml",
  "resume_argument_template": "--resume-from {checkpoint}",
  "process_match": "train.py --config configs/train.yaml",
  "log_path": "logs/train.log",
  "log_candidates": [
    "logs/train.log",
    "outputs/latest.log",
    "nohup.out"
  ],
  "status_log_tail": 40,
  "summary_log_tail": 400,
  "resource_warn_disk_pct": 85,
  "resource_warn_memory_pct": 90,
  "resource_warn_gpu_memory_pct": 90,
  "resource_low_gpu_util_pct": 30,
  "extra_ssh_options": []
}


path: README.md
content: # AutoDL Train Operator Skill

This directory is a publishable OpenClaw skill for operating remote model training jobs on an AutoDL Linux server.

## Included

- `SKILL.md`: Skill metadata and workflow.
- `agents/openai.yaml`: Minimal UI metadata for ClawHub/OpenClaw.
- `config.example.json`: Copy to `config.json` and fill with real values.
- `.env.example`: Optional environment variable configuration.
- `scripts/*.py`: CLI tools for start, status, resource checks, and log summarization.
- `references/*.md`: Usage guide and troubleshooting.

## Quick Start

1. Copy `config.example.json` to `config.json`.
2. Fill in SSH host, username, project path, environment settings, training command, and log path.
3. Run one of the scripts from this directory.

```bash
python scripts/remote_train.py --config config.json
python scripts/check_status.py --config config.json
python scripts/monitor_resources.py --config config.json
python scripts/summarize_log.py --config config.json --action summarize
```

## Packaging

If your environment provides `package_skill.py`, package this skill directory as a zip and upload the result to ClawHub.

## Publishing Checklist

1. Keep only publish-safe files inside this skill directory.
2. Store machine-specific config outside the skill directory, or use local-only names such as `config.local.json` and `.env`.
3. Never include real passwords, tokens, hostnames, IP addresses, SSH keys, or personal project paths in uploaded files.
4. Verify that `SKILL.md`, `agents/openai.yaml`, `config.example.json`, `scripts/`, and `references/` are present.
5. Rebuild the zip after any cleanup so the archive matches the final publishable directory.

## Recommended Local Practice

1. Put your private runtime config outside the skill directory, for example in `../local/`.
2. If you keep a local `.env`, do not upload it.
3. Use `config.example.json` as the only config file that ships with the skill.


path: references/troubleshooting.md
content: # Troubleshooting

## SSH Connection Fails

### Symptom

- `ssh exited with code 255`
- `Permission denied (publickey)`
- `Connection timed out`

### Checks

- Confirm `host`, `port`, `username`, and `ssh_key_path`.
- Test SSH manually with the same host and key.
- Ensure the key has correct permissions such as `chmod 600 ~/.ssh/id_rsa`.
- If host key prompts block automation, keep `strict_host_key_checking` as `accept-new` or pre-populate `known_hosts`.
- If the server is password-based, export `AUTOCLAW_TRAIN_SSH_PASSWORD` before running the scripts.

## Environment Activation Fails

### Symptom

- `conda: command not found`
- Training command starts but imports fail immediately.

### Fixes

- Set `conda_sh_path` to the correct AutoDL Miniconda path.
- Set `env_name` to the actual Conda environment.
- If the project uses a venv, set `venv_path` to the venv directory.
- If you already have a custom activation one-liner, set `env_activate` directly.

## Training Starts Then Exits Immediately

### Symptom

- `remote_train.py` returns a PID, but `check_status.py` reports `stopped` or `failed` soon after.

### Fixes

- Run `summarize_log.py --action detect-failure` to inspect recent errors.
- Confirm `train_command` is valid from inside the remote project directory.
- Check whether the launcher script writes logs somewhere other than `log_path` or `log_candidates`.
- If using distributed launchers, refine `process_match` so status checks can find the correct process.

## No Logs Found

### Symptom

- Log selection returns no file.

### Fixes

- Set `log_path` explicitly if your project writes to a custom location.
- Add additional entries to `log_candidates`.
- Check whether the training script writes only to stdout; if so, keep `remote_train.py` background mode so stdout is redirected to the configured log file.

## GPU Utilization Is Very Low

### Possible Reasons

- Data loading is too slow.
- Batch size is too small.
- The process is stuck in CPU preprocessing or dataset I/O.
- The training job already stopped but memory is still partially reserved.

### Suggested Actions

- Check `monitor_resources.py` for CPU load and memory pressure.
- Inspect log timestamps with `check_status.py`.
- Increase DataLoader workers, pin memory, or batch size if the framework allows it.

## CUDA Out Of Memory

### Typical Fixes

- Reduce batch size.
- Enable gradient accumulation.
- Use mixed precision.
- Shorten sequence length or image resolution.
- Resume from the latest checkpoint after adjusting memory-sensitive settings.

## Disk Nearly Full

### Typical Fixes

- Move checkpoints to a larger mount.
- Reduce checkpoint retention.
- Clean old experiment outputs manually after verification.
- Do not automate destructive cleanup inside this skill.

## NaN Or Divergence

### Typical Fixes

- Lower learning rate.
- Check mixed precision stability.
- Clip gradients.
- Validate data normalization and labels.
- Resume from an earlier stable checkpoint rather than the most recent broken state.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)