XinShu3047/SKC-checkpoints

Selective Knowledge Control for Continual Learning of GUI Agents

Stage-wise Checkpoints for Selective Knowledge Control Over Application Streams

  💻 Code   |   📑 Paper: Coming Soon   |   🤖 Model Checkpoints  

This repository provides the public, stage-wise model checkpoints for Selective Knowledge Control (SKC), a lightweight continual-learning method for GUI agents trained over a stream of applications.

The corresponding training code extends the DART-GUI infrastructure. For training, environment setup, SKC state construction, baselines, and implementation details, see the SKC code repository.

📢 Updates

• Released the first eight SKC application-stream checkpoints.
• Standardized checkpoint names as stage-{NN}-{application} using lowercase kebab-case.
• Each stage directory is a self-contained Hugging Face Transformers checkpoint.

🤖 Released Checkpoints

The checkpoints follow the continual-learning order used by SKC. stage-{NN} is the model obtained after training through the corresponding application stage.

Stage	Application	Directory
1	VLC	stage-01-vlc
2	Thunderbird	stage-02-thunderbird
3	LibreOffice Writer	stage-03-libreoffice-writer
4	LibreOffice Impress	stage-04-libreoffice-impress
5	LibreOffice Calc	stage-05-libreoffice-calc
6	GIMP	stage-06-gimp
7	VS Code	stage-07-vscode
8	Chrome	stage-08-chrome

For reproducible continual-learning evaluation, use the checkpoints in stage order. For final-stage evaluation, use stage-08-chrome.

🚀 Download

Download the complete repository:

huggingface-cli download XinShu3047/SKC-checkpoints \
  --local-dir /path/to/SKC-checkpoints

Download a single stage, for example the final checkpoint:

huggingface-cli download XinShu3047/SKC-checkpoints \
  --include "stage-08-chrome/*" \
  --local-dir /path/to/SKC-stage8

Load a downloaded checkpoint with Transformers:

from transformers import AutoModelForVision2Seq, AutoProcessor

checkpoint = "/path/to/SKC-checkpoints/stage-08-chrome"

processor = AutoProcessor.from_pretrained(
    checkpoint,
    trust_remote_code=True,
)
model = AutoModelForVision2Seq.from_pretrained(
    checkpoint,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

Use the same inference and GUI-environment conventions as UI-TARS/DART-GUI. Exact rollout-service and OSWorld worker instructions are documented in the SKC code repository.

🧠 Training Context

SKC controls updates to historically important model directions while learning a stream of GUI applications. The training workflow is:

1. Train the first application without a historical SKC state.
2. Build the protection state from successful trajectories and the completed stage checkpoint.
3. Enable SKC for the next application using the accumulated historical state.
4. Merge the new state after each stage and continue through the application stream.

The released directories contain model checkpoints only. Training trajectories, databases, logs, environment images, and SKC intermediate state artifacts are not included here.

🔗 Related Resources

• SKC code: https://github.com/shzirui/SKC
• DART-GUI: https://github.com/computer-use-agents/dart-gui
• Base model: https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B

🤝 Acknowledgments

This work builds on DART-GUI and verl, uses vLLM for rollout model serving, and starts from UI-TARS-1.5-7B. We thank the authors and maintainers of these projects.

📝 Citation

The SKC paper citation will be added after public release.

@misc{skc_gui_agent_continual_learning,
  title = {Selective Knowledge Control for Continual Learning of GUI Agents Over Application Streams},
  year = {2027},
  note = {Anonymous submission artifact}
}