Sanad

Sanad investigates VMware vSphere incidents. An engineer describes the problem in English or Arabic; Sanad reads vCenter through read-only tools, diagnoses the fault and proposes a fix. Every step is approved and run by the engineer, and the result is checked again in vCenter before the incident is closed.

This repository contains three LoRA adapters for Qwen3.5-4B and the runtime that connects them to vCenter.

Agents

Adapter	Role	Output
adapters/operator	Investigates with the tools, writes the diagnosis with its evidence, proposes actions	Report and propose_action calls
adapters/guardian	Reviews each proposal against the tool outputs and the action policy	ALLOW, RETRY, BLOCK or ASK_USER
adapters/verifier	Re-checks vCenter after the engineer runs a step	resolved, not_resolved or pending_external

engineer report → Operator ⇄ vCenter tools → code checks → Guardian → engineer approves and runs → Verifier

Between the Operator and the Guardian, deterministic code checks reject numbers or quoted log lines that do not appear in any tool output, ignored tool errors, and actions the policy forbids. A blocked action cannot be proposed again in the same incident; after two blocks the incident is escalated to the engineer. An investigation that repeats the same tool call or ends without an answer is also escalated.

An incident that ends without a verified fix stays open until the Verifier confirms the fix on a re-check against vCenter or an engineer closes it with a note. Open incidents are listed on every page and reported by browser notification and by e-mail through the data center's mail server, with reminders while they stay open.

Operator tools: get_vm_status, get_host_metrics, get_cluster_info, get_vcenter_health, check_datastore, search_logs, network_test, search_kb, create_ticket, propose_action. Guardian tools: inspect_tool_output, recompute, check_policy. None of them changes anything in vCenter.

Model details

Property	Value
Base model	Qwen/Qwen3.5-4B, loaded from its Unsloth copy unsloth/Qwen3.5-4B
Method	bf16 LoRA, one adapter per agent, loss on assistant tokens only
LoRA	r 16, alpha 16; q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training configuration	learning rate 2e-4, effective batch 16, 1 epoch, sequence length 6144 (Operator) and 4096 (Guardian, Verifier)
Libraries	Unsloth, TRL SFTTrainer, PEFT, Transformers 5.2.0, PyTorch 2.8.0
Chat format	Qwen chat template with tool calls; the tokenizer and template are stored with the Operator adapter

Training data

All training data comes from the simulator in sanad_sim/: a vSphere data center with 42 faults taken from public Broadcom knowledge base articles, plus incidents with no fault and incidents naming an unknown VM. Host, VM, datastore and cluster names change in every episode. Half of the engineer reports are Arabic (Gulf dialect), half English.

Agent	Train	Validation	Held-out evaluation
Operator	3,190	371	120
Guardian	3,360 (ALLOW 1,042 · RETRY 1,224 · BLOCK 924 · ASK_USER 170)	420	252
Verifier	1,680 (resolved 597 · not_resolved 896 · pending_external 187)	210	148

Every Operator example uses only names that appeared in the report or in an earlier tool output, passes the code checks, and its fix resolves the fault in the simulator. Verifier labels come from the simulator state after the action, and the same executed steps can end either way, so the status must be read from fresh tool output. python scripts/make_all_data.py regenerates the data.

Languages

Reports can be written in English, Arabic (Modern Standard or Gulf dialect) or a mix of both. The Operator answers in the language of the report; Arabic answers are written in Modern Standard Arabic. Tool names, arguments and JSON fields are always English, and inventory names are kept exactly as vCenter returns them.

Before inference, sanad_text.normalize_report maps pasted text onto the characters seen in training: Arabic presentation forms, bidirectional and zero-width marks, Arabic-Indic digits and the Arabic percent sign, and elongation inside words. Words and spelling are not changed. Reports are limited to 4,000 characters. The case library matches Arabic text with hamza, alef, taa marbuta and definite-article differences folded.

Evaluation

Each agent was evaluated live in the simulator on held-out episodes: the model calls the tools itself, the simulator answers, and the outcome is scored against the simulator state.

Agent	Result	Notes
Operator	Fault resolved in 98.3% of 60 held-out episodes	The one failure quoted a log line no tool returned; the code checks now reject it
Guardian	Correct decision in 96.7% of held-out review packets	Faulty proposals allowed through: 3.9%; both were ungrounded numbers that the code checks also catch
Verifier	Status correct in 98.3% of 60 held-out cases	Missed failures: 0.0% of cases that were not resolved
Full pipeline	28 of 30 held-out incidents ended correctly (93.3%)	All three adapters together. One investigation repeated a log search until it ran out of turns; the runtime now stops such runs and escalates them. One wrong proposal was stopped by the Guardian and escalated without any change. The Verifier's final status matched the simulator in every incident

These are simulator results. Accuracy on a production vCenter is measured in Shadow Mode, where engineers rate each suggestion against what they found and did.

Requirements

Component	Requirement
Server	Linux, Python 3.10 or later
GPU	One NVIDIA GPU with at least 16 GB and bf16 support, or an existing vLLM server
vCenter	7.0 or 8.0, an account with the Read-only role
Logs	Optional: syslog files or Aria Operations for Logs. vCenter events are always available
esxcli	Optional: govc for vmkping, /tmp usage and LACP status

Installation

hf download RabiaaAi/sanad --local-dir sanad
cd sanad
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt -r requirements-gpu.txt
python -m sanad_live.web --data /opt/sanad/data --host 127.0.0.1 --port 8080

The console prints a setup address with a one-time code. The setup wizard creates the administrator, tests the vCenter connection and the log source, runs every tool once against vCenter, checks the adapters and the GPU, and starts Sanad in Shadow Mode or Assist Mode. The adapter folders default to adapters/. DEPLOY.md covers offline installation, the vCenter role, log forwarding, vLLM, the systemd service and the rollout.

Inference without the web application

from sanad_live.models import make_backend
from sanad_pipeline import SanadPipeline
from sanad_sim import Simulator

agents = make_backend({"backend": "transformers", "adapters": {
    "operator": "adapters/operator", "guardian": "adapters/guardian", "verifier": "adapters/verifier"}}).agents()

episode = Simulator(seed=1).new_episode("C01", lang="ar")       # a simulated incident
incident = SanadPipeline(*agents).handle(episode)
print(incident.outcome)
print(incident.answer)

The base model is loaded once and each agent switches to its own adapter. With vLLM, start the server with the three adapters as LoRA modules named operator, guardian and verifier (deploy/vllm.service) and use {"backend": "vllm", "url": "http://127.0.0.1:8000", "tokenizer": "adapters/operator"}.

Limitations

• Training covers 42 simulated faults. An incident outside them may be matched to the closest known fault; the evidence lines and the Guardian review exist to expose that, and the engineer decides.
• Production tool outputs are larger and less regular than the simulator's. Lists are truncated at 40 items.
• Without govc or a log source, some fields are empty and the faults that depend on them are harder to diagnose.
• The Operator was not trained on the past_cases that the case library adds to search_kb results.
• High-risk actions get one Guardian review and engineer approval of every step. A second, independently trained Guardian for multi-vote review is not part of this release.
• The local GPU backend serves one request at a time; use vLLM for several engineers.
• The vCenter tools and the web application are tested against vcsim, the vSphere API simulator from govmomi, not yet against a production vCenter. Run Shadow Mode first.
• Sanad needs network access to vCenter, so it runs inside the data center and not as a hosted demo.

Repository layout

Path	Contents
adapters/	The three LoRA adapters; tokenizer and chat template in adapters/operator
sanad_live/	vCenter tools, log sources, model backends, web application and setup wizard
sanad_pipeline.py, sanad_cases.py, sanad_text.py	Agent pipeline, case library, report normalization
sanad_sim/	Simulator, fault catalog, knowledge base and training-data generators
training/	Training and evaluation notebooks, training utilities
scripts/, tests/, deploy/	Data and release scripts, tests, systemd service files
docs/DEVELOPMENT.md	Simulator, data generation, training and testing

License and attribution

The adapters and code are released under the Apache License 2.0 (LICENSE).

• Base model: Qwen3.5-4B, Copyright 2026 Alibaba Cloud, Apache License 2.0.
• Fault catalog and knowledge base entries summarize public Broadcom knowledge base articles; every entry links its source article.
• Training: Unsloth, TRL, PEFT, Transformers and PyTorch. Runtime: pyVmomi, FastAPI and Uvicorn. govc from VMware govmomi is optional and not included.