Teaching a coding agent to deploy production endpoints on Amazon SageMaker
Deploying a Hugging Face model to SageMaker is mostly repetitive work. The steps barely change from one model to the next, which is exactly the kind of thing you'd want to hand off to a coding agent. In theory you point an agent at a model ID on the Hub, walk away, and come back to a working endpoint. I wanted to see how well that actually holds up, so I gave the job to Claude Code running Opus 4.8, which per the Hub's own agent usage data is the agent most people use to work with Hugging Face right now. And I watched the entire run rather than just checking the endpoint at the end. Spoiler, it did not go well.
The first surprise was that the agent doesn't even fail the same way twice, and that how badly it fails tracks how new the model is. When I asked it to deploy a small text-generation model (Qwen/Qwen3-0.6B), it gave me an endpoint that worked but was quietly fragile. And keep in mind that this model has been on the Hub since April 2025, over a year at the time of writing and an eternity in this space, so this was the easy case. When I asked it to deploy google/diffusiongemma-26B-A4B-it, a multimodal diffusion model published in June 2026 and less than a month old, it handed back a confident deployment script built on a serving container that can't run that kind of model at all, and left me to discover that myself. The newer the model, the further it sits from anything in the agent's training data, and the worse the confident guesses get.
The unpredictability is the actual problem. If the knowledge isn't written down somewhere the agent can reach it, even a capable model gives you something different, and differently broken, each time. The same agent, once I gave it a set of skills I wrote, was consistent across every model I threw at it. And I picked a deliberately strong baseline: if a frontier model is this shaky on its own, a smaller or older one won't do better, so what follows is closer to a best case than a worst one.
The text-generation run is the clearest one to walk through, because there the stock agent did reach a live endpoint. That lets us put the two runs side by side, line for line. The request was the kind of thing a teammate would actually send:
I need to deploy a model on AWS SageMaker and I don't want to deal with all the console clicking and boto3 myself. The model is a small Qwen, called from an internal app. Figure out the best way to deploy it and walk me through it. Write the plan to a file first, and keep a log of every action you take.
The model was Qwen/Qwen3-0.6B, deployed to a real-time endpoint on a single ml.g5.xlarge instance. Here's what the two runs produced, before I get into how:
| Stock agent | With skills | |
|---|---|---|
| Serving container | TGI first → health-check failure → vLLM | vLLM, chosen before any resource was created |
| Image URI | discovered by trial and error | resolved from AWS's DLC catalog, with a fallback when ECR was denied |
| Autoscaling | none | target-tracking, 1–2 instances |
| Monitoring | none | three CloudWatch alarms (latency, errors, overhead) |
| Docs it left behind | README recommended TGI, the SageMaker SDK, Python 3.13 | plan and scripts agreed with what actually ran |
| Region / role / env | correct, by instinct | correct, by rule |
| Teardown | a script you could run | run, then verified the resources were gone |
The rest of this post is about how the right-hand column got built, and why the gap between the two has almost nothing to do with how smart the model is.
What the first run got wrong
To give the stock agent its due, it did a lot right unprompted. It read the local AWS config to find the account and region instead of asking me. It wrote out a plan and waited before touching anything. It went looking for an existing execution role rather than trying to create one. And it kept a clean log of every command. If you judged it purely on the endpoint it ended up with, you'd call it a success.
The trouble was in how it got there, and in what it left behind.
It picked TGI to serve the model. That's an understandable choice. TGI is Hugging Face's own text-generation server, and for years it was the default way to serve Hub LLMs on SageMaker, so the agent's training data is stuffed with tutorials that reach for it. The problem is that TGI has since been archived, and the builds available in the region predate the Qwen3 architecture, so the endpoint failed its health check. Since then we started publishing Hugging Face distributions of vLLM, SGLang and llama.cpp. Ideally the agent would find all our latest containers here and know which one would work best for that model. The agent didn't quit after one try. It bumped the TGI version, redeployed, failed again, and only switched to vLLM after roughly 80 minutes and four failed endpoint attempts, each one billing GPU time as it spun up and then fell over.
10:22:28 [aws] FAILURE endpoint=qwen3-06b-endpoint reason=ping-health-check-failed
10:22:28 [claude] HYPOTHESIS root_cause=TGI-3.0.1-predates-Qwen3-arch-support remediation=bump-to-3.3.6
11:14:19 [claude] PIVOT backend=TGI->vLLM reason=TGI-too-old-for-Qwen3-confirmed-via-console-logs
The switch to vLLM lands 52 minutes after that first health-check failure; counting from the first TGI attempt, the whole detour ate about 80 minutes. And choosing TGI for a current Qwen isn't a judgment call that needs an experiment to settle. It's a known fact today, and those 80 minutes were the price of the agent not having it.
The README.md was worse. The scripts that ran ended up on vLLM and plain boto3, but the README told the reader to use TGI, install the SageMaker Python SDK, and assured them Python 3.13 was fine. Every one of those is a reliable way to break a SageMaker deployment, and one of them was the exact failure the agent had just spent an hour working around. The documentation pointed the next person straight back at the wall.
The endpoint itself worked, but it was a demo. A single always-on GPU instance, no autoscaling, no alarms. And since a real-time endpoint has no scale-to-zero, it bills the full instance rate the entire time it exists, whether it's serving traffic or sitting idle. An ml.g5.xlarge on demand runs about $730 a month, and nothing in the deployment stops that meter. Shutting it down is a manual step you have to remember.
That was the loud failure. The second request, for google/diffusiongemma-26B-A4B-it (a multimodal mixture-of-experts model that generates by discrete diffusion rather than one token at a time) failed quietly instead. The agent looked the model up, confirmed it existed, and wrote a deployment script that reached, again, for TGI:
image_uri=get_huggingface_llm_image_uri("huggingface", version="3.0.1") # TGI DLC
TGI is a text-generation server; it has no backend for a discrete-diffusion image-text model. And the agent half-knew it. In the same breath it noted that "stock TGI may not have a backend for that" and that the instance it had picked was "a 4B-active sizing guess." It handed me a confident-looking script built on the wrong container and left the verification to me. Nothing blew up the way the Qwen health check did. You'd only find out when the endpoint refused to come up.
The two runs are the same root cause in different clothes: a loud death on one model, a quiet wrong answer on the other. Neither is a reasoning failure. The agent planned and debugged perfectly well. What it lacked was current, specific knowledge about the task: that TGI is archived and recent Qwen models need vLLM, that Python 3.13 has no working wheels for much of the stack, that container images should come from AWS's published catalog rather than the model's memory. That kind of knowledge doesn't sit cleanly in a model's weights. What lives there is the average of everything it has read, and a lot of what it has read is older tutorials still saying "use TGI" and "pip install sagemaker." On top of that, a model's knowledge has a hard cutoff date: a new architecture, a fresh container tag, or a newly archived server simply is not in the weights yet. The more recent the thing you ask an agent to work with, the more it has to guess, which is exactly why it handled a year-old Qwen better than a month-old DiffusionGemma. Even the things the agent got right, it got right on this account, on this day, with nothing guaranteeing the same calls would work on a stricter account or a different model. The difference was never how well it could reason. It was only what it had to work from.
Writing the knowledge down as skills
The fix isn't a bigger model. It's a way to hand the model that knowledge exactly when it needs it. I used skills: Markdown files, each with a name, a description that tells the agent when to load it, and a body of instructions, sometimes with helper scripts attached. A skill loads on demand when the task matches its description, so the model stays general while the procedure stays in version control.
Here's a trimmed version of the one that fixes the container problem:
---
name: serving-image-selection
description: >
Pick the right serving container for a SageMaker deployment and find its
current image URI. Use whenever an image URI needs to be chosen. Prefers the
HuggingFace-curated DLCs first: HuggingFace vLLM (LLMs), vLLM-Omni
(multimodal), TEI (embeddings/rerankers), HF Inference Toolkit, with
DJL-LMI and SGLang as alternatives. Never hardcode a container URI from
memory and never default to TGI.
---
# Serving Image Selection
The serving container is the single thing most likely to break a deployment that
"looked correct on paper". Wrong container, stale tag, or wrong AMI all produce
the same opaque `Failed to pass health check` error.
## Where image URIs come from
Primary source: AWS's official Deep Learning Containers catalog. Read the URI
from there, substitute the region, and pass it to deploy.py --image-uri. Never
recite a tag from memory.
Worth noticing: most of the families this skill chooses between are Hugging Face's own serving stacks, published as AWS Deep Learning Containers through the HF-AWS partnership. TEI (Text Embeddings Inference) for embeddings and rerankers, the HF Inference Toolkit for classic pipelines, HF-curated builds of vLLM and SGLang. The right container for a Hub model usually already exists. The agent's job is just to pick it on purpose instead of reciting one from memory.
I wrote six skills that cover deployment end to end, and the agent pulls them in as the task moves through its phases:
sagemaker-deployment-planner decide pathway (real-time / async / batch), ask only what's needed
│
├── aws-context-discovery profile, region, account, caller identity (read-only)
├── python-env-setup isolated env, supported Python, current boto3
├── sagemaker-iam-preflight find a usable execution role; create only if none exists
├── serving-image-selection pick the container family, resolve a current image URI
└── sagemaker-production-defaults deploy with autoscaling + alarms + tags, then smoke-test
The second run, step by step
With the skills in place, the same request produced a very different trace. These lines are straight from the action log the agent kept:
STEP 4 (serving image selection) START. Family=vllm (NOT TGI; Qwen3 requires vLLM DLC).
NOTE: ecr-public:DescribeImages DENIED for SSO role -> resolver used known-good FALLBACK tag.
IMAGE_URI = 763104351884.dkr.ecr.us-east-1.amazonaws.com/vllm:0.21.0-gpu-py312-cu130-ubuntu22.04-sagemaker-v1.4
AWS CALL: CreateModel -> "qwen3-06b"
AWS CALL: CreateEndpoint -> "qwen3-06b-20260529-0722". Status=Creating. BILLING ACTIVE.
RESULT: Endpoint reached InService after 483s (~8 min).
AWS CALL: RegisterScalableTarget + PutScalingPolicy -> autoscaling min=1 max=2, target 20 invocations/min/instance.
AWS CALL: PutMetricAlarm x3 -> CloudWatch alarms (latency/errors/overhead).
AWS CALL: InvokeEndpoint -> HTTP 200, ~1.9s. Model=Qwen/Qwen3-0.6B, server=vllm-0.21.0.
NOTE: Qwen3 reasoning/think mode is ON by default; response hit max_tokens(64) inside <think>.
The container question got settled before anything was created. serving-image-selection picked vLLM (it would have picked TEI for an embedding model, and it's the same skill that would have steered the diffusion model away from TGI) and resolved the image from the catalog instead of from memory. Notice that the weights never detour through my laptop or S3: the container pulls Qwen/Qwen3-0.6B directly from the Hub when the endpoint starts, and a gated model would just need a HUGGING_FACE_HUB_TOKEN alongside the variables below. The model went out with the vLLM DLC and these environment variables, which the skill knows the vLLM container expects:
SM_VLLM_MODEL=Qwen/Qwen3-0.6B
SM_VLLM_TRUST_REMOTE_CODE=true
SM_VLLM_MAX_MODEL_LEN=32768
SM_VLLM_GPU_MEMORY_UTILIZATION=0.9
The bad documentation couldn't happen again, because python-env-setup encodes the opposite of what the first run wrote down: an isolated environment every time, a Python version that ML wheels actually exist for (3.10–3.12, never 3.13+), and a current boto3. It also defaults to driving the deployment with boto3 directly rather than through the SageMaker SDK. That's a preference, not a hard ban: we recently merged a PR that aligns the SDK's routing logic with what these skills expect, so the SDK path works fine now too. But managing the calls directly still gives you the tightest control over what actually gets created, so that's the road the skill takes by default.
The execution role is where naive deployments most often die, because the obvious move, calling iam:CreateRole, fails on a corporate account whose SSO session has no IAM write access. sagemaker-iam-preflight flips the order. Its check_role.py searches the account for existing roles matching patterns like AmazonSageMaker-ExecutionRole-* and *SageMaker*Execution*, ranks them by last-used date, validates the trust policy, and prints the ARN. It only creates a role if there genuinely isn't one and the caller has permission. The agent found a usable role and moved on.
Finally, sagemaker-production-defaults is what turns the endpoint from a demo into something you can leave running. It creates the model, endpoint config, endpoint, autoscaling policy, and alarms together, from defaults defined in one place:
| Guardrail | Setting |
|---|---|
| Autoscaling | target-tracking on SageMakerVariantInvocationsPerInstance, target 20/instance, min 1 / max 4 |
| Scale cooldowns | 60s out, 300s in |
Alarm: ModelLatency |
> 30,000 ms |
Alarm: Invocation5XXErrors |
> 5 in a 5-minute period |
Alarm: OverheadLatency |
> 2,000 ms |
Those are the skill's defaults. Because this was a small model behind a low-traffic internal app, the agent capped autoscaling at two instances rather than the default four, which is why the log above reads max=2.
Before it crossed into anything billable, the agent stopped and asked for an explicit go-ahead. When I was done, it ran the teardown and then confirmed the endpoint, config, and model were actually gone rather than assuming the delete had worked.
The parts I didn't program
Two moments in the second run weren't things I'd designed for, and they were the most convincing part of the whole exercise.
The first was a permission error. To find the newest image tag, the agent tried to ask ECR directly, but the SSO role it was running under didn't have rights to call ecr-public:DescribeImages, so the call was denied. A naive script would have crashed there. Instead the agent fell back to the known-good tag the skill ships as a safety net, wrote a note in the log explaining exactly why it had done so, and carried on to a working endpoint.
The second was a quality catch. After the endpoint came up, the agent ran a smoke test to confirm it actually answered. The reply came back cut off. It had hit the 64-token limit while Qwen3 was still inside its reasoning block, so the "real" answer never got emitted. Rather than call the test a pass and move on, the agent noticed the truncation, understood the cause, and flagged that the calling app should raise its token limit.
Nobody told the agent to handle a denied ECR call gracefully or to sanity-check the content of the smoke test. The skills just gave it enough context about how this deployment is supposed to behave that it could recognize when something was off and do the sensible thing on its own.
Why six skills instead of one long prompt
I could have crammed all of this into a single system prompt. I tried, and it didn't hold up. A prompt long enough to cover every sharp edge gets partly ignored, goes stale the day AWS ships a new container, and has to be re-pasted into every session. Skills load only when they're relevant, live in version control next to the helper scripts they call, and get updated in one place.
That last point is really the whole argument. "Use vLLM, not TGI, for Qwen3" is true today and wasn't true two years ago, when TGI was the default for Hub LLMs. It'll flip again when the next serving stack shows up. The reason to write a fact like that into a file rather than trust the model to recall it is precisely that it's going to change. And when it does, you edit one skill instead of hoping every model out there has read the update. The helpers are plain Python and the AWS CLI, nothing shell-specific, so the same skills behave identically on macOS, Linux, and Windows.
Try it with your own Hub model
The skills from this post are open source in huggingface/skills as the hf-cloud-* skills. Install them with the Hugging Face CLI:
hf skills add hf-cloud-aws-context-discovery
hf skills add hf-cloud-python-env-setup
hf skills add hf-cloud-sagemaker-deployment-planner
hf skills add hf-cloud-sagemaker-iam-preflight
hf skills add hf-cloud-serving-image-selection
hf skills add hf-cloud-sagemaker-production-defaults
Then give your agent a request like this:
Deploy Qwen/Qwen3-0.6B to a real-time SageMaker endpoint for an internal app.
The important part is that the request stays ordinary. You do not need to tell the agent which container family to use, how to find the SageMaker execution role, how to resolve the DLC image URI, or which production defaults to attach. Those decisions live in the skills.
The model stays on the Hub, the serving stack is selected from the Hugging Face and AWS integration, and the deployment comes up with the production pieces already attached: autoscaling, alarms, tags, smoke test, and teardown.
The skills work with Claude Code, Codex, and Pi today, and the Hugging Face Skills repo makes the same instructions installable through hf skills. Because they are just Markdown files plus small helper scripts, the same pattern can be adapted to other agent harnesses that support loading task-specific instructions. If you try them on a model, region, account setup, or serving path I did not cover, open an issue or PR. The point is to keep this knowledge in the open, where the next agent run can actually use it.
The takeaway
The lesson from this experiment is not that agents cannot deploy models, indeed, they can. The lesson is that deployment knowledge changes faster than model weights do.
TGI used to be the obvious answer for many LLMs on SageMaker. For current Qwen models, e.g. the Qwen 3.6 family, it is the wrong one: it is no longer maintained, so newer, up-to-date solutions should be used instead. Python versions age in and out of ML wheel support. AWS container tags move. Corporate IAM policies differ from tutorial accounts. None of that is hard reasoning, but it is exactly the kind of detail an agent will confidently blur if it has to rely on past knowledge. And the effect gets stronger the newer things are: the agent was shaky on a model that had been public for over a year, and flatly wrong on one that had been out for less than a month. Whatever is released tomorrow starts even further from its training data.
Skills give that knowledge a home. They let the agent stay general while the deployment procedure stays current, reviewable, and shared. For Hugging Face workflows, that matters because the Hub is already the source of truth for the model. The next step is making the operational knowledge around that model just as reachable.
A coding agent that can create an endpoint is useful. A coding agent that knows which Hugging Face serving stack to use, verifies what it created, and leaves behind something your team can operate is the one I actually want near production.