OneShot Deployment

Run the following commands from the model directory to serve POST /v1/systemone.

1. Install (First Time Only)

Requirements: Linux, Bash ≥ 4.3, Python 3.12, CUDA Toolkit 12.8 (with nvcc available), and an NVIDIA driver compatible with CUDA 12.8. The GPU must support BF16 and FlashAttention 2.

python3.12 -m venv .venv
PY="$PWD/.venv/bin/python"
"$PY" -m pip install --upgrade pip setuptools wheel packaging ninja --index-url https://pypi.org/simple
"$PY" -m pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
"$PY" -m pip install --no-build-isolation -r serving/requirements.txt --index-url https://pypi.org/simple

2. Start the Server

GPU=0 HOST=0.0.0.0 bash serving/serve.sh

The default API port is 30100; override it with API_PORT.

In another terminal, wait until the health check returns status: "ok":

curl --fail http://127.0.0.1:30100/health

3. Call the /v1/systemone

curl --fail http://127.0.0.1:30100/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{"state":"The same order was charged twice.","questions":{"department":{"type":"choice","instructions":"Select the department that should handle this issue.","criteria":{"payments":"Payments, billing, and refunds","technical":"Errors and technical support"}}}}'