Instructions to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jab1718/qwen3.8-flash-coder-85gb-bf16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jab1718/qwen3.8-flash-coder-85gb-bf16")
model = AutoModelForCausalLM.from_pretrained("Jab1718/qwen3.8-flash-coder-85gb-bf16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jab1718/qwen3.8-flash-coder-85gb-bf16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jab1718/qwen3.8-flash-coder-85gb-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jab1718/qwen3.8-flash-coder-85gb-bf16

SGLang

How to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jab1718/qwen3.8-flash-coder-85gb-bf16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jab1718/qwen3.8-flash-coder-85gb-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jab1718/qwen3.8-flash-coder-85gb-bf16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jab1718/qwen3.8-flash-coder-85gb-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with Docker Model Runner:
```
docker model run hf.co/Jab1718/qwen3.8-flash-coder-85gb-bf16
```

Incredible project questions about total download size,coding quality,benchmarks

by Abdulazizv - opened 2 days ago

Discussion

Abdulazizv

2 days ago

•

edited 2 days ago

Awesome project , and
Am i correct that You are trying to make model size smaller while preserving agentic coding quality it will be close to the original model right ?
And also will the final model be able to run as fast as other 35b moe models.
do you plan to do benchmarks
I am really hyped for this project .
Fully supporting you !!

Jab1718

Owner 1 day ago

yep, but im not so sure if im going to do a full benchmark comparison since i only have around 76gb vram. (will consider about that later on )

labdhee

1 day ago

following this thread

Abdulazizv

about 8 hours ago

•

edited about 8 hours ago

Hi ,
saw your discussion where you asked and I would lean toward option B, because the single gpu version would make this much more accesible to users with limited vram.

However, i also think the bf16 dora work is also really valuable.if the failures are like mostly edge cases rather than capability issues it means tuning can could make it bf16 checkpoint a much stronger foundation.
I think ideal path is to keep the bf16 checkpoint as the main reference model and make int4 int8 variabts from the improved version.
It would be also interesting to see int4 vs int8 results on the same 100 tasks
This is already a pretty fascinating thing.

Jab1718

Owner about 8 hours ago

good idea, imma try that soon

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment