[serve] Cursor support, move docs into separate page, add more examples
Author: gante
Pull request: yes

# What does this PR do?

This PR:
- Adds support to CORS requests (e.g. Cursor)
- Changes emitted payload to avoid sending empty fields (as expected by Cursor)
- Moves `transformers serve` into the `Serving` docs page
- Adds more details to the existing `transformers serve` docs
- Adds example to use `transformers serve` with Jan and port forwarding 🤩 
- Adds example to use `transformers serve` with Cursor with `ngrok` 🤩  
- (Removes stray TF docs)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39133). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

server-side changes needed for Cursor + Cursor usage docs added 😈 

Added `transformers chat` one-line example 👍

I'll double-check the docs after they are built, then I'll merge and push Cursor comms 🔥  

hello, I'm new to transformers for serving my local LLM's. I usually just use ollama but the gguf quantized version of gemma3n doesn't support vision yet. I'm trying out the transformer library to serve LLm's especially gemma3n (with vision support).

I tried running 

``` 
transformers serve  
```

at the root of my project using cmd within vscode.... it gives this error 

```
RuntimeError: Impossible to instantiate a pipeline without either a task or a model being specified. Please provide a task class or a model
```

I double checked the docs. There is no mention of stating what model to use or what path for the models.
I thought running

```transformers serve```

would simply make all of my cached models (in the hugging face HOME and cached folders) available for other frontends to use (like Ollama).


It tried searching the github repo if this dilemma had already been answered but I couldn't find any..

Any help or suggestions would be much appreciated.


Thanks!


Hi @RadEdje, you need to install transformers from the `main` branch, because this feature hasn't made it to a PyPi release yet.

```bash
pip install git+https://github.com/huggingface/transformers
```

@pcuenca  Thanks a lot! It's running now... Figuring out transformer chat.... At the moment .... THANKS AGAIN.... ☺️

> @pcuenca Thanks a lot! It's running now... Figuring out transformer chat.... At the moment .... THANKS AGAIN.... ☺️

@pcuenca trying out transformers chat..... it seems it's all loading in may cpu and ram.... not utlizing my gpu.... is there a way to set transformers to use the gpu?

@RadEdje it should be automatic 🤔 can you share your setup, installed versions, and which model you're trying to use?

I think `transformers serve --device cuda` should work.

@gante I think the [default device is `cpu`](https://github.com/huggingface/transformers/blob/0687d481e2c71544501ef9cb3eef795a6e79b1de/src/transformers/commands/serving.py#L175), but the [model is loaded using `auto`](https://github.com/huggingface/transformers/blob/0687d481e2c71544501ef9cb3eef795a6e79b1de/src/transformers/commands/serving.py#L676), and then [moved to the destination](https://github.com/huggingface/transformers/blob/0687d481e2c71544501ef9cb3eef795a6e79b1de/src/transformers/commands/serving.py#L686) but only if there's no `hf_device_map`. So in computers with > 1 GPU it will remain on GPU, otherwise it will be moved to the default device.

@pcuenca there is quite possibly a bug there indeed, and it is poorly documented (we haven't spent much energy there). But on machines with 1 GPU `model.hf_device_map` will be something like `{'': 0}`, so the model stays on GPU. At least that's what happening on my setup atm.

@RadEdje If `--device cuda` works, then some info about model/setup will help me find more resilient solutions quickly 🤗 

@pcuenca @gante hello, thank you for your quick replies... I did as you asked: and tried 

```
transformers serve --device cuda
```

still not usiing my gpu which is an rtx 4070 super with 12 gigs vram.

the model is in cpu ram.

MY set up. 

```
CPU: AMD Ryzen 7 7700 8-Core
GPU 0: NVIDIA RTX 4070 super 12 gig vram
GPU 1:  AMD Radeon (built in to the cpu)
RAM: 32 gigs.

pip versions:
transformers : 4.54.0.dev0
torch: 2.7.1
accelerate: 1.8.1
datasets: 3.6.0
 evaluate:  0.4.4
pillow: 11.3.0
 timm: 1.0.16
rich: 14.0.0
```




IDE: VS code
terminal: cmd (in vs code)
python virtual environment using VENV.

I've set the download folder for models and cache for hugging face using:

```
set HF_HOME=D:\models_huggingface
```

since I don't want to fill up my drive C: with models.

after running the server in one terminal, I open a second terminal for the transformers chat

i look into the server terminal and this is what I get when I say hello or ask anything in the chat terminal:


```

INFO:     Started server process [18808]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
Loading google/gemma-3n-E4B-it@main
loading file tokenizer.model from cache at D:\models_huggingface\hub\models--google--gemma-3n-E4B-it\snapshots\----------------i ommited some numbers since I DON'T know if my api key is leaking out in the errors---------------\tokenizer.model
loading file tokenizer.json from cache at D:\models_huggingface\hub\models--google--gemma-3n-E4B-it\snapshots\---------------i ommited some numbers since I DON'T know if my api key is leaking out in the errors---------------\tokenizer.json
loading file added_tokens.json from cache at None
loading file special_tokens_map.json from cache at D:\models_huggingface\hub\models--google--gemma-3n-E4B-it\snapshots\---------------i ommited some numbers since I DON'T know if my api key is leaking out in the errors---------------------\special_tokens_map.json
loading file tokenizer_config.json from cache at D:\models_huggingface\hub\models--google--gemma-3n-E4B-it\snapshots\---------------i ommited some numbers since I DON'T know if my api key is leaking out in the errors\tokenizer_config.json
loading file chat_template.jinja from cache at D:\models_huggingface\hub\models--google--gemma-3n-E4B-it\snapshots\---------------i ommited some numbers since I DON'T know if my api key is leaking out in the errors\chat_template.jinja
loading configuration file config.json from cache at D:\models_huggingface\hub\models--google--gemma-3n-E4B-it\snapshots\---------------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------\config.json
text_config is None. Using default Gemma3nTextConfig.
vision_config is None. Using default Gemma3nVisionConfig.
audio_config is None. Using default Gemma3nAudioConfig.
Model config Gemma3nConfig {
  "architectures": [
    "Gemma3nForConditionalGeneration"
  ],
  "audio_config": {
    "conf_attention_chunk_size": 12,
    "conf_attention_context_left": 13,
    "conf_attention_context_right": 0,
    "conf_attention_logit_cap": 50.0,
    "conf_conv_kernel_size": 5,
    "conf_num_attention_heads": 8,
    "conf_num_hidden_layers": 12,
    "conf_positional_bias_size": 256,
    "conf_reduction_factor": 4,
    "conf_residual_weight": 0.5,
    "gradient_clipping": 10000000000.0,
    "hidden_size": 1536,
    "input_feat_size": 128,
    "model_type": "gemma3n_audio",
    "rms_norm_eps": 1e-06,
    "sscp_conv_channel_size": [
      128,
      32
    ],
    "sscp_conv_eps": 0.001,
    "sscp_conv_group_norm_eps": 0.001,
    "sscp_conv_kernel_size": [
      [
        3,
        3
      ],
      [
        3,
        3
      ]
    ],
    "sscp_conv_stride_size": [
      [
        2,
        2
      ],
      [
        2,
        2
      ]
    ],
    "torch_dtype": "bfloat16",
    "vocab_offset": ---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------,
    "vocab_size": 128
  },
  "audio_soft_tokens_per_image": 188,
  "audio_token_id": ---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------,
  "boa_token_id": ---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------,
  "boi_token_id": ---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------,
  "eoa_token_id": ---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------,
  "eoi_token_id": ---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------,
  "eos_token_id": [
    1,
    106
  ],
  "image_token_id": ---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------,
  "initializer_range": 0.02,
  "model_type": "gemma3n",
  "text_config": {
    "activation_sparsity_pattern": [
      0.95,
      0.95,
      0.95,
      0.95,
      0.95,
      0.95,
      0.95,
      0.95,
      0.95,
      0.95,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    "altup_active_idx": 0,
    "altup_coef_clip": 120.0,
    "altup_correct_scale": true,
    "altup_lr_multiplier": 1.0,
    "altup_num_inputs": 4,
    "attention_bias": false,
    "attention_dropout": 0.0,
    "final_logit_softcapping": 30.0,
    "head_dim": 256,
    "hidden_activation": "gelu_pytorch_tanh",
    "hidden_size": 2048,
    "hidden_size_per_layer_input": 256,
    "initializer_range": 0.02,
    "intermediate_size": [
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384,
      16384
    ],
    "laurel_rank": 64,
    "layer_types": [
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention"
    ],
    "max_position_embeddings": 32768,
    "model_type": "gemma3n_text",
    "num_attention_heads": 8,
    "num_hidden_layers": 35,
    "num_key_value_heads": 2,
    "num_kv_shared_layers": 15,
    "query_pre_attn_scalar": 256,
    "rms_norm_eps": 1e-06,
    "rope_local_base_freq": 10000.0,
    "rope_scaling": null,
    "rope_theta": 1000000.0,
    "sliding_window": 512,
    "torch_dtype": "bfloat16",
    "use_cache": true,
    "vocab_size": 262400,
    "vocab_size_per_layer_input": 262144
  },
  "torch_dtype": "bfloat16",
  "transformers_version": "4.54.0.dev0",
  "vision_config": {
    "architecture": "mobilenetv5_300m_enc",
    "do_pooling": true,
    "hidden_size": 2048,
    "initializer_range": 0.02,
    "label_names": [
      "LABEL_0",
      "LABEL_1"
    ],
    "model_args": null,
    "model_type": "gemma3n_vision",
    "num_classes": 2,
    "rms_norm_eps": 1e-06,
    "torch_dtype": "bfloat16",
    "vocab_offset": ---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------,
    "vocab_size": 128
  },
  "vision_soft_tokens_per_image": 256
}

loading weights file model.safetensors from cache at D:\models_huggingface\hub\models--google--gemma-3n-E4B-it\snapshots\---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------\model.safetensors.index.json
Will use torch_dtype=torch.bfloat16 as defined in model's config object
Instantiating Gemma3nForConditionalGeneration model under default dtype torch.bfloat16.
Generate config GenerationConfig {
  "eos_token_id": [
    1,
    ---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------
  ]
}

Instantiating TimmWrapperModel model under default dtype torch.bfloat16.
Instantiating Gemma3nTextModel model under default dtype torch.bfloat16.
Instantiating Gemma3nAudioEncoder model under default dtype torch.bfloat16.
Loading checkpoint shards: 100%|███████████████████████████████████████████| 4/4 [00:00<00:00,  9.32it/s]
All model checkpoint weights were used when initializing Gemma3nForConditionalGeneration.

All the weights of Gemma3nForConditionalGeneration were initialized from the model checkpoint at google/gemma-3n-E4B-it.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Gemma3nForConditionalGeneration for predictions without further training.
loading configuration file generation_config.json from cache at D:\models_huggingface\hub\models--google--gemma-3n-E4B-it\snapshots\---------i ommited some numbers since I DON'T know if my api key is leaking out in the errors-------\generation_config.json
Generate config GenerationConfig {
  "bos_token_id": 2,
  "cache_implementation": "hybrid",
  "do_sample": true,
  "eos_token_id": [
    1,
    106
  ],
  "pad_token_id": 0,
  "top_k": 64,
  "top_p": 0.95
}


```

This is my output from the server.
so far it works.
Just wish it would work with my GPU like ollama out of the box?

Thanks again for any advice.






Any idea how to solve this poblem ? or it works only in the free plan 

<img width="2282" height="1558" alt="image" src="https://github.com/user-attachments/assets/3ba9c7d9-72c2-4a1b-8669-0164fbc695fb" />


@RadEdje 

I see in your devices `GPU 1:  AMD Radeon (built in to the cpu)` -- this may be the source of problems 🤔 Maybe `--device cuda:0`? (I am not familiar with setups that have more than 1 kind of GPU)

@farouk09 

EDIT: hold up, I'm retesting the setup, and it seems it is no longer working for me either. Trying to figure out what's wrong.

EDIT2: nvm I was dumb

@farouk09 

I was able to reproduce a working env from `main` (more specifically, `4741e1f1b747a4a758e664582392155f73a9f3c8`). I am using Cursor with their "Business Plan".

However, the exact same setup in the "Free Plan" doesn't work -- Cursor blocks the request (See image, especially warning on the right. Got a similar warning in "Ask" mode).

My advice: use an IDE that's friendly towards local requests :)

<img width="996" height="876" alt="Screenshot 2025-07-25 at 10 40 22" src="https://github.com/user-attachments/assets/0fecea58-b987-45b2-9928-4e5cd39123d7" />