Memory leak when using openai/clip-vit-base-patch32
Author: Adefey
Pull request: no
Labels: name: bug

### System Info

Model runs on CPU+RAM. Ryzen 6800H, 14 GB DDR5, Host system: Fedora 41, Docker base image: python:3.13.7

FastAPI microservice is deployed in docker with these requirements:

fastapi==0.117.1
huggingface-hub==0.35.1
numpy==2.3.3
pillow==11.3.0
pydantic==2.11.9
python-multipart==0.0.20
requests==2.32.4
sentencepiece==0.2.1
torch==2.8.0
torchvision==0.23.0
transformers==4.56.2
uvicorn==0.37.0

transformers env:

embedding-service       | - `transformers` version: 4.56.2
embedding-service       | - Platform: Linux-6.16.7-100.fc41.x86_64-x86_64-with-glibc2.41
embedding-service       | - Python version: 3.13.7
embedding-service       | - Huggingface_hub version: 0.35.1
embedding-service       | - Safetensors version: 0.6.2
embedding-service       | - Accelerate version: not installed
embedding-service       | - Accelerate config: not found
embedding-service       | - DeepSpeed version: not installed
embedding-service       | - PyTorch version (accelerator?): 2.8.0+cu128 (NA)
embedding-service       | - Tensorflow version (GPU?): not installed (NA)
embedding-service       | - Flax version (CPU?/GPU?/TPU?): not installed (NA)
embedding-service       | - Jax version: not installed
embedding-service       | - JaxLib version: not installed
embedding-service       | - Using distributed or parallel set-up in script?: (I'm not sure tbh...)


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Use CPU+RAM
2. UPD: Add at least 1000 images to `data` folder
3. Git clone this repository: https://github.com/Adefey/search_dir, checkout to this commit: `d8a40baa434c4b00ad04dada9d7221edb111f4aa`
4. Run: docker compose up --build
5. Call API: POST http://localhost:8003/api/v1/start_discovery
6. Monitor logs, depending on available system memory embedding-service may get OOM (embedding-service exited with code 137) after some batches (on my system with 14 GB ram it's 10 batches)
But actually there are only 3 methods (`__init__`, `_encode`,  `encode_images`) that may be relevant. Repeated calls of `encode_images` result in OOM. I'll show relevant methods here:

```
def __init__(self):
        self.model_checkpoint = "openai/clip-vit-base-patch32"
        os.system("transformers env")
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        logger.info(f"Start setting up model {self.model_checkpoint} on {self.device}")
        self.model = AutoModel.from_pretrained(self.model_checkpoint).to(self.device)
        self.processor = AutoProcessor.from_pretrained(self.model_checkpoint, use_fast=False)
        logger.info(f"Finished setting up model {self.model_checkpoint} on {self.device}")
```

```
def _encode(self, inputs: dict) -> list[float]:
        inputs = {k: v.to(self.device) for k, v in inputs.items()}

        if "pixel_values" in inputs:
            features = self.model.get_image_features(**inputs)
        else:
            features = self.model.get_text_features(**inputs)

        result = features.cpu().detach().numpy().tolist()

        del inputs
        del features
        if self.device == "cuda":
            torch.cuda.empty_cache()

        # ??????????
        # trim_memory()

        return result
```

```
def encode_images(self, images: list[bytes]) -> list[list[float]]:
        """
        Process images into embeddings
        """
        logger.info(f"Start encoding images")
        image_list = [Image.open(io.BytesIO(image)) for image in images]
        with torch.inference_mode():
            inputs = self.processor(
                images=image_list,
                return_tensors="pt",
                padding=True,
            )
            result = self._encode(inputs)
        for image in image_list:
            image.close()
        logger.info(f"Finished encoding images")
        return result
```

Also, there is a working fix: calling `trim_memory()` after each model call:

```
def trim_memory():
    libc = ctypes.CDLL("libc.so.6")
    return libc.malloc_trim(0)
```

But I think this is a workaround and transformers library should manage resources correctly on its own.

### Expected behavior

Usecase: microservice with Model gets batches of 30 images to calculate embeddings, then it gets another batch. After 10 batches service is killed because of OOM. Manually monitoring memory in htop - usage increases with every batch by 600...800 MB. Expected behavior - constant memory usage for at every time during batch processing.
I suppose there is a memory leak or memory fragmentation issue where new memory keeps being allocated and not reused. 

UPD: exactly same OOM issue happens with `google/siglip2-base-patch16-256` and again `malloc_trim` workaround works

Comments:
Hi, we can't run a reproducer that depends on checking out your entire repo and spinning up an entire docker image. Can you try to cut this down to a minimal reproducer (maybe using a Hub dataset so you can just download it in one line?) so we can just run the code in a notebook and see the issue?

> Hi, we can't run a reproducer that depends on checking out your entire repo and spinning up an entire docker image. Can you try to cut this down to a minimal reproducer (maybe using a Hub dataset so you can just download it in one line?) so we can just run the code in a notebook and see the issue?

Hi,
I made a reproduction in Google Colab. It does not reproduce every time, however it reproduced consistently with with FastAPI/Uvicorn. This exact reproduction was done after running this notebook twice (without reset)
https://colab.research.google.com/drive/11xq99KQh1z72PVmPoECPEmT26ybZjirN?usp=sharing

Only model code runs there with and without `malloc_trim`. You can see that with it memory consumption is constant and without it slowly grows. In my scenario with FastAPI/Uvicorn it grown much faster for some reason.

Also there are some existing issues in Pytorch with same issue:
https://github.com/pytorch/pytorch/issues/68114
https://github.com/pytorch/pytorch/issues/114455



@Adefey Could you check if #41215 fix the issue (especially with `FastAPI/Uvicorn`) 🙏 

Hi
@ydshieh
I checked it, but the issue remains with PR

Also I noticed that after manual image preprocessing (resize to 224x224) memory grows much slower (but still noticeable), almost unnoticeable. Without it (as was reported in this bug) memory grows much faster. My first approach without image preprocessing was relying in internal preprocessing done in AutoProcessor. As you can see in my reproducer in Colab - I feed processor with raw images without any preprocessing.
Is it possible that the issue in Processor and not Model?

Thanks for double checking.

> I checked it, but the issue remains with PR

Is this (still) happening with your original reproducible (i.e. with FastAPI) or also on the Colab reproducer ?

> Is it possible that the issue in Processor and not Model?

Could you try to disable the part involving models (i.e. only keep the processor part) and see if we still have the memory issue?



> Thanks for double checking.
> 
> > I checked it, but the issue remains with PR
> 
> Is this (still) happening with your original reproducible (i.e. with FastAPI) or also on the Colab reproducer ?
> 
> > Is it possible that the issue in Processor and not Model?
> 
> Could you try to disable the part involving models (i.e. only keep the processor part) and see if we still have the memory issue?

Yes, it still happens with FastAPI/uvicorn. I tried to reprodue it again (in the same colab) with processor only, but it did not reproduce. I'm not sure what exactly causes problem especially with FastAPI app.  

I could try to dive into it, if you are able to a minimal example: ideally, a single simple python script (loading process / models, simple usage of FastAPI etc), an instruction of installing dependencies, i.e. don't include the whole things in your repo . I think it's doable despite it will take you more time to prepare the minimal example.



Hi

It seems it's a FastAPI/uvicorn issue after all. I wrote 2 scripts - model usage in loop and model usage in loop through http request to a FastAPI app. How can usage of FastAPI/uvicorn affect memory consumption?

<details>
  <summary>Logs from script with FastAPI</summary>
<code>
[23:27:09] {main_fastapi.py:48} INFO - Start setting up model openai/clip-vit-base-patch32 on cpu
[23:27:14] {main_fastapi.py:51} INFO - Finished setting up model openai/clip-vit-base-patch32 on cpu
INFO:     Started server process [78881]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[23:27:26] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 49.527%
INFO:     127.0.0.1:60488 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:29] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 43.766%
INFO:     127.0.0.1:60494 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:32] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 38.029%
INFO:     127.0.0.1:60498 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:36] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 32.219%
INFO:     127.0.0.1:50230 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:39] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 29.929%
INFO:     127.0.0.1:50234 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:43] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 25.857%
INFO:     127.0.0.1:50236 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:46] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 21.836%
INFO:     127.0.0.1:52208 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:49] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 19.173%
INFO:     127.0.0.1:52218 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:52] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 19.021%
INFO:     127.0.0.1:52226 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:56] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 17.044%
INFO:     127.0.0.1:59560 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:27:59] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 16.052%
INFO:     127.0.0.1:59574 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:28:02] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 15.310%
INFO:     127.0.0.1:59578 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:28:05] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 15.769%
INFO:     127.0.0.1:45780 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:28:09] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 11.048%
INFO:     127.0.0.1:45784 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:28:12] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 11.279%
INFO:     127.0.0.1:45798 - "POST /api/v1/images_embeddings HTTP/1.1" 200 OK
[23:28:16] {main_fastapi.py:132} INFO - Batch processed. Memory free WITH FASTAPI: 9.564%
</code>
</details>

<details>
  <summary>Logs from script without FastAPI</summary>
<code>
[23:25:20] {main.py:45} INFO - Start setting up model openai/clip-vit-base-patch32 on cpu
[23:25:25] {main.py:48} INFO - Finished setting up model openai/clip-vit-base-patch32 on cpu
[23:25:31] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 53.200%
[23:25:34] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 54.135%
[23:25:37] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 53.213%
[23:25:40] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 53.845%
[23:25:43] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 54.037%
[23:25:46] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 54.455%
[23:25:49] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 53.056%
[23:25:52] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 54.033%
[23:25:55] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 53.470%
[23:25:58] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 53.765%
[23:26:01] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 54.126%
[23:26:03] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 53.525%
[23:26:06] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 54.570%
[23:26:09] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 54.218%
[23:26:12] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 54.573%
[23:26:15] {main.py:114} INFO - Batch processed. Memory free NO FASTAPI: 54.735%
</code>
</details>

Prepare environment:

`pip install -r requirements.txt`

Run without FastAPI:

`python main.py`

Run with FastAPI:

`python main_fastapi.py`

Repo with reproduction:
https://github.com/Adefey/transformers_leak_reproduction

@Adefey I don't have the insight about it, but it seems it's a common issue with FastAPI if one searches with Google

https://github.com/fastapi/fastapi/discussions/11079
https://www.reddit.com/r/learnpython/comments/1669z84/unresolved_memory_management_issues_in/
https://www.google.com/search?q=fastapi+memory+leak&oq=fastapi+memory+leak&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIICAEQABgWGB4yCAgCEAAYFhgeMggIAxAAGBYYHjIICAQQABgWGB4yCAgFEAAYFhgeMggIBhAAGBYYHjIICAcQABgWGB4yCAgIEAAYFhgeMggICRAAGBYYHtIBCDQxOTNqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8

But it might require some time to read the discussions and try.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.