Memory leak when acquiring CLIP embeddings
Author: timgianitsos
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This function is called twice, both times going into the `else` block.
```python
def get_embeddings(data_partition_name, version):
    load_file = Path(f'{data_partition_name.lower()}-embeddings.pt')
    if load_file.exists():
        print(f'Loading "{load_file}"...')
        pt = torch.load(load_file)
        embeddings = pt['embeddings']
        classes = pt['classes']
    else:
        encoder = CLIPVisionModelWithProjection.from_pretrained('openai/clip-vit-base-patch32')
        processor = AutoProcessor.from_pretrained('openai/clip-vit-base-patch32')
        data = pd.read_csv(f'{data_partition_name.lower()}-swings.csv')
        data_dir = Path('data')
        print(f'Generating embeddings for {data_partition_name} data...')
        extension = '*.jpeg'
        filenames = sorted((data_dir / data_partition_name).glob(extension))
        batch_size = 30

        loader = DataLoader(
            ImageDataset(filenames),
            batch_size=batch_size,
            num_workers=1,
            shuffle=False,
        )
        embeddings = []
        for batch in tqdm(loader, desc=f'Loading image batches of size {batch_size}'):
            embeddings.append(encoder(**processor(images=batch, return_tensors='pt')).image_embeds)
            gc.collect()
            torch.cuda.empty_cache()
            gc.collect()
        embeddings = torch.cat(embeddings)
        classes = torch.cat([
            torch.tensor(data[data["name"] == jpg.name][version].values)
            for jpg in filenames
        ])
        torch.save({'embeddings': embeddings, 'classes': classes}, load_file)

    return embeddings, classes
```

### Expected behavior

There seems to be a memory leak.

I have about 2000 jpeg images which takes up more the 1 GB of space, and I am passing them to a HuggingFace model to produce embeddings.

The problem is that the program crashes because of memory issues. Upon investigation with `htop`, it seems that the `Swp` usage keeps climbing higher and higher until it surpasses around 50GB in which case it crashes with `Killed: 9` and _no_ stack trace. This happens despite the fact that the memory utilization is not full.

Why is my program allocating so much from swap? Is this a bug in HuggingFace?

The function is called twice, once for train and once for test.

- No memory/swp is reclaimed after `get_embeddings('train')` is run. That is, even when `get_embeddings('test')` is run, the swap remains very high at 40+GB of swap used, and then keeps climbing.
- I am loading in batches with a PyTorch `DataLoader`, but the bug persists whether or not I use `DataLoader`.
- The bug persists whether or not I use `pandas` to read in the csv file.
- The bug persists whether or not I do the garbage collection step
- I am not using a GPU

The dataset and [csv files](https://github.com/zapan4/BadmintonProject/blob/master/train-swings.csv) are in a private repo currently, but they accord straightforwardly with the description.

Comments:
This does not seem like a memory leak, you're not using `torch.no_grad()`, which means that with every forward pass you're stacking up more gradients into your memory.

Thanks @NielsRogge!

Why doesn't memory get reclaimed between multiple invocations to `get_embeddings`? Shouldn't it be garbage collected since the model is initialized within the function?

It seems that you're doing multiple forward passes within your function, so gradients get accumulated for all batches.

Are you only using the model for inference to get embeddings? If yes then you can use the `@torch.no_grad()` annotator above your function. It will make sure no gradients are accumulated.

Yes I was able to use `@torch.inference_mode()` to fix the issue. Much appreciated!

I'm just curious how it is possible for memory to not get deallocated between function calls.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

There weren't any global variables, so the reference to the model should disappear, and the object itself should be deallocated. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

To clarify, I replaced
```python
for batch in tqdm(loader, desc=f'Loading image batches of size {batch_size}'):
    embeddings.append(encoder(**processor(images=batch, return_tensors='pt')).image_embeds)
    gc.collect()
    torch.cuda.empty_cache()
    gc.collect()
```
with
```python
with torch.inference_mode():
    for batch in tqdm(loader, desc=f'Loading image batches of size {batch_size}'):
        embeddings.append(encoder(**processor(images=batch, return_tensors='pt')).image_embeds)
```
and still got a memory leak. [EDIT I meant to say "and *stopped* having a memory leak"]

> There weren't any global variables, so the reference to the model should disappear, and the object itself should be deallocated.

In the example script provided, `encoder` and `embeddings` are still being referenced so `gc.collect()` and `torch.cuda.empty_cache` won't do anything in the for-loop. Using `torch.no_grad` works because no gradients are when doing the forward pass of the model and, as a result, the `embedding` array being generated at each step of the for-loop is smaller. 

Yes, using `torch.no_grad` and `torch.inference_mode` would indeed prevent the backward graph and intermediate tensors from being created in memory in the first place. But to address my original inquiry about why *if* they are created they *continue* to persist outside of the function scope, I think I found the answer after reviewing this with fresh eyes - I think it's because the `embeddings.grad_fn` attribute is a lingering reference to a closure that has the whole computational backward graph. Using `embeddings.detach()` on that last return statement should likely avoid the leak because it evaluates to a tensor with the same underlying data but without `.grad_fn`.