Repository: huggingface/peft

Issue:
Loading LORA weights in `diffusers` with a `peft` backend increases in latency as more paths are added to `PYTHONPATH`
### System Info

```
accelerate==0.21.0
diffusers==0.26.3
peft==0.9.0
safetensors==0.3.3
tokenizers==0.15.2
torch==2.2.1
transformers==4.36.2
```

### Who can help?

@sayakpaul 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder
- [X] My own task or dataset (give details below)

### Reproduction

```python
from diffusers import DiffusionPipeline
import time
import torch
import sys
import os
import shutil

pipe_id = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DiffusionPipeline.from_pretrained(pipe_id, torch_dtype=torch.float16).to("cuda")

loras = [
    {
        "adapter_name": "anime"
        "location": "./anime",
        "weight_name": "anime.safetensors",
        "token": "my_anime"
    },
]

def run_dynamic_lora_inference(lora):
    start_load_time = time.time()
    pipe.load_lora_weights(lora["location"], weight_name=lora["weight_name"], adapter_name=lora["adapter_name"])
    end_load_time = time.time()
    prompt = f"Illustration of a dog in the style of {lora["token"]}"

    start_fuse_time = time.time()
    pipe.fuse_lora()
    end_fuse_time = time.time()

    start_set_adapter_time = time.time()
    pipe.set_adapters(lora_name)
    end_set_adapter_time = time.time()

    start_inference_time = time.time()
    image = pipe(
        prompt, num_inference_steps=30, generator=torch.manual_seed(0)
    ).images[0]
    end_inference_time = time.time()

    start_unfuse_time = time.time()
    pipe.unfuse_lora()
    end_unfuse_time = time.time()

    start_unload_time = time.time()
    pipe.unload_lora_weights()
    end_unload_time = time.time()

    image.save(f"./{lora_name}.png")

    print("Load time:", end_load_time - start_load_time)
    print("Fuse time:", end_fuse_time - start_fuse_time)
    print("Set adapter time", end_set_adapter_time - start_set_adapter_time)
    print("Inference time:", end_inference_time - start_inference_time)
    print("Unfuse time:", end_unfuse_time - start_unfuse_time)
    print("Unload time:", end_unload_time - start_unload_time)

def add_to_python_path():
    root_path = "./folders"
    shutil.rmtree(root_path)
    os.mkdir(root_path)

    folders =  [f"folder_{x}" for x in range(0, 10000)]
    for folder in folders:
        os.mkdir(os.path.join(root_path, folder))
        sys.path.append(os.path.join(root_path, folder))

def main():
    for lora in loras:
        run_dynamic_lora_inference(lora)

main()
```

Flamegraph: 
![profile](https://github.com/huggingface/peft/assets/54301847/63b4d841-e740-4cb8-af51-96d011a8bbae)



### Expected behavior

I run a system with a somewhat large `PYTHONPATH` that we can't truncate, and we are currently blocked from upgrading `diffusers` to any version that uses `peft` for LORA inference. 

It's loosely based on this post: https://huggingface.co/blog/lora-adapters-dynamic-loading

We've observed a behavior where the time taken for `load_lora_weights` increases significantly with more paths added to `PYTHONPATH`. This can be reproduced in the example provided - with 10,000 folders added to PYTHONPATH, we get the following latencies:

```
Load time: 291.78441095352173
Fuse time: 0.12406659126281738
Set adapter time 0.06171250343322754
Inference time: 9.685987710952759
Unfuse time: 0.08063459396362305
Unload time: 0.15737533569335938
```

Benchmarking against 1, 10, 100, 1000, 10000 and 50000 entries in the `PYTHONPATH`, we get a pretty astounding increase in load latency: 

![image](https://github.com/huggingface/peft/assets/54301847/cd8894d5-3c6c-47fd-b8ca-ead34a32e36d)

Even at 100 entries, we're looking at an extra 4 seconds per `load` call which is a pretty significant increase.

We looked briefly at it and came to the conclusion that it's something to do with the way `peft` runs module imports, particularly repetitive calls to import modules, where the imports are not cached, eg, `importlib.util.find_spec` doesn't cache imports. 

Instead of this behaviour, we'd expect that `load_lora_weights` retains a relatively constant load time, regardless of the length of our python path. 


Discussion:
Interesting, thanks for bringing this to our attention. My first instinct would be to add a cache to all the functions that use `importlib.util.find_spec`, as something like:

```python
def is_bnb_available() -> bool:
    return importlib.util.find_spec("bitsandbytes") is not None
```

should be safe to cache. WDYT, would that solve your issue?
Potentially, yeah - is it possible to do this once at a higher level in the code, rather than every function call? Otherwise decorating them with @functools.cache might also help :) 
> is it possible to do this once at a higher level in the code

You mean at the caller site of these functions? Very unlikely, as they can be used in many different places. However, I think that a cache on these functions should be fast enough. Do you want to give this a try?

Resolving patch:
diff --git a/src/peft/import_utils.py b/src/peft/import_utils.py
--- a/src/peft/import_utils.py
+++ b/src/peft/import_utils.py
@@ -18,10 +18,12 @@
 import packaging.version
 
 
+@lru_cache
 def is_bnb_available() -> bool:
     return importlib.util.find_spec("bitsandbytes") is not None
 
 
+@lru_cache
 def is_bnb_4bit_available() -> bool:
     if not is_bnb_available():
         return False
@@ -31,6 +33,7 @@ def is_bnb_4bit_available() -> bool:
     return hasattr(bnb.nn, "Linear4bit")
 
 
+@lru_cache
 def is_auto_gptq_available():
     if importlib.util.find_spec("auto_gptq") is not None:
         AUTOGPTQ_MINIMUM_VERSION = packaging.version.parse("0.5.0")
@@ -44,6 +47,7 @@ def is_auto_gptq_available():
             )
 
 
+@lru_cache
 def is_optimum_available() -> bool:
     return importlib.util.find_spec("optimum") is not None
 
@@ -65,9 +69,11 @@ def is_torch_tpu_available(check_device=True):
     return False
 
 
+@lru_cache
 def is_aqlm_available():
     return importlib.util.find_spec("aqlm") is not None
 
 
+@lru_cache
 def is_auto_awq_available():
     return importlib.util.find_spec("awq") is not None


Test patch: