How to load PixArtAlphaPipeline in 8bit?
Author: FurkanGozukara
Pull request: no

I know there is example but I couldn't make it work. I am trying to make an auto installer and gradio interface for Pix Art Alpha Pipeline so common people can install and use on their Windows PCs

Currently my below code working and I want to make it load in 8 bit is that possible?

```
if torch.cuda.is_available():
    pipe = PixArtAlphaPipeline.from_pretrained(
        "PixArt-alpha/PixArt-XL-2-1024-MS",
        torch_dtype=torch.float16,
        use_safetensors=True,
    )

    if ENABLE_CPU_OFFLOAD:
        pipe.enable_model_cpu_offload()
    else:
        pipe.to(device)
        print("Loaded on Device!")

    # speed-up T5
    pipe.text_encoder.to_bettertransformer()

    if USE_TORCH_COMPILE:
        pipe.transformer = torch.compile(pipe.transformer, mode="reduce-overhead", fullgraph=True)
        print("Model Compiled!")
```

```
        seed = int(randomize_seed_fn(seed, randomize_seed))
        generator = torch.Generator().manual_seed(seed)

        if schedule == 'DPM-Solver':
            if not isinstance(pipe.scheduler, DPMSolverMultistepScheduler):
                pipe.scheduler = DPMSolverMultistepScheduler()
            num_inference_steps = dpms_inference_steps
            guidance_scale = dpms_guidance_scale
        elif schedule == "SA-Solver":
            if not isinstance(pipe.scheduler, SASolverScheduler):
                pipe.scheduler = SASolverScheduler.from_config(pipe.scheduler.config, algorithm_type='data_prediction', tau_func=lambda t: 1 if 200 <= t <= 800 else 0, predictor_order=2, corrector_order=2)
            num_inference_steps = sas_inference_steps
            guidance_scale = sas_guidance_scale
        else:
            raise ValueError(f"Unknown schedule: {schedule}")

        if not use_negative_prompt:
            negative_prompt = None  # type: ignore
        prompt, negative_prompt = apply_style(style, prompt, negative_prompt)

        images = pipe(
            prompt=prompt,
            width=width,
            height=height,
            guidance_scale=guidance_scale,
            num_inference_steps=num_inference_steps,
            generator=generator,
            num_images_per_prompt=NUM_IMAGES_PER_PROMPT,
            use_resolution_binning=use_resolution_binning,
            output_type="pil",
        ).images
```

### Who can help?

@sayakpaul @Narsil @SunMarc @younesbelkada @gante 


I tried below but it broken the app

```
text_encoder = T5EncoderModel.from_pretrained(
    "PixArt-alpha/PixArt-XL-2-1024-MS",
    subfolder="text_encoder",
    load_in_8bit=True,
    device_map="auto",

)
pipe = PixArtAlphaPipeline.from_pretrained(
    "PixArt-alpha/PixArt-XL-2-1024-MS",
    text_encoder=text_encoder,
    transformer=None,
    device_map="auto"
)
```

The error I am getting is like below

```
Downloading shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<?, ?it/s]
bin G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.dll
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:06<00:00,  3.09s/it]
Loading pipeline components...:   0%|                                                                                        | 0/4 [00:00<?, ?it/s]Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading pipeline components...: 100%|████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  9.50it/s]
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
batch_count 1
Traceback (most recent call last):
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\queueing.py", line 427, in call_prediction
    output = await route_utils.call_process_api(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\blocks.py", line 1484, in process_api
    result = await self.call_function(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\blocks.py", line 1106, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\utils.py", line 665, in wrapper
    response = f(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\utils.py", line 665, in wrapper
    response = f(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\app.py", line 176, in generate
    images = pipe(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\diffusers\pipelines\pixart_alpha\pipeline_pixart_alpha.py", line 731, in __call__
    ASPECT_RATIO_1024_BIN if self.transformer.config.sample_size == 128 else ASPECT_RATIO_512_BIN
AttributeError: 'NoneType' object has no attribute 'config'
Traceback (most recent call last):
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\queueing.py", line 427, in call_prediction
    output = await route_utils.call_process_api(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\blocks.py", line 1484, in process_api
    result = await self.call_function(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\blocks.py", line 1106, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\utils.py", line 665, in wrapper
    response = f(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\utils.py", line 665, in wrapper
    response = f(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\app.py", line 176, in generate
    images = pipe(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\diffusers\pipelines\pixart_alpha\pipeline_pixart_alpha.py", line 731, in __call__
    ASPECT_RATIO_1024_BIN if self.transformer.config.sample_size == 128 else ASPECT_RATIO_512_BIN
AttributeError: 'NoneType' object has no attribute 'config'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\queueing.py", line 472, in process_events
    response = await self.call_prediction(awake_events, batch)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\queueing.py", line 436, in call_prediction
    raise Exception(str(error) if show_error else None) from error
Exception: None

```


Comments:
Hello, @sayakpaul and @SunMarc should know more about this. Untagging myself. Thank you.

You need to point to the exact line of code that breaks stuff and I am unable to see what that is. 

```python
text_encoder = T5EncoderModel.from_pretrained(
    "PixArt-alpha/PixArt-XL-2-1024-MS",
    subfolder="text_encoder",
    load_in_8bit=True,
    device_map="auto",

)
```

is totally possible given `bitsandbytes` is installed. 

> You need to point to the exact line of code that breaks stuff and I am unable to see what that is.
> 
> ```python
> text_encoder = T5EncoderModel.from_pretrained(
>     "PixArt-alpha/PixArt-XL-2-1024-MS",
>     subfolder="text_encoder",
>     load_in_8bit=True,
>     device_map="auto",
> 
> )
> ```
> 
> is totally possible given `bitsandbytes` is installed.

thank you for reply

bitsandbytes  properly installed

everything loads no error

during inference exactly breaking line like this

  **File "G:\pixArt installer\PixArt-alpha\app_8bit.py", line 176, in generate**

the real error happens in below

```
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\diffusers\pipelines\pixart_alpha\pipeline_pixart_alpha.py", line 731, in __call__
    ASPECT_RATIO_1024_BIN if self.transformer.config.sample_size == 128 else ASPECT_RATIO_512_BIN
AttributeError: 'NoneType' object has no attribute 'config'
```

use_resolution_binning is set True

here full code

```

if torch.cuda.is_available():
	text_encoder = T5EncoderModel.from_pretrained(
		"PixArt-alpha/PixArt-XL-2-1024-MS",
		subfolder="text_encoder",
		load_in_8bit=True,
		device_map="auto",

	)
	pipe = PixArtAlphaPipeline.from_pretrained(
		"PixArt-alpha/PixArt-XL-2-1024-MS",
		text_encoder=text_encoder,
		transformer=None,
		device_map="auto"
)


def create_output_folders():
    base_dir = "outputs"
    today = datetime.now().strftime("%Y-%m-%d")
    folder_path = os.path.join(base_dir, today)
    if not os.path.exists(folder_path):
        os.makedirs(folder_path)
    return folder_path

# Modified save_image function
def save_image(img):
    folder_path = create_output_folders()
    unique_name = str(uuid.uuid4()) + ".png"
    file_path = os.path.join(folder_path, unique_name)
    img.save(file_path)
    return file_path

# Modified randomize_seed_fn function
def randomize_seed_fn(seed: int, randomize_seed: bool) -> int:
    if randomize_seed:
        seed = random.randint(0, MAX_SEED)
    return seed

# Modified generate function to include batch count
def generate(
    prompt: str,
    negative_prompt: str = "",
    style: str = DEFAULT_STYLE_NAME,
    use_negative_prompt: bool = False,
    seed: int = 0,
    width: int = 1024,
    height: int = 1024,
    schedule: str = DEFAULT_SCHEDULE_NAME,
    dpms_guidance_scale: float = 4.5,
    sas_guidance_scale: float = 3,
    dpms_inference_steps: int = 20,
    sas_inference_steps: int = 25,
    randomize_seed: bool = False,
    batch_count: str = "1",
    use_resolution_binning: bool = True,
    progress=gr.Progress(track_tqdm=True),
):
    image_paths = []
    print(f"batch_count {batch_count}")
    batch_count_int = int(batch_count)
    for _ in range(batch_count_int):
        seed = int(randomize_seed_fn(seed, randomize_seed))
        generator = torch.Generator().manual_seed(seed)

        if schedule == 'DPM-Solver':
            if not isinstance(pipe.scheduler, DPMSolverMultistepScheduler):
                pipe.scheduler = DPMSolverMultistepScheduler()
            num_inference_steps = dpms_inference_steps
            guidance_scale = dpms_guidance_scale
        elif schedule == "SA-Solver":
            if not isinstance(pipe.scheduler, SASolverScheduler):
                pipe.scheduler = SASolverScheduler.from_config(pipe.scheduler.config, algorithm_type='data_prediction', tau_func=lambda t: 1 if 200 <= t <= 800 else 0, predictor_order=2, corrector_order=2)
            num_inference_steps = sas_inference_steps
            guidance_scale = sas_guidance_scale
        else:
            raise ValueError(f"Unknown schedule: {schedule}")

        if not use_negative_prompt:
            negative_prompt = None  # type: ignore
        prompt, negative_prompt = apply_style(style, prompt, negative_prompt)

        images = pipe(
            prompt=prompt,
            width=width,
            height=height,
            guidance_scale=guidance_scale,
            num_inference_steps=num_inference_steps,
            generator=generator,
            num_images_per_prompt=NUM_IMAGES_PER_PROMPT,
            use_resolution_binning=use_resolution_binning,
            output_type="pil",
        ).images

        image_paths.extend([save_image(img) for img in images])

    return image_paths, seed
```

The error of above code is shown below

```
To create a public link, set `share=True` in `launch()`.
batch_count 1
Traceback (most recent call last):
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\queueing.py", line 427, in call_prediction
    output = await route_utils.call_process_api(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\blocks.py", line 1484, in process_api
    result = await self.call_function(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\blocks.py", line 1106, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\utils.py", line 665, in wrapper
    response = f(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\utils.py", line 665, in wrapper
    response = f(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\app_8bit.py", line 176, in generate
    images = pipe(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\diffusers\pipelines\pixart_alpha\pipeline_pixart_alpha.py", line 731, in __call__
    ASPECT_RATIO_1024_BIN if self.transformer.config.sample_size == 128 else ASPECT_RATIO_512_BIN
AttributeError: 'NoneType' object has no attribute 'config'
Traceback (most recent call last):
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\queueing.py", line 427, in call_prediction
    output = await route_utils.call_process_api(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\blocks.py", line 1484, in process_api
    result = await self.call_function(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\blocks.py", line 1106, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\utils.py", line 665, in wrapper
    response = f(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\utils.py", line 665, in wrapper
    response = f(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\app_8bit.py", line 176, in generate
    images = pipe(
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\diffusers\pipelines\pixart_alpha\pipeline_pixart_alpha.py", line 731, in __call__
    ASPECT_RATIO_1024_BIN if self.transformer.config.sample_size == 128 else ASPECT_RATIO_512_BIN
AttributeError: 'NoneType' object has no attribute 'config'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\queueing.py", line 472, in process_events
    response = await self.call_prediction(awake_events, batch)
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\gradio\queueing.py", line 436, in call_prediction
    raise Exception(str(error) if show_error else None) from error
Exception: None
```


@sayakpaul sorry i had given incorrect code above (it was working one) fixed now

The error happens on this file : https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py

Line 731

```
        if use_resolution_binning:
            aspect_ratio_bin = (
                ASPECT_RATIO_1024_BIN if self.transformer.config.sample_size == 128 else ASPECT_RATIO_512_BIN
            )
            orig_height, orig_width = height, width
            height, width = self.classify_height_width_bin(height, width, ratios=aspect_ratio_bin)
```

When use_resolution_binning is disabled a new error occurs

```
  File "G:\pixArt installer\PixArt-alpha\venv\lib\site-packages\diffusers\pipelines\pixart_alpha\pipeline_pixart_alpha.py", line 790, in __call__
    latent_channels = self.transformer.config.in_channels
AttributeError: 'NoneType' object has no attribute 'config'
```

Line 790

```
        # 5. Prepare latents.
        latent_channels = self.transformer.config.in_channels
        latents = self.prepare_latents(
```

**So basically when I try 8 bit there is no self.transformer .** 

ok now i understand my error

you are loading without transformer computing latents deleting and reloading 

wow this would be so slow i assume

**also your example code deletes and loads everything for every image is that really expected?** 

it would be immensely slow

====

ok i made it work but still using 13,989 VRAM compared to 16,222 VRAM  :)

I guess only way to work it with low VRAM is keep deleting and loading again which would be super slow. thank you

ok here comparison

16 bit 

![image](https://github.com/huggingface/transformers/assets/19240467/e740037f-4d3c-4dbb-bdf2-5e0273df397d)

8 bit text encoder

![image](https://github.com/huggingface/transformers/assets/19240467/c1869d8c-6e60-4734-b92c-ad95dced3205)

16 bit 512 model

![image](https://github.com/huggingface/transformers/assets/19240467/a52ee303-8796-4002-a730-298f955275a6)

8 bit 512 model

![image](https://github.com/huggingface/transformers/assets/19240467/925b8ea4-6520-484b-88a3-a370356cde40)

512 model and 1024 model VRAM usages are almost same :(


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.