(picture made with V3)

Hi! Could you send screen of the errors?

Hi, certainly. I have this:

#@title Installation for Kandinsky 3.1
!pip install git+https://github.com/huggingface/diffusers.git
!pip install transformers
!pip install torch
!pip install einops
!pip install accelerate
!pip install sentencepiece

Then:

#@title Downloads and Initialization for Kandinsky 3.1
import torch
from kandinsky3 import get_T2I_Flash_pipeline

Setting up device and data type mappings as per Kandinsky 3.1 requirements

device_map = {'cuda:0': 'unet', 'cuda:0': 'text_encoder', 'cuda:0': 'movq'}
dtype_map = {
'unet': torch.float32,
'text_encoder': torch.float16,
'movq': torch.float32,
}

Initialize the Kandinsky 3.1 pipeline with the specified device and data type mappings

t2i_pipe = get_T2I_Flash_pipeline(
device_map=device_map,
dtype_map=dtype_map
)

Example to ensure the pipeline is correctly initialized

test_prompt = "A cute corgi lives in a house made out of sushi."
result_image = t2i_pipe(test_prompt).images[0]
result_image.show() # Display the generated image
print("Pipeline initialized and ready to generate images.")

That goes:

ModuleNotFoundError Traceback (most recent call last)
in <cell line: 3>()
1 #@title Downloads and Initialization for Kandinsky 3.1
2 import torch
----> 3 from kandinsky3 import get_T2I_Flash_pipeline
4
5 # Setting up device and data type mappings as per Kandinsky 3.1 requirements

ModuleNotFoundError: No module named 'kandinsky3'

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

We could use a simple colab like this one https://colab.research.google.com/github/poomshift/PromptAlchemist_Playground-2.5-colab/blob/main/Playground_2_5_colab.ipynb to "begin" with many options, like Dynamic Prompting, .csv reading, etc.

I think you need to do the following

!git clone https://github.com/ai-forever/Kandinsky-3.git
!cd Kandinsky-3

It will provide you with access to kandinsky 3. The second option is to convert weights of kandinsky to diffusers format using this code https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/kandinsky3/convert_kandinsky3_unet.py

And then download them to KandinskyPipeline in diffusers

Oh waow, thank you. I'm on it!

My cells are

#@title Installation for Kandinsky 3.1
!pip install git+https://github.com/huggingface/diffusers.git
!pip install transformers
!pip install torch
!pip install einops
!pip install accelerate
!pip install sentencepiece
!pip install omegaconf # Install the missing dependency
!git clone https://github.com/ai-forever/Kandinsky-3.git

AND

#@title Downloads and Initialization for Kandinsky 3.1

import sys
import os
import torch

Add Kandinsky-3 directory to the Python path

sys.path.append(os.path.join(os.getcwd(), "Kandinsky-3"))

from kandinsky3 import get_T2I_Flash_pipeline

Setting up device and data type mappings as per Kandinsky 3.1 requirements

device_map = {
'unet': torch.device('cuda:0'),
'text_encoder': torch.device('cuda:0'),
'movq': torch.device('cuda:0')
}

dtype_map = {
'unet': torch.float32,
'text_encoder': torch.float16,
'movq': torch.float32,
}

Initialize the Kandinsky 3.1 pipeline with the specified device and data type mappings

t2i_pipe = get_T2I_Flash_pipeline(
device_map=device_map,
dtype_map=dtype_map
)

Example to ensure the pipeline is correctly initialized

test_prompt = "A cute corgi lives in a house made out of sushi."
result_image = t2i_pipe(test_prompt).images[0]
result_image.show() # Display the generated image
print("Pipeline initialized and ready to generate images.")

Then I have this one:

#@title Downloads and Initialization for Kandinsky 3.1

import sys
import os
import torch

Add Kandinsky-3 directory to the Python path

sys.path.append(os.path.join(os.getcwd(), "Kandinsky-3"))

from kandinsky3 import get_T2I_Flash_pipeline

Setting up device and data type mappings as per Kandinsky 3.1 requirements

device_map = {
'unet': torch.device('cuda:0'),
'text_encoder': torch.device('cuda:0'),
'movq': torch.device('cuda:0')
}

dtype_map = {
'unet': torch.float32,
'text_encoder': torch.float16,
'movq': torch.float32,
}

Initialize the Kandinsky 3.1 pipeline with the specified device and data type mappings

t2i_pipe = get_T2I_Flash_pipeline(
device_map=device_map,
dtype_map=dtype_map
)

Example to ensure the pipeline is correctly initialized

test_prompt = "A cute corgi lives in a house made out of sushi."
result_image = t2i_pipe(test_prompt).images[0]
result_image.show() # Display the generated image
print("Pipeline initialized and ready to generate images.")

It works but crashes :

kandinsky3_flash.pt: 100%
 12.2G/12.2G [02:43<00:00, 103MB/s]
Fetching 13 files: 100%
 13/13 [02:29<00:00, 24.22s/it]
(…)ights/flan_ul2_encoder/added_tokens.json: 100%
 665/665 [00:00<00:00, 48.1kB/s]
weights/flan_ul2_encoder/config.json: 100%
 790/790 [00:00<00:00, 68.1kB/s]
(…)ul2_encoder/pytorch_model.bin.index.json: 100%
 26.5k/26.5k [00:00<00:00, 2.40MB/s]
projection_inpainting.pt: 100%
 67.1M/67.1M [00:01<00:00, 84.3MB/s]
projection_flash.pt: 100%
 67.1M/67.1M [00:01<00:00, 59.8MB/s]
pytorch_model-00003-of-00004.bin: 100%
 9.87G/9.87G [02:02<00:00, 84.0MB/s]
pytorch_model-00002-of-00004.bin: 100%
 10.0G/10.0G [02:28<00:00, 67.9MB/s]
pytorch_model-00001-of-00004.bin: 100%
 9.92G/9.92G [02:16<00:00, 86.4MB/s]
(…)flan_ul2_encoder/special_tokens_map.json: 100%
 2.20k/2.20k [00:00<00:00, 57.5kB/s]
pytorch_model-00004-of-00004.bin: 100%
 5.10G/5.10G [01:08<00:00, 50.7MB/s]
projection.pt: 100%
 67.1M/67.1M [00:01<00:00, 47.2MB/s]
spiece.model: 100%
 792k/792k [00:00<00:00, 13.8MB/s]
(…)s/flan_ul2_encoder/tokenizer_config.json: 100%
 2.35k/2.35k [00:00<00:00, 134kB/s]
movq.pt: 100%
 1.08G/1.08G [00:13<00:00, 82.1MB/s]
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 100%
 4/4 [01:11<00:00, 17.98s/it]
100%|██████████| 4/4 [00:01<00:00, 2.86it/s]

OutOfMemoryError Traceback (most recent call last)
in <cell line: 33>()
31 # Example to ensure the pipeline is correctly initialized
32 test_prompt = "A cute corgi lives in a house made out of sushi."
---> 33 result_image = t2i_pipe(test_prompt).images[0]
34 result_image.show() # Display the generated image
35 print("Pipeline initialized and ready to generate images.")

12 frames
/content/Kandinsky-3/kandinsky3/movq.py in forward(self, f, zq)
34 if self.add_conv:
35 zq = self.conv(zq)
---> 36 norm_f = norm_f * self.conv_y(zq) + self.conv_b(zq)
37 return norm_f
38

OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 39.56 GiB of which 210.81 MiB is free. Process 47055 has 39.35 GiB memory in use. Of the allocated memory 38.68 GiB is allocated by PyTorch, and 161.31 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Well, I got a A100 with high memory, and get a out of memory crash

I give you the URL, you can see the code and the messages.

https://colab.research.google.com/drive/1zwVefrDxfSgrFODDhspIeQ0bMtaVOXnR?usp=sharing

I would love so much to make a colab for 3.1! If you find some time...

thank you!!