Improve support for image generation with Chameleon & Anole
Author: leloykun
Pull request: yes

# What does this PR do?

- Adds modelling for the `VQVAE` decoder & also includes it in the conversion script.
  - I've also uploaded the converted model here: https://huggingface.co/leloy/Anole-7b-v0.1-hf
- Adds support for decoding the BPE tokens -> discrete image tokens ->  pixel values
- Moves masking of image tokens in text-only generation mode to a LogitsProcessor.
- Adds masking of non-image tokens for image-only generation mode.
- ~~Reimplements Chameleon's FSM to be more compatible with Transformers and Outlines (for structured generation)~~
  - ~~This PR _will not_ add the FSM, but instead just makes it easier for external libraries like Outlines & MMSG to integrate with Transformers to add the interleaved generation mode back in~~
  - Reimplements Chameleon's Finite-State Machine that it uses to dynamically switch between text- and image-generation modes as Logits Processors. We can now support interleaved text-image generation natively.

Required TODOs:
- [ ] Improve docs
  - [x] Write docs for image-only generation with Chameleon/Anole
    - [x] Provide minimal example on how to use
    - [x] Remove the need for passing `max_length` or `max_new_tokens` on image-only generation mode
    - [x] Force the model to generate at least one image on image-only generation mode. Chameleon doesn't officially support image-generation yet so it just immediately closes `begin-image-token`s with either an `end-image-token` or an EOS token. And finetunes like Anole haven't fully removed this issue yet so they occasionally still does that.
  - [ ] Write docs for interleaved text-image generation
    - [x] Provide minimal example on how to use
    - [ ] Show how to split token sequences by modality in the example for interleaved text-image generation
  - [ ] Improve docs (as comments) for newly-added logits processors
  - [ ] Add sample usage for newly-added logits processors 
- [ ] Add tests
  - [x] Add test for image postprocessing
  - [ ] Add tests for each generation mode
    - [x] Add tests for `text-only` generation mode
    - [ ] Add tests for `image-only` generation mode
      - Note: don't rely on hashing pytorch tensors to compare arrays
      - [ ] Add test where `max_new_tokens` is unset on `image-only` generation mode
    - [ ] Add tests for `interleaved-text-image` generation mode
    - [ ] Add tests for `unrestricted` generation mode
    - [ ] Add tests for invalid generation mode
  - [ ] Add tests for VQVAE decoder
  - [ ] Add tests for each newly-added logits processors
  - [x] Add tests for multi-GPU model sharding
- [x] Improve modelling of VQVAE encoder & decoder
- [x] VQVAE: dynamically compute `quant_state_flattened_dims` which scales with the resolution instead of hardcoding it in the configs
- [x] Improve postprocessing: only accept and return pytorch tensors
- [ ] Logits processors
  - [ ] Add new logits processors to import structure

Optional TODOs or for future PRs:
- [ ] Run a hyperparameter search for image generation
- [x] Fix bugs caused by sharding the model into multiple GPUs
- [ ] Implement features that were in the [Chameleon paper](https://arxiv.org/abs/2405.09818v1) but are not crucial here
  - [ ] Implement support for [z-loss](https://arxiv.org/abs/2309.14322)
  - [ ] Implement Classifier Free Guidance. The Lumina team claims that "though it is not that indispensable, its impact is still significant" https://github.com/Alpha-VLLM/Lumina-mGPT/issues/13#issuecomment-2293176135
    - [ ] Implement Classifier Free Guidance for image-only generation
    - [ ] Implement Classifier Free Guidance for interleaved text-image generation
- [ ] Refactor VQVAE (sub-)modules
  - [ ] Convert `mid`, `down`, and `up` blocks into explicit subclasses of `nn.Module()` (as suggested by @amyeroberts)
  - [ ] Make it clearer which attention types are allowed in the VQVAE (sub-)modules. Tho Chameleon currently only supports "vanilla" attention.
- [ ] Implement support for other finetunes of Chameleon
  - [ ] Implement support for [Lumina-mGPT](https://github.com/Alpha-VLLM/Lumina-mGPT) as suggested by @minostauros 

Links:
- [Multimodal Structured Generation](https://github.com/leloykun/mmsg): Repo that supports interleaved text-image generation and interleaved multimodal structured generation
- [Official Anole repo by GAIR-NLP team](https://github.com/GAIR-NLP/anole)
- [Official Anole model by GAIR-NLP team](https://huggingface.co/GAIR/Anole-7b-v0.1)
- [Transformers-compatible Anole with image generation support](https://huggingface.co/leloy/Anole-7b-v0.1-hf)

(partially) Implements # (issue)
- https://github.com/GAIR-NLP/anole/issues/18

@ArthurZucker @zucchini-nlp @JoyBoy-Su

Comments:
@zucchini-nlp @ArthurZucker this should now be ready for review

the test errors seem to be related to huggingface_hub & bert & I'm now sure how they relate to this PR.

@zucchini-nlp this should now be finished I think

the failing test seems to have been caused by the issue here: https://github.com/huggingface/transformers/pull/32094 
which is unrelated to this PR

Hi @leloykun , thank you for aweesome work!

I run [your example](https://github.com/leloykun/transformers/blob/fc--anole/docs/source/en/model_doc/chameleon.md#text-to-image-generation), but all the output are black... Is there anything missing here?
![image](https://github.com/user-attachments/assets/1246b85a-69a4-4088-857c-0a20e4544773)

Thank you!!


Thanks for the feedback, @thaoshibe!

Apparently, we just need to enable sampling during generation (by passing `do_sample=True` to `.generate`). If I'm not mistaken, this is because most of the image tokens during training were for "empty" patches. So, greedy decoding of image tokens wouldn't work well.

I've also just updated the docs. Please let me know if you encounter any more issues.

One more thing: loading the model in `bfloat16` (the dtype used for finetuning Anole) also seems to improve generation. See:

```python
model = ChameleonForConditionalGeneration.from_pretrained(
            model_id,
            torch_dtype=torch.bfloat16,
            low_cpu_mem_usage=True,
            attn_implementation="flash_attention_2",
            device_map="auto",
)
```

We can actually add those to generation config after uploading model to the hub. We did same for official Chameleon, it performs best with `do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.2`

Same for bf16, can go the the docs in usage tips section. Then we can change all example snippets to bf16

![image](https://github.com/user-attachments/assets/664b7bc3-324f-4edd-a249-0d086a067f5e)

Got it -- Thank youu @leloykun -- I ran your code and I got the correct output :D

> We can actually add those to generation config after uploading model to the hub. We did same for official Chameleon, it performs best with `do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.2`
> 
> Same for bf16, can go the the docs in usage tips section. Then we can change all example snippets to bf16

@zucchini-nlp , increasing the repetition penalty might not be good for image generation cuz a lot of the image tokens _are_ repeating (e.g. snow tokens when generating a snowman).

I'll only include the others for now then run a hp search

The test errors are all huggingface hub related hnng

@leloykun Just triggered a re-run. Hopefully just transient issues! 

> Some comments about running the example in the model doc.

Thanks @minostauros! I'll make sure to fix these in the next commit & tag you when it's ready

Supporting [Lumina-mGPT](https://github.com/Alpha-VLLM/Lumina-mGPT) may be the next PR!

> Supporting [Lumina-mGPT](https://github.com/Alpha-VLLM/Lumina-mGPT) may be the next PR!

dang, this looks cool

does it support interleaved text-image generation too?

> does it support interleaved text-image generation too?

Sadly none of their examples show interleaved text-image generation while it is mentioned the model is trained with interleaved data.

<img width="700" alt="image" src="https://github.com/user-attachments/assets/16566b36-5cb3-499e-87e1-1a8f58c74b63">

One interesting approach is the prompt is aware of image resolutions.

<img width="500" alt="image" src="https://github.com/user-attachments/assets/8b158dd0-da1b-4f7b-8cc4-7feba73e9fc8">

And here's a unique generation parameter handling of Lumina-mGPT
<img width="600" alt="image" src="https://github.com/user-attachments/assets/1623084b-d582-45ed-b78f-e35a3d1da4aa">


hmmm

one crucial difference is that Chameleon uses classifier free guidance while this doesn't

I'll look into implementing it, but I think I'm gonna need help with that

Hey, we have a CFG in transformers which was added exclusively for audio-models. 

https://github.com/huggingface/transformers/blob/48101cf8d127bbf22d751c7df118a6ce357e2e27/src/transformers/generation/logits_process.py#L2062-L2076

Also, there is a slightly different implementation for language models, I believe this second one should be compatible with Chameleon

https://github.com/huggingface/transformers/blob/48101cf8d127bbf22d751c7df118a6ce357e2e27/src/transformers/generation/logits_process.py#L2171-L2183

Hi, I find a bug, if u use ChameleonForConditionalGeneration and calculate ce loss, u should use new special image tokens encoded by vqmodel to update the labels for ce loss calculation~ @leloykun 

> Hi, I find a bug, if u use ChameleonMoeForConditionalGeneration and calculate ce loss, u should use new special image tokens encoded by vqmodel to update the labels for ce loss calculation~ @leloykun 

That doesn't sound right...

We should calculate the CE loss using the BPE-compatible tokens (i.e. the tokens compatible with Chameleon's tokenizer). That's because those are the outputs of the decoder model.

Pls check the img2bpe & bpe2img converter utils

> > Hi, I find a bug, if u use ChameleonForConditionalGeneration and calculate ce loss, u should use new special image tokens encoded by vqmodel to update the labels for ce loss calculation~ @leloykun
> 
> That doesn't sound right...
> 
> We should calculate the CE loss using the BPE-compatible tokens (i.e. the tokens compatible with Chameleon's tokenizer). That's because those are the outputs of the decoder model.
> 
> Pls check the img2bpe & bpe2img converter utils

Thanks for reply. 
Yes, we should calculate CE loss using BPE tokens instead of "image" (id is 8711), but we only can get BPE token in the ChameleonModel and not return the input ids in the output to update the labels in ChameleonForConditionalGeneration. So my suggestion is that when using ChameleonForConditionalGeneration, we can get BPE tokens encoded by vqmodel in ChameleonForConditionalGeneration forward function to update input_ids and labels instead of in ChameleonModel

> > > Hi, I find a bug, if u use ChameleonForConditionalGeneration and calculate ce loss, u should use new special image tokens encoded by vqmodel to update the labels for ce loss calculation~ @leloykun
> > 
> > That doesn't sound right...
> > 
> > We should calculate the CE loss using the BPE-compatible tokens (i.e. the tokens compatible with Chameleon's tokenizer). That's because those are the outputs of the decoder model.
> > 
> > Pls check the img2bpe & bpe2img converter utils
> 
> Thanks for reply. 
> Yes, we should calculate CE loss using BPE tokens instead of "image" (id is 8711), but we only can get BPE token in the ChameleonModel and not return the input ids in the output to update the labels in ChameleonForConditionalGeneration. So my suggestion is that when using ChameleonForConditionalGeneration, we can get BPE tokens encoded by vqmodel in ChameleonForConditionalGeneration forward function to update input_ids and labels instead of in ChameleonModel

Apologies, I'm a bit confused

Can you share your code so we can debug it together?

> > > > Hi, I find a bug, if u use ChameleonForConditionalGeneration and calculate ce loss, u should use new special image tokens encoded by vqmodel to update the labels for ce loss calculation~ @leloykun
> > > 
> > > 
> > > That doesn't sound right...
> > > We should calculate the CE loss using the BPE-compatible tokens (i.e. the tokens compatible with Chameleon's tokenizer). That's because those are the outputs of the decoder model.
> > > Pls check the img2bpe & bpe2img converter utils
> > 
> > 
> > Thanks for reply.
> > Yes, we should calculate CE loss using BPE tokens instead of "image" (id is 8711), but we only can get BPE token in the ChameleonModel and not return the input ids in the output to update the labels in ChameleonForConditionalGeneration. So my suggestion is that when using ChameleonForConditionalGeneration, we can get BPE tokens encoded by vqmodel in ChameleonForConditionalGeneration forward function to update input_ids and labels instead of in ChameleonModel
> 
> Apologies, I'm a bit confused
> 
> Can you share your code so we can debug it together?

Hi, I have submitted a PR to your repository, you can refer to and modify this code for free

Hi @YeLuoSuiYou! Thanks for the PR!

My current understanding of the matter is:
1. Internally, if `pixel_values` is not `None`, then we use the `vqmodel` to tokenize them and add the tokens to the `input_ids`
2. But we never touch the `labels`, regardless of whether `pixel_values` is `None` or not. We keep it as-is during training. 

(2) isn't actually a bug and your fix doesn't fit the library imo. We don't want the inputs and the outputs to interact _before the calculating the loss_ in order to minimize bugs. cc @zucchini-nlp

What you should do, instead, is to pass `labels` _with_ the image tokens already added to it. Here's what I do in my finetuning scripts:
1. Pass text and images to the processor. Get `input_ids` and `pixel_values` in return.
2. Clone the `input_ids` as `labels` (i.e. `labels = input_ids.clone()`)
3. Use the `vqmodel` to tokenize the `pixel_values`
4. Add the tokens to `labels`
5. Pass `input_ids`, `pixel_values`, & `labels` to the model

> What you should do, instead, is to pass labels with the image tokens already added to it.

I couldn't locate the PR but this is exactly what is expected. We (transformers) return `input_ids` already expanded to account for the image tokens and the user has only to clone input-ids and mask pad-token-ids, similar to Language Modeling task. And Chameleon, as one of the latest VLMs added to the library, follows this. Older VLMs like LLaVA are still in progress and might need more intervention from the user

@leloykun btw, seems like the most important thing left to add on this PR is the tests. Let me know if you need any help with that, would be super nice to have this PR merged soon :)

> Hi @YeLuoSuiYou! Thanks for the PR!
> 
> My current understanding of the matter is:
> 
> 1. Internally, if `pixel_values` is not `None`, then we use the `vqmodel` to tokenize them and add the tokens to the `input_ids`
> 2. But we never touch the `labels`, regardless of whether `pixel_values` is `None` or not. We keep it as-is during training.
> 
> (2) isn't actually a bug and your fix doesn't fit the library imo. We don't want the inputs and the outputs to interact _before the calculating the loss_ in order to minimize bugs. cc @zucchini-nlp
> 
> What you should do, instead, is to pass `labels` _with_ the image tokens already added to it. Here's what I do in my finetuning scripts:相反，您应该做的是传递已添加图像标记_的_`labels` 。这是我在微调脚本中所做的事情：
> 
> 1. Pass text and images to the processor. Get `input_ids` and `pixel_values` in return.
> 2. Clone the `input_ids` as `labels` (i.e. `labels = input_ids.clone()`)
> 3. Use the `vqmodel` to tokenize the `pixel_values`
> 4. Add the tokens to `labels`
> 5. Pass `input_ids`, `pixel_values`, & `labels` to the model

Thanks for reply, I got it

@leloykun feel free to ping @zucchini-nlp again for a review, I'll do the final one afterwards!

@leloykun hey, I can take over and write tests if you are busy, so we can merge faster. I think everything else was approved earlier 

Hi @zucchini-nlp ! I'd really appreciate it as I don't see myself being able to continue working on it for the next few weeks.

@ArthurZucker ready for review! Added some more tests and removed unused logits processor to reduce maintainment burden. Actually I believe the generation might be done with simple prefix constraint and one new logit processor but didn't have time to check it out. Should be very similar to Emu3 inference

Reviewing now!


Yep, we can make a new model called Anole using modular. When I get some time, I will find a way to make controlled generation with prefix constraints and thus remove the newly added logits processors. Hopefully it works for simple cases, and for interleaving we'll point to MMSG


Hi, 

I tried the [example](https://github.com/leloykun/transformers/blob/fc--anole/docs/source/en/model_doc/chameleon.md#text-to-image-generation) and the generated response_ids contains the eoi_token_id (8196) in the middle instead of only at the end. I experimented with different hyperparameters, but the issue persists. I believe this caused the error:
 ```
Traceback (most recent call last):
  File "/home/usr/anaconda3/envs/dir/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/usr/anaconda3/envs/dir/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/usr/dir/evaluation/t2i_pretrained.py", line 156, in <module>
    pixel_values = model.decode_image_tokens(response_ids[:, 1:-1])
  File "/home/usr/dir/transformers/src/transformers/models/chameleon/modeling_chameleon.py", line 2148, in decode_image_tokens
    return self.model.decode_image_tokens(bpe_tokens)
  File "/home/usr/dir/transformers/src/transformers/models/chameleon/modeling_chameleon.py", line 1547, in decode_image_tokens
    return self.vqmodel.decode(image_tensor)
  File "/home/usr/dir/transformers/src/transformers/models/chameleon/modeling_chameleon.py", line 1209, in decode
    hidden_states = self.post_quant_conv(codebook_entry)
  File "/home/usr/anaconda3/envs/dir/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/usr/anaconda3/envs/dir/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/usr/anaconda3/envs/dir/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/usr/anaconda3/envs/dir/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [1,0,0], thread: [92,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
```

The response_ids: 

```
tensor([[5228, 6770, 5153, 5130, 5988, 8010, 5130, 1837, 4021,   36, 3146, 5988,
         2984, 5152, 2974, 2802, 8064,  468, 2179, 5988, 1606, 2493, 1328, 5988,
         5988,  607, 7236, 2487, 5573, 2189, 5204, 4599, 2135, 1969, 7877, 3245,
         4345,   46, 4852, 3536, 5408, 5048, 3372, 5616, 4168, 8093, 7269, 3372,
         7753, 4933, 7201, 1260, 2190, 1332, 7711, 2519, 5563, 2231, 4565, 7090,
         2145, 4947, 6946, 2183, 7712, 2890, 3985, 6710, 2433, 5961, 2863, 4749,
         3364, 7004, 3534, 7923, 1576,  378, 3817, 8017, 6580, 2696, 4867, 7624,
         4009, 1962, 1035, 1558, 1656, 7012,  281, 6265, 5478,  850, 1682, 6573,
         7792, 5534,  900, 5635, 7989, 3482, 7314, 5040, 7433, 5653, 6780,  747,
         3847, 6893, 3481, 5124,  328, 6814, 5758, 3637, 6581, 5681, 1072, 8044,
         5711, 1240, 5652, 7417, 1317, 4676,  330, 7278, 3554, 5394, 2918, 1810,
         4078, 4625, 5911, 4096, 1503, 2908, 6003, 6440, 4469, 7055, 4283, 1365,
           39,  964, 2162, 4821, 6564, 4550, 6849,  159, 3645,  469, 7668, 4008,
         6754, 5954, 3519, 7487, 7838, 4476, 7794,  488, 4472,  453, 7489, 7198,
         6372, 1125, 2787, 5058, 3348, 4402, 3410, 2966, 2269,  610, 2823, 5556,
         3311, 6290, 1215, 6906, 6584, 2870, 3951, 4384, 2500,  712, 3842, 4503,
         3543, 7001, 3017, 1578, 7406, 7748,  200, 4811,  177, 4849, 6482, 5062,
         2325, 2681, 3412, 5328, 5598, 6273, 4666,   79, 5483, 4440, 2501, 6015,
         7518,  379, 2895, 6004, 6869, 6695, 4134, 3823, 3899, 6057, 5496,  903,
         5796, 4523, 3163,  559, 4955, 2885, 8113, 4818, 1198, 2153, 6980, 6853,
         1657, 4383, 2047, 5611, 3686, 5315,  381, 6462, 6664, 3144, 8140, 4313,
          439, 6488, 7124, 4038, 4970, 2557, 4590, 4581, 4686, 6718, 6143, 2456,
         5453, 4949, 6248, 5708,  725, 7299, 3255, 7069, 2371, 2976,  689, 8065,
         4317, 5467, 7065, 1390, 5834,  548,  474, 4146, 1278, 5239, 6332, 2171,
         6557, 3907, 5645, 3656, 6833, 6860, 7044, 7020, 7879, 1930, 2290, 5419,
         6634, 7485, 4258, 4159, 7174, 5989, 7536, 8148, 1213, 4307, 7815, 4986,
         3707, 7615, 3369, 1440, 7154, 7893, 3237, 2380, 3156, 5592,  805, 6778,
         2941, 1526, 6820, 7300, 1272, 4189, 4873, 7732, 7380,  587, 4790, 7714,
         6155, 2001, 3706, 7547, 3345, 1201, 3605, 7229, 7444, 2766, 7266, 4695,
          350, 2953, 3690, 7252, 7431, 6557,  863, 2680, 6486, 5662,  990, 7372,
         6896, 4123, 4591, 4582, 7414,  862, 4837,  285, 1702, 1712, 4120, 6620,
         6669, 6133, 4284, 6577, 2330,  729,  601, 8021, 3447, 8160,  326, 7101,
         4950, 1463, 7200, 5296, 7552,  552,   75, 6260, 6302, 2268, 3891, 7626,
         7595, 5593, 2572, 7307,  519, 4727,  973, 5488, 2865, 3967,  357, 7402,
         6252, 5308, 7412, 3960,  467, 5774,  701, 6706, 5745, 6277, 1935,  714,
         7720, 6145, 7902, 8081, 1912, 1298, 1133, 3123, 1593, 3651, 2336, 4462,
         7036,   26, 1664, 7524, 5369, 1088,  275, 2277, 2424, 2537, 6309, 3119,
         1300,  200, 6782,  427, 4691, 4707,  854, 3996, 8082, 7334, 7238, 1561,
          117, 4077, 8069, 8062, 3650, 3459, 5375, 5615, 3248, 6752, 3653,  363,
           66, 6876, 6952, 1506, 5242, 6256,  770, 3567, 5929, 4659, 6094, 6162,
         6586, 7268, 1472, 2371, 4911, 6305, 4354, 3005, 6286, 7032, 7550, 1723,
         3785, 3644, 3218,  416, 1541, 4456, 7990, 4596,  693, 5255, 7691, 8111,
         5313, 5051, 6965, 2607, 5252, 1479, 5402, 4891, 4806, 2187, 5342, 6273,
         2827, 4124, 7971, 1003, 1343,  386, 6009, 4784, 3033, 4983,  302, 4407,
         4177,  588,  361,  536,  249, 2784, 6613, 5320, 3629,  816, 6406, 2229,
         3319, 4581, 6742, 6291, 6491, 2182, 5683,   77,   53, 5559, 1340, 6405,
         7395, 5730, 6883, 5366,  792, 6534, 2132, 4086, 5144, 3819, 5002, 2228,
         6725, 4026, 6380,  974, 3596,  735, 5936, 3167, 1306, 7406, 3849, 5513,
         1659,  750, 4304, 5486, 2566,  611, 5757, 4207, 6028, 1613, 1919, 2645,
         1613,  362, 6548, 1930, 2036, 7108, 5161, 3313,  997, 2707,  869, 6922,
         4725, 2787, 4970,  685, 2105, 1245, 5654, 1089, 1424, 5775, 6369, 1994,
         4588, 6203, 2730, 1641, 7600, 5441, 3147, 2055, 6066, 4819, 3025, 5547,
         4478, 4029, 3110, 1209, 6993, 1537, 2492, 5034, 2869, 2443, 6435, 7875,
         2046, 6004, 3772, 5518, 1018,  685, 3279, 1337, 2287, 4187,  420, 1333,
         1648, 4586, 5278, 8177, 2805, 2482, 7633, 3446, 1385, 3863,  604, 8048,
          238, 4610, 2217, 7819,  182, 2301, 6828, 3589, 4099, 4801, 4709, 1754,
         6898, 2868, 8060, 7954, 5902, 7100, 2717, 5344, 2818, 6645, 6851, 4655,
         5899, 6810, 6473, 7443, 1470, 1647,  435,  996, 5010, 5974, 3578,  780,
         5427, 2288, 4432, 4672, 1858, 1084, 1093, 4859, 7752, 6271, 3188, 3874,
         4102, 6705, 2911, 7162, 8102, 3777, 5410, 6383, 7319, 7956, 6388, 3768,
         7320, 6154, 6327, 2965, 1757, 1178, 5813, 2307, 1363, 1191,  702, 1275,
         7183, 4052, 1181,  221, 1354, 3983, 8067, 6715,  754,  956, 3661,  956,
         4259, 6849, 4729, 1910, 5953, 4804, 6515, 6698, 6522, 6440, 2872, 5556,
         1703, 6220, 6032, 4128, 1454, 4475, 7235, 8148, 1789, 3320, 7647, 4976,
          597, 2541, 5924, 2202,  350, 5291, 7044, 1106, 1213, 4195, 1213,  902,
         4488, 5448,  445, 1596, 8169, 3665, 7793, 3584, 4164, 6090, 4237, 4957,
         3949, 6331,  625,  953, 7834, 5949, 7726, 3373, 6723, 4398,  554, 6864,
         1307, 1590, 8045, 7237, 5952,  412,  768, 1504,  543, 7928, 1611, 4307,
         7815, 6701, 7815, 4199, 7815, 2658, 5419, 4950, 6939, 6813, 5312, 1724,
         1263, 3601, 6541, 7544, 1861, 2455, 3356, 6399, 7102, 3256, 2184, 3159,
         5353, 7254, 2084, 2833,   19, 2778, 2253, 3396,  384, 4554, 7575, 1100,
         2613,  407, 1513, 3834,   90,  181,  525, 2590,  384, 8196, 3309, 7640,
         7046, 2886, 1584, 4214, 5334, 3631, 7227, 4082, 6784, 4792, 1062, 1356,
         6861,  397, 6130, 5140, 2004, 2962, 8015, 4276, 7363, 5154, 8147,  922,
         7287, 1292, 6114, 4593, 6078, 6792, 7411, 2804, 4094, 6977,  355,  635,
         5932, 5465, 5946, 4033, 3775, 7110,  873, 7082, 1828, 8165, 2884, 5878,
         5589,   94, 5906, 5481,  743, 6377, 2961,  269, 1610, 3296,  297, 2585,
         4280, 4909, 5105, 7763, 2216,  789, 6247, 1557, 6086, 3867, 6308, 3868,
         3141, 4863, 6039, 4720, 5682, 1996, 6415, 1006, 5309, 3173, 2215, 7349,
         5189, 4536,  728, 1829, 7332, 4213, 2691, 7303, 6578, 4362, 7011, 2873,
         3830, 2701, 2881, 5865, 7373, 6709,  288, 7195, 5901, 4414, 1250,   26,
         3252,  415, 4196, 3291, 4671, 6996,  777, 1186, 3614, 7182, 2982, 7513,
         7460, 6732, 6750,  675, 8061, 7047, 3636, 3195, 2402, 3587, 6024, 4136,
         6969, 5759, 6315, 1689, 4331, 7171, 4754, 7996,  191, 5439, 7573,  527,
         1630, 8030, 6144, 6510, 2318,  838, 2394, 7922, 4261, 2205, 2847,  944,
         2644, 2016, 2252, 5723, 7292, 7339]], device='cuda:0')
```

Could you please take a look at this behavior?

Hey - is there yet no effort to make an open source "gemini-2.0-flash-exp-image-generation" alternative?  

@ehartford hey! We stopped any efforts on adding Anole since there were much better models open sourced after that, though it doesn't do any image editing tasks, only image generation. We are currently working in Janus-Pro models for image gen and Cosmos for video gen. 

Though the process might be a bit slow given the non-standard architecture of the models compared to what we have in transformers

If you have any open source models that can do image editing etc. tasks on top, feel free to open an issue and suggest adding a new model, If the model is a high usage model (many downloads on the hub) or gets a lot of community reactions, we can add it :)

Lets close as we wont add the model anymore