TaihoC

/

Anima-ControlNet-VACE-Depth

like 12

Are you using the Load ControlNet node from Comfy Core or from the custom node package? Comfy Core's Load ControlNet does not support this model. I can share a workflow later, but I am not at my computer right now.

I used "Load Advanced ControlNet Model" from the custom node package.

Q1. Between the two, which one should I use?

The problem was that I pulled the git repository but it was on main branch, not feat/anima-vace-controlnet. I moved the HEAD to the branch and fixed the problem. However I encountered another problem which doesn't generate images guided by the ControlNet. The generated image does have contour of the reference image but is full of noises.

Q2. Am I supposed to "Apply ControlNet" node from custom node package as well? I had been using Comfy Core one. I'll try using it and get back to you.

I tried using the above instead of the below but didn't fix the issue. Hmm I am lost. Seeing the actual example workflow might be of a great help. I'll be waiting. Take your time.

I somehow managed to make it work but I have encountered two weird issues.

1. AssertionError: H,W (180, 135) should be divisible by spatial_patch_size 2
   When I try to generate an image of 1080x1440, I get this error. Only way to resolve it was restarting ComfyUI and having empty latent image to be 1088x1440. Does the your ControlNet model require an image to be in a certain size? I never encountered such error messages when I was messing around with SDXL/Illustrious ControlNet models. Specifically, Xinsir's Union ControlNet model.

2. Image quality degrades to be like blurry and noisy after encountering the first issue. When I deliberately make errors with 1080x1440 sized empty latent images, the generated images become a chunk of noises rather an image.

Here's my workflow

Reference Image

Before Error

After First Error

After Second Error

After Third Error

Issue 1: I have encountered this issue but I did not push the fix up. Essentially what happens is the vae compresses the image by 8 (1080->135), and then Anima further patchifies the image by 2 (135->?), and thus every 16*16 blocks becomes a single token. 1080 is not divisible by 16, and thus you see the crash. The main Anima branch forward resolve this simply by padding the input (135->136), and then cropping the output back (which is standard practice in other models too), but my fork does not do this right now. I will push the fix up after some cleanup.
I am not sure I quite understand the second issue. Do you mean after the crash at 1080 resolution once, you adjusted the resolution to 1088 to be able to generate, but the image becomes blurry? It might be because the crash caused the model to not unpatch correctly, so when you generate a second time, the model is patched again, essentially applying the CN twice, three times, etc. I haven't verified this hypothesis, but the issue should go away anyway assuming the crash is fixed.

Thank you. By the way, do you have plans to make PRs to https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet? Seems like the original repository also added support for kohya-ss's LLLite ControlNets. It may be great if relevant features could be accessed from single extension.

Regarding the second issue, yes your understanding is correct. It gets blurrier every time there are crashes at 1080 resolution. Like you mentioned, it should be irrelevant once you push the fix.

I have made a commit containing the fix.
With regards to merging into the original ACN: I have opened an issue over there and the author replied he will be supporting this.

Awesome. Thank you very much!