Alternate comfy workflow
Had trouble with the provided workflow as it seemed to fall apart with some prompts for me, I changed to using a guider setup:
Workflow embedded. Some minor changes, this seemed more behaved for comfy, I changed the schedule to beta, as huggingface diffusers seems to be using a beta schedule for vpred. Also changed sampler to euler_a just because it seemed to have better aesthetic quality for me, but it seems to work as expected on most samplers.
One note I was not clear on starting out: Any text for the prompt that is not an e621 tag/danbooru tag tends to be (very) destructive, so ensure you're prompting only in the distribution of danbooru/e621 tags for best results.
Not destructive at all in regular euler though.
here's the image that should have the workflow I use with regular euler settings & what not... Nothing ever falls apart/breaks for me. i haven't checked yours but you are properly using things like modelsamplerdiscrete to set v-prediction right?