|
# Dream Project Animation Nodes for ComfyUI |
|
|
|
This repository contains various nodes for supporting Deforum-style animation generation with ComfyUI. I created these |
|
for my own use (producing videos for my "Alt Key Project" music - |
|
[youtube channel](https://www.youtube.com/channel/UC4cKvJ4hia7zULxeCc-7OcQ)), but I think they should be generic enough |
|
and useful to many ComfyUI users. |
|
|
|
I have demonstrated the use of these custom nodes in this [youtube video](https://youtu.be/pZ6Li3qF-Kk). |
|
|
|
# Notice! |
|
|
|
This custom node pack is currently not being updated. Stable Diffusion video generation is moving towards a different |
|
workflow with AnimateDiff and Stable Video Diffusion. I decided to not try to update this node pack, but I am instead |
|
creating a separate custom node pack here: |
|
|
|
[github](https://github.com/alt-key-project/comfyui-dream-video-batches) |
|
|
|
This new node pack will be getting my attention from now on (at least as long as stable diffusion video generation is done mostly |
|
in batches). |
|
|
|
## Installation |
|
|
|
### Simple option |
|
|
|
You can install Dream Project Animation Nodes using the ComfyUI Manager. |
|
|
|
### Manual option |
|
|
|
Run within (ComfyUI)/custom_nodes/ folder: |
|
|
|
* git clone https://github.com/alt-key-project/comfyui-dream-project.git |
|
* cd comfyui-dream-project |
|
|
|
Then, if you are using the python embedded in ComfyUI: |
|
* (ComfyUI)/python_embedded/python.exe -s -m pip install -r requirements.txt |
|
|
|
With your system-wide python: |
|
* pip install -r requirements.txt |
|
|
|
Finally: |
|
* Start ComfyUI. |
|
|
|
After startup, a configuration file 'config.json' should have been created in the 'comfyui-dream-project' directory. |
|
Specifically check that the path of ffmpeg works in your system (add full path to the command if needed). |
|
|
|
## Upgrade |
|
|
|
When upgrading, it is good to re-run the pip install command as specified in the install section. This will install any |
|
new dependencies. |
|
|
|
## Configuration |
|
|
|
### debug |
|
|
|
Setting this to true will enable some trace-level logging. |
|
|
|
### ffmpeg.file_extension |
|
|
|
Sets the output file extension and with that the envelope used. |
|
|
|
### ffmpeg.path |
|
|
|
Path to the ffmpeg executable or just the command if ffmpeg is in PATH. |
|
|
|
### ffmpeg.arguments |
|
|
|
The arguments sent to FFMPEG. A few of the values are provided by the node: |
|
|
|
* %FPS% the target framerate |
|
* %FRAMES% a frame ionput file |
|
* %OUTPUT% output video file path |
|
|
|
### encoding.jpeg__quality |
|
|
|
Sets the encoding quality of jpeg images. |
|
|
|
### ui.top_category |
|
|
|
Sets the name of the top level category on the menu. Set to empty string "" to remove the top level. If the top level |
|
is removed you may also want to disable the category icons to get nodes into existing category folders. |
|
|
|
### prepend_icon_to_category / append_icon_to_category |
|
|
|
Flags to add a icon before and/or after the category name at each level. |
|
|
|
### prepend_icon_icon_to_node / append_icon_icon_to_node |
|
|
|
Flags to add an icon before and/or after the node name. |
|
|
|
### ui.category_icons |
|
|
|
Each key defines a unicode symbol as an icon used for the specified category. |
|
|
|
### mpeg_coder.bitrate_factor |
|
|
|
This factor allows changing the bitrate to better fit the required quality and codec. A value of 1 is typically |
|
suitable for H.265. |
|
|
|
### mpeg_coder.codec_name |
|
|
|
Codec names as specified by ffmpeg. Some common options include "libx264", "libx264" and "mpeg2video". |
|
|
|
### mpeg_coder.encoding_threads |
|
|
|
Increasing the number of encoding threads in mpegCoder will generally reduce the overall encoding time, but it will also |
|
increase the load on the computer. |
|
|
|
### mpeg_coder.file_extension |
|
|
|
Sets the output file extension and with that the envelope used. |
|
|
|
### mpeg_coder.max_b_frame |
|
|
|
Sets the max-b-frames parameter for as specified in ffmpeg. |
|
|
|
## Concepts used |
|
|
|
These are some concepts used in nodes: |
|
|
|
### Frame Counter |
|
|
|
The frame counter is an abstraction that keeps track of where we are in the animation - what frame is rendered |
|
and how does the current frame fit into the current animation. |
|
|
|
### Curves |
|
|
|
A curve is simply a node that produces a value based on the frame counter (changing over time). |
|
|
|
### Palette |
|
|
|
A palette is a collection of color values. |
|
|
|
### Sequence |
|
|
|
A sequence is a full set of animation frames and a corresponding timeline for these frames. The sequence is |
|
created by the 'Image Sequence Saver' node and it may be used to trigger post processing tasks such as generating the |
|
video file using ffmpeg. These nodes should be seen as a convenience and they are severely limited. Never put sequence |
|
nodes in parallel - they will not work as intended! |
|
|
|
## The nodes |
|
### Analyze Palette [Dream] |
|
Output brightness, red, green and blue averages of a palette. Useful to control other processing. |
|
|
|
### Beat Curve [Dream] |
|
Beat pattern curve with impulses at specified beats of a measure. |
|
|
|
### Big *** Switch [Dream] |
|
Switch nodes for different type for up to ten inputs. |
|
|
|
### Boolean To Float/Int [Dream] |
|
Converts a boolean value to two different numeric values. |
|
|
|
### Build Prompt [Dream] (and Finalize Prompt [Dream]) |
|
Weighted text prompt builder utility. Chain any number of these nodes and terminate with 'Finalize Prompt'. |
|
|
|
### Calculation [Dream] |
|
Mathematical calculation node. Exposes most of the mathematical functions in the python |
|
[math module](https://docs.python.org/3/library/math.html), mathematical operators as well as round, abs, int, |
|
float, max and min. |
|
|
|
### Compare Palettes [Dream] |
|
Analyses two palettes and produces the quotient for each individual channel (b/a) and brightness. |
|
|
|
### CSV Curve [Dream] |
|
CSV input curve where first column is frame or second and second column is value. |
|
|
|
### CSV Generator [Dream] |
|
CSV output, mainly for debugging purposes. First column is frame number and second is value. |
|
Recreates file at frame 0 (removing and existing content in the file). |
|
|
|
### Common Frame Dimensions [Dream] |
|
Utility for calculating good width/height based on common video dimensions. |
|
|
|
### Video Encoder (FFMPEG) [Dream] |
|
Post processing for animation sequences calling FFMPEG to generate video files. |
|
|
|
### File Count [Dream] |
|
Finds the number of files in a directory matching specified patterns. |
|
|
|
### Float/Int/string to Log Entry [Dream] |
|
Logging for float/int/string values. |
|
|
|
### Frame Count Calculator [Dream] |
|
Simple utility to calculate number of frames based on time and framerate. |
|
|
|
### Frame Counter (Directory) [Dream] |
|
Directory backed frame counter, for output directories. |
|
|
|
### Frame Counter (Simple) [Dream] |
|
Integer value used as frame counter. Useful for testing or if an auto-incrementing primitive is used as a frame |
|
counter. |
|
|
|
### Frame Counter Info [Dream] |
|
Extracts information from the frame counter. |
|
|
|
### Frame Counter Offset [Dream] |
|
Adds an offset (in frames) to a frame counter. |
|
|
|
### Frame Counter Time Offset [Dream] |
|
Adds an offset in seconds to a frame counter. |
|
|
|
### Image Brightness Adjustment [Dream] |
|
Adjusts the brightness of an image by a factor. |
|
|
|
### Image Color Shift [Dream] |
|
Allows changing the colors of an image with a multiplier for each channel (RGB). |
|
|
|
### Image Contrast Adjustment [Dream] |
|
Adjusts the contrast of an image by a factor. |
|
|
|
### Image Motion [Dream] |
|
Node supporting zooming in/out and translating an image. |
|
|
|
### Image Sequence Blend [Dream] |
|
Post processing for animation sequences blending frame for a smoother blurred effect. |
|
|
|
### Image Sequence Loader [Dream] |
|
Loads a frame from a directory of images. |
|
|
|
### Image Sequence Saver [Dream] |
|
Saves a frame to a directory. |
|
|
|
### Image Sequence Tweening [Dream] |
|
Post processing for animation sequences generating blended in-between frames. |
|
|
|
### Laboratory [Dream] |
|
Super-charged number generator for experimenting with ComfyUI. |
|
|
|
### Log Entry Joiner [Dream] |
|
Merges multiple log entries (reduces noodling). |
|
|
|
### Log File [Dream] |
|
The text logging facility for the Dream Project Animation nodes. |
|
|
|
### Linear Curve [Dream] |
|
Linear interpolation between two values over the full animation. |
|
|
|
### Noise from Area Palettes [Dream] |
|
Generates noise based on the colors of up to nine different palettes, each connected to position/area of the |
|
image. Although the palettes are optional, at least one palette should be provided. |
|
|
|
### Noise from Palette [Dream] |
|
Generates noise based on the colors in a palette. |
|
|
|
### Palette Color Align [Dream] |
|
Shifts the colors of one palette towards another target palette. If the alignment factor |
|
is 0.5 the result is nearly an average of the two palettes. At 0 no alignment is done and at 1 we get a close |
|
alignment to the target. Above one we will overshoot the alignment. |
|
|
|
### Palette Color Shift [Dream] |
|
Multiplies the color values in a palette to shift the color balance or brightness. |
|
|
|
### Sample Image Area as Palette [Dream] |
|
Randomly samples a palette from an image based on pre-defined areas. The image is separated into nine rectangular areas |
|
of equal size and each node may sample one of these. |
|
|
|
### Sample Image as Palette [Dream] |
|
Randomly samples pixels from a source image to build a palette from it. |
|
|
|
### Saw Curve [Dream] |
|
Saw wave curve. |
|
|
|
### Sine Curve [Dream] |
|
Simple sine wave curve. |
|
|
|
### Smooth Event Curve [Dream] |
|
Single event/peak curve with a slight bell-shape. |
|
|
|
### String Tokenizer [Dream] |
|
Splits a text into tokens by a separator and returns one of the tokens based on a given index. |
|
|
|
### Triangle Curve [Dream] |
|
Triangle wave curve. |
|
|
|
### Triangle Event Curve [Dream] |
|
Single event/peak curve with triangular shape. |
|
|
|
### WAV Curve [Dream] |
|
Use an uncompressed WAV audio file as a curve. |
|
|
|
### Other custom nodes |
|
|
|
Many of the nodes found in 'WAS Node Suite' are useful the Dream Project Animation nodes - I suggest you install those |
|
custom nodes as well! |
|
|
|
## Examples |
|
|
|
### Image Motion with Curves |
|
|
|
This example should be a starting point for anyone wanting to build with the Dream Project Animation nodes. |
|
|
|
[motion-workflow-example](examples/motion-workflow-example.json) |
|
|
|
### Image Motion with Color Coherence |
|
|
|
Same as above but with added color coherence through palettes. |
|
|
|
[motion-workflow-with-color-coherence](examples/motion-workflow-with-color-coherence.json) |
|
|
|
### Area Sampled Noise |
|
|
|
This flow demonstrates sampling image areas into palettes and generating noise for these areas. |
|
|
|
[area-sampled-noise](examples/area-sampled-noise.json) |
|
|
|
### Prompt Morphing |
|
|
|
This flow demonstrates prompt building with weights based on curves and brightness and contrast control. |
|
|
|
[prompt-morphing](examples/prompt-morphing.json) |
|
|
|
### Laboratory |
|
|
|
This flow demonstrates use of the Laboratory and Logging nodes. |
|
|
|
[laboratory](examples/laboratory.json) |
|
|
|
## Known issues |
|
|
|
### FFMPEG |
|
|
|
The call to FFMPEG currently in the default configuration (in config.json) does not seem to work for everyone. The good |
|
news is that you can change the arguments to whatever works for you - the node-supplied parameters (that probably all need to be in the call) |
|
are: |
|
|
|
* -i %FRAMES% (the input file listing frames) |
|
* -r %FPS% (sets the frame rate) |
|
* %OUTPUT% (the path to the video file) |
|
|
|
If possible, I will change the default configuration to one that more versions/builds of ffmpeg will accept. Do let me |
|
know what arguments are causing issues for you! |
|
|
|
### Framerate is not always right with mpegCoder encoding node |
|
|
|
The mpegCoder library will always use variable frame rate encoding if it is available in the output format. With most |
|
outputs this means that your actual framerate will differ slightly from the requested one. |
|
|