Instructions to use MelloModels/Mellos_HTTYD_Lora_for_WAN2.2_t2v with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use MelloModels/Mellos_HTTYD_Lora_for_WAN2.2_t2v with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import export_to_video # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("Wan-AI/Wan2.2-T2V-A14B", dtype=torch.bfloat16, device_map="cuda") pipe.load_lora_weights("MelloModels/Mellos_HTTYD_Lora_for_WAN2.2_t2v") prompt = "t00thle55, HTTYD2 Render Style, Hiccup. A low-angle tracking shot of a 3D rendered black dragon. The dragon flies rapidly through the clouds. The boy is wearing a green tunic, a brown fur vest, and leather boots. The dragon is in mid-flight with his body streamlined and angled slightly upward. His main wings are beating with powerful rhythmic downstrokes. The dragon's legs are tucked flush against his underbelly for aerodynamics. His tail and tail fins shift and tilt to steer through the turn. The dragon has a round head with black horns on the sides of his face along with long ears on the top of his head that are pulled back. The dragon has detailed black scales. The dragon tracks a flying object past the viewer's right. His green eyes are wide open showing his vertical slit pupils. He has a neutral expression with a closed mouth. The camera follows the subject from behind. The background is of a clear blue sky above a vast ocean. The lighting is neutral daylight." output = pipe(prompt=prompt).frames[0] export_to_video(output, "output.mp4") - Wan2.2
How to use MelloModels/Mellos_HTTYD_Lora_for_WAN2.2_t2v with Wan2.2:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
Mello's HTTYD LoRA Model for WAN 2.2
Never really planned on releasing this but here it is. Trained on 163 clips from 3 "movies" (Ranging from 17-101 frames), 15,000 steps, roughly 44 epochs. (working on updating to minimax)
β‘ Quick Info & Settings
| Parameter | Recommended Value |
|---|---|
| Trigger Word | t00thle55 |
| Base Model | WAN 2.2 T2V 14B (Wan-AI/Wan2.2-T2V-A14B) |
| Recommended LoRA Weight | 1.0 |
| Task Type | Text-to-Video (T2V) |
π Dataset & Training Specs
- Total Clips: 163 clips (ranging from 17 to 101 frames) from 3 movies
- Training Steps: 15,000 steps
- Epochs: ~44 epochs
Frame Length Breakdown:
- 17 Frames: 13 videos
- 33 Frames: 30 videos
- 41 Frames: 28 videos
- 53 Frames: 34 videos
- 69 Frames: 23 videos
- 85 Frames: 13 videos
- 101 Frames: 22 videos
π Links & Resources
- π¬ Tutorial Video: https://www.youtube.com/watch?v=TCcHq87aC4c
- πΎ Mirror: https://www.melloai.art/renders/models/httyd/
π Prompt Reference Guide
Note: When building your prompt, replace text inside parentheses
(). Plain text outside parentheses does not necessarily have to be modified.
t00thle55, (Enter style ex. HTTYD1 Render Style, HTTYD2 Render Style, HTTYD3 Render Style), (If another character is in frame name them here ex. Hiccup). A (shot type ex. wide establishing, medium close-up, low-angle tracking, wide angle from behind) shot of (describe the main focus of the video ex. a 3D rendered black dragon, a 3D rendered teenage boy). (If a dragon or subject is in frame, only describe their primary action, posture, and movement without detailing their specific anatomy yet ex. walks in a circle on all fours in a grass field, flies rapidly through the clouds, crouches defensively with his right paw stretched out). (If a character is in frame and is wearing clothes or gear, describe it here ex. The boy is wearing a green tunic, a brown fur vest, and leather boots).
IF THE DRAGON IS FLYING, ADD THESE TO THE PROMPT, IF NOT REMOVE:
The dragon is in mid-flight with his body (describe aerodynamic posture ex. streamlined and angled slightly upward, banking sharply to the left, pitched straight down for a dive). His main wings are (describe wing state and movement ex. fully extended and gliding smoothly on the updraft, beating with powerful rhythmic downstrokes, tucked tightly to his body to reduce drag). The dragon's legs are (describe leg position ex. tucked flush against his underbelly for aerodynamics, dangling downward as he prepares to land). His tail and tail fins (describe tail and fin mechanics ex. trail straight behind him acting as a rudder, flare wide open to act as an airbrake, shift and tilt to steer through the turn). (If the red prosthetic tail fin and saddle gear are visible, describe them acting in the wind ex. the leather cables pull taut as the red prosthetic fin snaps open). (Optional: describe environmental flight interactions ex. Wingtip vortices trail behind his wingtips, the wind aggressively rushes past his sleek form).
IF TOOTHLESS (THE BLACK DRAGON) IS IN FRAME, ADD THESE TO THE PROMPT, IF NOT REMOVE:
The dragon has a round head with black horns on the sides of his face along with long ears on the top of his head that are (pulled back/perked up/flat) (Remove this or change it if his head isn't visible). The dragon has detailed black scales. The dragon (Describe where and how the dragon is looking, ex. is looking down to the ground, tracks a flying object past the viewer's right). (If the dragon is NOT flying but his wings are visible, add "The dragon's wings are [describe grounded state ex. folded neatly against his sides/resting partially extended on the ground]"). (If the dragon is NOT flying but his tail is visible, add "The dragon's tail [describe grounded movement ex. sweeps over the grass/rests curled around his paws]"). His green eyes are (Describe the dragon's eyes ex. intermittently open, wide open with a curious expression, narrowed aggressively against the wind) showing his (elliptical/round/vertical slit) pupils. He (Describe his mouth and expression ex. has a neutral expression with a closed mouth, is snarling with his teeth exposed, has a gummy toothless smile). (If the dragon blinks, add "The dragon blinks").
IF HICCUP IS IN FRAME, ADD THESE TO THE PROMPT, IF NOT REMOVE:
The boy has messy auburn-brown hair and pale skin. He has a slender face with a small nose and faint freckles (remove this and change it if his head isn't visible). His green eyes are (Describe the boy's eyes ex. wide open with an amazed expression, squinting against the wind). The boy (Describe where and how the boy is looking, ex. looks off to the side at the dragon, stares directly forward while leaning into the wind). He (Describe his mouth and expression ex. has a closed mouth with a subtle nervous smile[if he is talking just say "He is talking with a [describe expression]"]). (If the boy blinks, add "The boy blinks").
IF ANOTHER DRAGON OR VIKING IS IN FRAME, APPLY THE PROMPT CHARACTERISTICS AS THE PREVIOUS SUBJECTS:
(Describe their scales/fur/skin texture. Describe their head shape, horns/spikes, and ears. Describe their wings and tail. Describe their eye color, pupil shape, and gaze direction. Describe their mouth, teeth, and overall expression. Note if they are blinking.)
The camera (enter the camera movements here ex. follows the subject from behind with handheld shake, then rotates towards the front of the man as he stops, is steady). The background is of a (describe the environment ex. blurred forest floor covered in moss and damp earth along with large blurred stone rock cliffs with green moss, a clear blue sky above a vast ocean, a rocky canyon). The lighting is (describe the lighting conditions ex. from an offscreen setting sun and slightly overcast sky, dramatic directional lighting from the fire, neutral daylight).
---
- Downloads last month
- -
Model tree for MelloModels/Mellos_HTTYD_Lora_for_WAN2.2_t2v
Base model
Wan-AI/Wan2.2-T2V-A14B