Package: imaginAIry
ImaginAIry 🤖🧠AI imagined images. Pythonic generation of stable diffusion imagesand videos*!."just works" on Linux and macOS(M1) (and sometimes windows).# on macOS, make sure rust is installed first# be sure to use Python 3.10, Python 3.11 is not supported at the moment>>pipinstallimaginairy
>>imagine"a scenic landscape""a photo of a dog""photo of a fruit bowl""portrait photo of a freckled woman""a bluejay"# Make an AI video>>aimgvideogen--start-imagerocket.pngStable Video DiffusionRushed release of Stable Diffusion Video!Works with Nvidia GPUs.  Does not work on Mac or CPU.On Windows you'll need to install torch 2.0 first viahttps://pytorch.org/get-started/locally/Usage: aimg videogen [OPTIONS]

  AI generate a video from an image

  Example:

      aimg videogen --start-image assets/rocket-wide.png

Options:
  --start-image TEXT       Input path for image file.
  --num-frames INTEGER     Number of frames.
  --num-steps INTEGER      Number of steps.
  --model TEXT             Model to use. One of: svd, svd_xt, svd_image_decoder, svd_xt_image_decoder
  --fps INTEGER            FPS for the AI to target when generating video
  --output-fps INTEGER     FPS for the output video
  --motion-amount INTEGER  How much motion to generate. value between 0 and 255.
  -r, --repeats INTEGER    How many times to repeat the renders.   [default: 1]
  --cond-aug FLOAT         Conditional augmentation.
  --seed INTEGER           Seed for random number generator.
  --decoding_t INTEGER     Number of frames decoded at a time.
  --output_folder TEXT     Output folder.
  --help                   Show this message and exit.ImagesWhats NewSee full Changelog here14.1.1tests: add installation tests for windows, mac, and condafix: dependency issues14.1.0🎉 feature: make video generation smooth by adding frame interpolationfeature: SDXL weights in the compvis format can now be usedfeature: allow video generation at any size specified by userfeature: video generations output in "bounce" formatfeature: choose video output format: mp4, webp, or giffeature: fix random seed handling in video generationdocs: auto-publish docs on push to masterbuild: remove imageio dependencybuild: vendorize facexlib so we don't install its unneeded dependencies14.0.4docs: add a documentation website athttps://brycedrennan.github.io/imaginAIry/build: remove fairscale dependencyfix: video generation was broken14.0.3fix: several critical bugs with packagetests: add a wheel smoketest to detect these issues in the future14.0.0🎉 video generation usingStable Video Diffusionadd--videogento any image generation to create a short video from the generated imageor useaimg videogento generate a video from an image🎉 SDXL (Stable Diffusion Extra Large) models are now supported.try--model opendalleor--model sdxlinpainting and controlnets are not yet supported for SDXL🎉 imaginairy is now backed by therefiners libraryThis was a huge rewrite which is why some features are not yet supported.  On the plus side, refiners supports
cutting edge features (SDXL, image prompts, etc) which will be added to imaginairy soon.self-attention guidancewhich makes details of images more accurate🎉 feature: larger image generations now work MUCH better and stay faithful to the same image as it looks at a smaller size.
For example--size 720p --seed 1and--size 1080p --seed 1will produce the same image for SD15🎉 feature: loading diffusers based models now supported. Example--model https://huggingface.co/ainz/diseny-pixar --model-architecture sd15🎉 feature: qrcode controlnet!Run API server and StableStudio web interface (alpha)Generate images via API or web interface.  Much smaller featureset compared to the command line tool.>>aimgserverVisithttp://localhost:8000/andhttp://localhost:8000/docsImage Structure Controlby ControlNet(Not supported for SDXL yet)Generate images guided by body poses, depth maps, canny edges, hed boundaries, or normal maps.Openpose Controlimagine--control-imageassets/indiana.jpg--control-modeopenpose--caption-textopenpose"photo of a polar bear"Canny Edge Controlimagine--control-imageassets/lena.png--control-modecanny"photo of a woman with a hat looking at the camera"HED Boundary Controlimagine--control-imagedog.jpg--control-modehed"photo of a dalmation"Depth Map Controlimagine--control-imagefancy-living.jpg--control-modedepth"a modern living room"Normal Map Controlimagine--control-imagebird.jpg--control-modenormal"a bird"Image Shuffle ControlGenerates the image based on elements of the control image. Kind of similar to style transfer.imagine--control-imagepearl-girl.jpg--control-modeshuffle"a clown"The middle image is the "shuffled" input imageEditing Instructions ControlSimilar to instructPix2Pix (below) but works with any SD 1.5 based model.imagine--control-imagepearl-girl.jpg--control-modeedit--init-image-strength0.01--steps30--negative-prompt""--modelopenjourney-v2"make it anime""make it at the beach"Add Details Control (upscaling/super-resolution)Replaces existing details in an image. Good to use with --init-image-strength 0.2imagine--control-image"assets/wishbone.jpg"--control-modedetails"sharp focus, high-resolution"--init-image-strength0.2--steps30-w2048-h2048Image (re)Colorization (using brightness control)Colorize black and white images or re-color existing images.The generated colors will be applied back to the original image. You can either provide a caption or
allow the tool to generate one for you.aimgcolorizepearl-girl.jpg--caption"photo of a woman"Instruction based image editsby InstructPix2Pix(Broken as of 14.0.0)Just tell imaginairy how to edit the image and it will do it for you!Click to see shell commandsUse prompt strength to control how strong the edit is. For extra control you can combine with prompt-based masking.# enter imaginairy shell>>aimg
🤖🧠>editscenic_landscape.jpg-p"make it winter"--prompt-strength20🤖🧠>editdog.jpg-p"make the dog red"--prompt-strength5🤖🧠>editbowl_of_fruit.jpg-p"replace the fruit with strawberries"🤖🧠>editfreckled_woman.jpg-p"make her a cyborg"--prompt-strength13🤖🧠>editbluebird.jpg-p"make the bird wear a cowboy hat"--prompt-strength10🤖🧠>editflower.jpg-p"make the flower out of paper origami"--arg-scheduleprompt-strength[1:11:0.3]--steps25--compilation-animgif# create a comparison gif🤖🧠>editpearl_girl.jpg-p"make her wear clown makeup"--compare-gif# create an animation showing the edit with increasing prompt strengths🤖🧠>editmona-lisa.jpg-p"make it a color professional photo headshot"--negative-prompt"old, ugly, blurry"--arg-schedule"prompt-strength[2:8:0.5]"--compilation-animgif
🤖🧠>editgg-bridge.jpg-p"make it night time"--prompt-strength15--steps30--arg-scheduleprompt-strength[1:15:1]--compilation-animgifQuick Image Edit DemoWant just quickly have some fun? Tryedit-demoto apply some pre-defined edits.>>aimgedit-demopearl_girl.jpgPrompt Based Maskingby clipsegSpecify advanced text based masks using boolean logic and strength modifiers.
Mask syntax:mask descriptions must be lowercasekeywords (AND,OR,NOT) must be uppercaseparentheses are supportedmask modifiers may be appended to any mask or group of masks.  Example:(dog OR cat){+5}means that we'll
select any dog or cat and then expand the size of the mask area by 5 pixels.  Valid mask modifiers:{+n}- expand mask by n pixels{-n}- shrink mask by n pixels{*n}- multiply mask strength. will expand mask to areas that weakly matched the mask description{/n}- divide mask strength. will reduce mask to areas that most strongly matched the mask description. probably not usefulWhen writing strength modifiers keep in mind that pixel values are between 0 and 1.>>imagine\--init-imagepearl_earring.jpg\--mask-prompt"face AND NOT (bandana OR hair OR blue fabric){*6}"\--mask-modekeep\--init-image-strength.2\--fix-faces\"a modern female president""a female robot""a female doctor""a female firefighter"➡️>>imagine\--init-imagefruit-bowl.jpg\--mask-prompt"fruit OR fruit stem{*6}"\--mask-modereplace\--mask-modify-original\--init-image-strength.1\"a bowl of kittens""a bowl of gold coins""a bowl of popcorn""a bowl of spaghetti"➡️Face Enhancementby CodeFormer>>imagine"a couple smiling"--steps40--seed1--fix-faces➡️Upscalingby RealESRGAN>>imagine"colorful smoke"--steps40--upscale# upscale an existing image>>aimgupscalemy-image.jpgPython Examplefromimaginairy.enhancers.upscale_realesrganimportupscale_imagefromPILimportImageimg=Image.open("my-image.jpg")big_img=upscale_image(i)➡️Tiled Images>>imagine"gold coins""a lush forest""piles of old books"leaves--tile360 degree imagesimagine--tile-x-w1024-h512"360 degree equirectangular panorama photograph of the desert"--upscaleImage-to-ImageUse depth maps for amazing "translations" of existing images.>>imagine--init-imagegirl_with_a_pearl_earring_large.jpg--init-image-strength0.05"professional headshot photo of a woman with a pearl earring"-r4-w1024-h1024--steps50➡️OutpaintingGiven a starting image, one can generate it's "surroundings".Example:imagine --init-image pearl-earring.jpg --init-image-strength 0 --outpaint all250,up0,down600 "woman standing"➡️Work with different generation modelsClick to see shell commandimagine"valley, fairytale treehouse village covered, , matte painting, highly detailed, dynamic lighting, cinematic, realism, realistic, photo real, sunset, detailed, high contrast, denoised, centered, michael whelan"--steps60--seed1--arg-schedulemodel[sd14,sd15,sd20,sd21,openjourney-v1,openjourney-v2]--arg-schedule"caption-text[sd14,sd15,sd20,sd21,openjourney-v1,openjourney-v2]"Prompt ExpansionYou can use{}to randomly pull values from lists.  A list of values separated by|and enclosed in{ }will be randomly drawn from in a non-repeating fashion. Values that are surrounded by_ _will
pull from a phrase list of the same name.   Folders containing .txt phraselist files may be specified via--prompt_library_path. The option may be specified multiple times.  Built-in categories:3d-term, adj-architecture, adj-beauty, adj-detailed, adj-emotion, adj-general, adj-horror, animal, art-scene, art-movement, 
  art-site, artist, artist-botanical, artist-surreal, aspect-ratio, bird, body-of-water, body-pose, camera-brand,
  camera-model, color, cosmic-galaxy, cosmic-nebula, cosmic-star, cosmic-term, desktop-background, dinosaur, eyecolor, f-stop, 
  fantasy-creature, fantasy-setting, fish, flower, focal-length, food, fruit, games, gen-modifier, hair, hd,
  iso-stop, landscape-type, national-park, nationality, neg-weight, noun-beauty, noun-fantasy, noun-general, 
  noun-horror, occupation, painting-style, photo-term, pop-culture, pop-location, punk-style, quantity, rpg-item, scenario-desc, 
  skin-color, spaceship, style, tree-species, trippy, world-heritage-siteExamples:imagine "a {lime|blue|silver|aqua} colored dog" -r 4 --seed 0(note that it generates a dog of each color without repetition)imagine "a {_color_} dog" -r 4 --seed 0will generate four, different colored dogs. The colors will be pulled from an included
phraselist of colors.imagine "a {_spaceship_|_fruit_|hot air balloon}. low-poly" -r 4 --seed 0will generate images of spaceships or fruits or a hot air balloonPython examplefromimaginairy.enhancers.prompt_expansionimportexpand_promptsmy_prompt="a giant{_animal_}"expanded_prompts=expand_prompts(n=10,prompt_text=my_prompt,prompt_library_paths=["./prompts"])Credit tonoodle-soup-promptswhere most, but not all, of the wordlists originate.Generate image captions (viaBLIP)>>aimgdescribeassets/mask_examples/bowl001.jpg
abowlfullofgoldbarssittingonatableExample Use Cases>>aimg# Generate endless 8k art🤖🧠>imagine-w1920-h1080--upscale"{_art-scene_}. {_painting-style_} by {_artist_}"-r1000--steps30--modelsd21v# generate endless desktop backgrounds🤖🧠>imagine--tile"{_desktop-background_}"-r100# convert a folder of images to pencil sketches🤖🧠>editother/images/*.jpg-p"make it a pencil sketch"# upscale a folder of images🤖🧠>upscalemy-images/*.jpg# generate kitchen remodel ideas🤖🧠>imagine--control-imagekitchen.jpg-w1024-h1024"{_interior-style_} kitchen"--control-modedepth-r100--init-image0.01--upscale--steps35--caption-text"{prompt}"Additional FeaturesGenerate images either in code or from command line.It just works. Proper requirements are installed. Model weights are automatically downloaded. No huggingface account needed.
(if you have the right hardware... and aren't on windows)Noisy logs are gone (which was surprisingly hard to accomplish)WeightedPrompts let you smash together separate prompts (cat-dog)Prompt metadata saved into image file metadataHave AI generate captions for imagesaimg describe <filename-or-url>Interactive prompt: just runaimgHow ToFor full command line instructions runaimg --helpfromimaginairyimportimagine,imagine_image_files,ImaginePrompt,WeightedPrompt,LazyLoadingImageurl="https://upload.wikimedia.org/wikipedia/commons/thumb/6/6c/Thomas_Cole_-_Architect%E2%80%99s_Dream_-_Google_Art_Project.jpg/540px-Thomas_Cole_-_Architect%E2%80%99s_Dream_-_Google_Art_Project.jpg"prompts=[ImaginePrompt("a scenic landscape",seed=1,upscale=True),ImaginePrompt("a bowl of fruit"),ImaginePrompt([WeightedPrompt("cat",weight=1),WeightedPrompt("dog",weight=1),]),ImaginePrompt("a spacious building",init_image=LazyLoadingImage(url=url)),ImaginePrompt("a bowl of strawberries",init_image=LazyLoadingImage(filepath="mypath/to/bowl_of_fruit.jpg"),mask_prompt="fruit OR stem{*2}",# amplify the stem mask x2mask_mode="replace",mask_modify_original=True,),ImaginePrompt("strawberries",tile_mode=True),]forresultinimagine(prompts):# do somethingresult.save("my_image.jpg")# orimagine_image_files(prompts,outdir="./my-art")Requirements~10 gb space for models to downloadA CUDA supported graphics card with >= 11gb VRAM (and CUDA installed) or an M1 processor.Python installed. Preferably Python 3.10.  (not conda)For macOSrustand setuptools-rust must be installed to compile thetokenizerlibrary.
They can be installed via:curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | shandpip install setuptools-rustRunning in DockerSee example Dockerfile (works on machine where you can pass the gpu into the container)dockerbuild.-timaginairy# you really want to map the cache or you end up wasting a lot of time and space redownloading the model weightsdockerrun-it--gpusall-v$HOME/.cache/huggingface:/root/.cache/huggingface-v$HOME/.cache/torch:/root/.cache/torch-v`pwd`/outputs:/outputsimaginairy/bin/bashRunning on Google ColabExample ColabQ&AQ: How do I change the cache directory for where models are stored?A: Set theHUGGINGFACE_HUB_CACHEenvironment variable.Q: How do I free up disk space?A: The AI models are cached in~/.cache/(orHUGGINGFACE_HUB_CACHE). To delete the cache remove the following folders:~/.cache/imaginairy~/.cache/clip~/.cache/torch~/.cache/huggingfaceNot Supportedexploratory features that don't work well