Package: live-illustrate
TTRPG live_illustrateASR + LLM + Diffusion = ???This project:UsesWhisperto transcribe live audio of a tabletop RPG sessionUsesGPT-3.5to extract a description of the current setting from the transcriptUsesDALL-Eto draw the settingUsesFlask&HTMXto display a new image every few minutesAnd like most AI projects, it simultaneously works better and worse than one might expect.
The images generated are usually an amusingly flawed rendition of what's going on, but are almosttoogood to be just ambient background flavor.Demo ReelSome scenes from our party's first trial session:The party enjoys dinner together on the deck of theDaydream. No one's quite sure where the other ship came from, but it looks nice.The party sails theDaydreamthrough a narrow canal in a swamp, searching for the hidden pirate city of Siren's Cove.
Perhaps they should ask the barrel people for directions.The party eavesdrops on a red-haired gnome and a halfling in a Siren's Cove tavern who are plotting to steal a competitor's shipping manifest.
Pay no attention to the faces of the other patrons.The party seeks further gossip at a luxe brothel calledThe Rich Dagger, guarded by a Goliath bouncer and famed for its perplexing architecture.InstallationI recommend installing in avirtual environment.# From PyPI:
pip install live-illustrate

# Or for hacking:
git clone git@github.com:ehennenfent/live_illustrate.git
cd live_illustrate
pip install -e ".[dev]"Whisper will bemuchfaster if you use a cuda-enabled pytorch build. I recommend installing this manually afterwards.pip install --index-url https://download.pytorch.org/whl/cu118 torch torchvision torchaudio  # https://pytorch.org/get-started/locally/You'll need an OpenAI API key, exposed via environment variable or in the.envfile, like so:OPENAI_API_KEY=<my_secret_api_key>.With the default settings, it costs about $1/hour to run. You can lower the cost by reducing the size of the generated images, or
increasing the interval between them.RunningOnce installed, run theillustratecommand line tool, which will automatically start recording with your default microphone.
Adata\directory will be created containing the generated images and transcripts, and a web server will start onlocalhost:8080to display the generated images.A few words about the most important command line options:--wait_minutes: This controls how frequently the tool draws an image, which directly translates into how expensive it is
to run. The default of 7.5 minutes seems to work well for our campaign.--max_context: Each interval, the tool looks back at the transcript and collects up tomax_contexttokens to send to GPT3.
It will get as close as possible, so some of these tokens may come frombeforethe previous image was generated. GPT can be
a bit slow about summarizing large amounts of text, so be careful about making this too large. The default of 2000 tokens seems
to correspondveryroughly to about ten minutes of conversation from one of our sessions, but YMMV.--persistence_of_memoryWhen summarizing long conversations, the LLM can seem to get "stuck" on the first setting described.
This argument controls what fraction of the previous context is retained each time an image is generated. The default setting of 0.2
may lead to some discontinuity if your party is in one place for a long time.Optionally, it's possible to upload generated images to a Discord server automatically by configuring aDiscord webhookand supplying the URL in theDISCORD_WEBHOOKenvironment variable.