Package: dreamsound
DreamSoundDreamSound is a python package for sonic deep dream generation.DescriptionInspired by theDeepDreamproject, DreamSound plays a sound file toYamnet, a pre-trained neural network, and Yamnet returns a dreamed sound.Internally, DreamSound takes the gradients of a class from the pre-trained yamnet model and filters them with an original sound with some combination technique.ExampleHead tothis Google Colabfor a quick example on how to get started with the module. An old version can be accessiblein this  other Google Colab. Yet an older version ishere, which goes hand in hand with anearly paperwe did on the matter of Convolutional Neural Network Activation Layer Sonification, or what we called DreamSound.InstallDreamsound depends on the following pip packages you canpip install:requests
numpy
matplotlib
IPython
librosa
tensorflow
soundfileFirst, install the dependenciespython3-mpipinstall-rrequirements.txtInstall the dreamsound package using pip:python3-mpipinstalldreamsoundThe pip project is hosted at PyPi:https://pypi.org/project/dreamsound/NOTE: you need to download the yamnet model before importing the dreamsound module. Please, continue reading.PrepareCreate a directory for your project and relocate there.mkdirdream_testcddream_testRun the Yamnet Downloader.The Yamnet Downloader filedoes notcome with the pip distribution. However, it is distributed on this repository. If you do not want to clone this repository, simply do:curl -O https://raw.githubusercontent.com/fdch/dreamsound/main/yamnet_downloader.pyon this same directory, and run:python3yamnet_downloader.pyAlternatively, you can get Yamnet yourself, crudely like this:gitclonehttps://github.com/fdch/models.gitmodels
mvmodels/research/audioset/yamnet/*.
rm-rfmodels
curl-Ohttps://storage.googleapis.com/audioset/yamnet.h5Usage exampleYou must have the yamnet model on the same directory. Now, you can import the dreamsound module and use the class. This code loads some files from disk and passes them to theDreamSoundclass from thedreamsoundmodule. This looks something like this:>>> import dreamsound
INFO:tensorflow:Enabling eager execution
INFO:tensorflow:Enabling v2 tensorshape
INFO:tensorflow:Enabling resource variables
INFO:tensorflow:Enabling tensor equality
INFO:tensorflow:Enabling control flow v2
>>> ds = dreamsound.DreamSound(["../audio/original.wav", "../audio/cat.wav"])
Loading audio files...
Done.
I have now 2 audio files in memory.
Using last layer.
Yamnet loaded, using layer:activation_1
Dreamer started.FilteringThere are two types of filtering, auto or targetted:Auto Filtering:Filter the first audio with it's dreamed self>>> ds(audio_index=0)
Running step 0, class: Whistling...
...
Writing ./audio/Whistle-9-orig.wav...
Writing ./audio/Whistle-9-diff.wav...
Writing ./audio/Whistle-9-filt.wav...
Writing ./audio/Whistle-9-hard.wav...
Writing ./audio/Whistle-9-grad.wav...Targetted Filtering:Filter the first with a dreamed target>>> ds(audio_index=0, tgt=1)
Target class: Animal...
Running step 0, class: Whistling...
Running step 1, class: Whistle...
Running step 2, class: Whistle...
Running step 3, class: Whistle...
Running step 4, class: Whistle...
Running step 5, class: Whistle...
Running step 6, class: Whistle...
Running step 7, class: Whistle...
Running step 8, class: Flute...
Running step 9, class: Whistle...
Writing ./audio/Whistle-9-orig-tgt-Animal.wav...
Writing ./audio/Whistle-9-diff-tgt-Animal.wav...
Writing ./audio/Whistle-9-filt-tgt-Animal.wav...
Writing ./audio/Whistle-9-hard-tgt-Animal.wav...
Writing ./audio/Whistle-9-grad-tgt-Animal.wav...RecurseFinally, you can pass no arguments to continue filtering recursively>>> ds()
Target class: Animal...
Running step 10, class: Whistle...
Running step 11, class: Whistle...
Running step 12, class: Wind instrument, woodwind instrument...
Running step 13, class: Wind instrument, woodwind instrument...
Running step 14, class: Flute...
Running step 15, class: Flute...
Running step 16, class: Wind instrument, woodwind instrument...
Running step 17, class: Whistle...
Running step 18, class: Whistle...
Running step 19, class: Music...
Writing ./audio/Music-19-orig-tgt-Animal.wav...
Writing ./audio/Music-19-diff-tgt-Animal.wav...
Writing ./audio/Music-19-filt-tgt-Animal.wav...
Writing ./audio/Music-19-hard-tgt-Animal.wav...
Writing ./audio/Music-19-grad-tgt-Animal.wav...Class VariablesYou can change any of the following before or after calling the class:sr=22050max_dur=10patch_hop=0.1win_length=2048hop_length=128pad_end=Falseloss_power=0.001plot_every=10figsize=(10,8)top_db=80.0step_size=0.95output_type=3steps=10threshold=1e-07classid=Nonemaxloss=Trueelapsed=0recurse=Falsetarget=Nonepower=1.0audio_dir="./audio/"image_dir="./image/"AuthorsFede Camara Halac (https://github.com/fdch)
Matias Delgadino (https://github.com/zaytam)AcknowledgementsYamNet
AudioSet