Package: allin1
All-In-One Music Structure AnalyzerThis package provides models for music structure analysis, predicting:Tempo (BPM)BeatsDownbeatsFunctional segment boundariesFunctional segment labels (e.g., intro, verse, chorus, bridge, outro)Table of ContentsInstallationUsage for CLIUsage for PythonVisualization & SonificationAvailable ModelsSpeedAdvanced Usage for ResearchConcerning MP3 FilesTrainingCitationInstallation1. Install PyTorchVisitPyTorchand install the appropriate version for your system.2. Install NATTEN (Required for Linux and Windows; macOS will auto-install)Linux: Download fromNATTEN websitemacOS: Auto-installs withallin1.Windows: Build from source:pipinstallninja# Recommended, not requiredgitclonehttps://github.com/SHI-Labs/NATTENcdNATTEN
make3. Install the packagepipinstallgit+https://github.com/CPJKU/madmom# install the latest madmom directly from GitHubpipinstallallin1# install this package4. (Optional) Install FFmpeg for MP3 supportFor ubuntu:sudoaptinstallffmpegFor macOS:brewinstallffmpegUsage for CLITo analyze audio files:allin1your_audio_file1.wavyour_audio_file2.mp3Results will be saved in the./structdirectory by default:./struct
└──your_audio_file1.json
└──your_audio_file2.jsonThe analysis results will be saved in JSON format:{"path":"/path/to/your_audio_file.wav","bpm":100,"beats":[0.33,0.75,1.14,...],"downbeats":[0.33,1.94,3.53,...],"beat_positions":[1,2,3,4,1,2,3,4,1,...],"segments":[{"start":0.0,"end":0.33,"label":"start"},{"start":0.33,"end":13.13,"label":"intro"},{"start":13.13,"end":37.53,"label":"chorus"},{"start":37.53,"end":51.53,"label":"verse"},...]}All available options are as follows:$allin1-h

usage:allin1[-h][-oOUT_DIR][-v][--viz-dirVIZ_DIR][-s][--sonif-dirSONIF_DIR][-a][-e][-mMODEL][-dDEVICE][-k][--demix-dirDEMIX_DIR][--spec-dirSPEC_DIR]paths[paths...]positionalarguments:pathsPathtotracks

options:-h,--helpshowthishelpmessageandexit-oOUT_DIR,--out-dirOUT_DIRPathtoadirectorytostoreanalysisresults(default:./struct)-v,--visualizeSavevisualizations(default:False)--viz-dirVIZ_DIRDirectorytosavevisualizationsif-visprovided(default:./viz)-s,--sonifySavesonifications(default:False)--sonif-dirSONIF_DIRDirectorytosavesonificationsif-sisprovided(default:./sonif)-a,--activSaveframe-levelrawactivationsfromsigmoidandsoftmax(default:False)-e,--embedSaveframe-levelembeddings(default:False)-mMODEL,--modelMODELNameofthepretrainedmodeltouse(default:harmonix-all)-dDEVICE,--deviceDEVICEDevicetouse(default:cudaifavailableelsecpu)-k,--keep-byproductsKeepdemixedaudiofilesandspectrograms(default:False)--demix-dirDEMIX_DIRPathtoadirectorytostoredemixedtracks(default:./demix)--spec-dirSPEC_DIRPathtoadirectorytostorespectrograms(default:./spec)Usage for PythonAvailable functions:analyze()load_result()visualize()sonify()analyze()Analyzes the provided audio files and returns the analysis results.importallin1# You can analyze a single file:result=allin1.analyze('your_audio_file.wav')# Or multiple files:results=allin1.analyze(['your_audio_file1.wav','your_audio_file2.mp3'])A result is a dataclass instance containing:AnalysisResult(path='/path/to/your_audio_file.wav',bpm=100,beats=[0.33,0.75,1.14,...],beat_positions=[1,2,3,4,1,2,3,4,1,...],downbeats=[0.33,1.94,3.53,...],segments=[Segment(start=0.0,end=0.33,label='start'),Segment(start=0.33,end=13.13,label='intro'),Segment(start=13.13,end=37.53,label='chorus'),Segment(start=37.53,end=51.53,label='verse'),Segment(start=51.53,end=64.34,label='verse'),Segment(start=64.34,end=89.93,label='chorus'),Segment(start=89.93,end=105.93,label='bridge'),Segment(start=105.93,end=134.74,label='chorus'),Segment(start=134.74,end=153.95,label='chorus'),Segment(start=153.95,end=154.67,label='end'),]),Unlike CLI, it does not save the results to disk by default. You can save them as follows:result=allin1.analyze('your_audio_file.wav',out_dir='./struct',)Parameters:paths:Union[PathLike, List[PathLike]]List of paths or a single path to the audio files to be analyzed.out_dir:PathLike(optional)Path to the directory where the analysis results will be saved. By default, the results will not be saved.visualize:Union[bool, PathLike](optional)Whether to visualize the analysis results or not. If a path is provided, the visualizations will be saved in that directory. Default is False. If True, the visualizations will be saved in './viz'.sonify:Union[bool, PathLike](optional)Whether to sonify the analysis results or not. If a path is provided, the sonifications will be saved in that directory. Default is False. If True, the sonifications will be saved in './sonif'.model:str(optional)Name of the pre-trained model to be used for the analysis. Default is 'harmonix-all'. Please refer to the documentation for the available models.device:str(optional)Device to be used for computation. Default is 'cuda' if available, otherwise 'cpu'.include_activations:bool(optional)Whether to include activations in the analysis results or not.include_embeddings:bool(optional)Whether to include embeddings in the analysis results or not.demix_dir:PathLike(optional)Path to the directory where the source-separated audio will be saved. Default is './demix'.spec_dir:PathLike(optional)Path to the directory where the spectrograms will be saved. Default is './spec'.keep_byproducts:bool(optional)Whether to keep the source-separated audio and spectrograms or not. Default is False.multiprocess:bool(optional)Whether to use multiprocessing for extracting spectrograms. Default is True.Returns:Union[AnalysisResult, List[AnalysisResult]]Analysis results for the provided audio files.load_result()Loads the analysis results from the disk.result=allin1.load_result('./struct/24k_Magic.json')visualize()Visualizes the analysis results.fig=allin1.visualize(result)fig.show()Parameters:result:Union[AnalysisResult, List[AnalysisResult]]List of analysis results or a single analysis result to be visualized.out_dir:PathLike(optional)Path to the directory where the visualizations will be saved. By default, the visualizations will not be saved.Returns:Union[Figure, List[Figure]]List of figures or a single figure containing the visualizations.Figureis a class frommatplotlib.pyplot.sonify()Sonifies the analysis results.
It will mix metronome clicks for beats and downbeats, and event sounds for segment boundaries
to the original audio file.y,sr=allin1.sonify(result)# y: sonified audio with shape (channels=2, samples)# sr: sampling rate (=44100)Parameters:result:Union[AnalysisResult, List[AnalysisResult]]List of analysis results or a single analysis result to be sonified.out_dir:PathLike(optional)Path to the directory where the sonifications will be saved. By default, the sonifications will not be saved.Returns:Union[Tuple[NDArray, float], List[Tuple[NDArray, float]]]List of tuples or a single tuple containing the sonified audio and the sampling rate.Visualization & SonificationThis package provides a simple visualization (-vor--visualize) and sonification (-sor--sonify) function for the analysis results.allin1-v-syour_audio_file.wavThe visualizations will be saved in the./vizdirectory by default:./viz
└──your_audio_file.pdfThe sonifications will be saved in the./sonifdirectory by default:./sonif
└──your_audio_file.sonif.wavFor example, a visualization looks like this:You can try it atHugging Face Space.Available ModelsThe models are trained on theHarmonix Setwith 8-fold cross-validation.
For more details, please refer to thepaper.harmonix-all: (Default) An ensemble model averaging the predictions of 8 models trained on each fold.harmonix-foldN: A model trained on fold N (0~7). For example,harmonix-fold0is trained on fold 0.By default, theharmonix-allmodel is used. To use a different model, use the--modeloption:allin1--modelharmonix-fold0your_audio_file.wavSpeedWith an RTX 4090 GPU and Intel i9-10940X CPU (14 cores, 28 threads, 3.30 GHz),
theharmonix-allmodel processed 10 songs (33 minutes) in 73 seconds.Advanced Usage for ResearchThis package provides researchers with advanced options to extractframe-level raw activations and embeddingswithout post-processing. These have a resolution of 100 FPS, equivalent to 0.01 seconds per frame.CLIActivationsThe--activoption also saves frame-level raw activations from sigmoid and softmax:$allin1--activyour_audio_file.wavYou can find the activations in the.npzfile:./struct
└──your_audio_file1.json
└──your_audio_file1.activ.npzTo load the activations in Python:>>>importnumpyasnp>>>activ=np.load('./struct/your_audio_file1.activ.npz')>>>activ.files['beat','downbeat','segment','label']>>>beat_activations=activ['beat']>>>downbeat_activations=activ['downbeat']>>>segment_boundary_activations=activ['segment']>>>segment_label_activations=activ['label']Details of the activations are as follows:beat: Raw activations from thesigmoidlayer forbeat tracking(shape:[time_steps])downbeat: Raw activations from thesigmoidlayer fordownbeat tracking(shape:[time_steps])segment: Raw activations from thesigmoidlayer forsegment boundary detection(shape:[time_steps])label: Raw activations from thesoftmaxlayer forsegment labeling(shape:[label_class=10, time_steps])You can access the label names as follows:>>>allin1.HARMONIX_LABELS['start','end','intro','outro','break','bridge','inst','solo','verse','chorus']EmbeddingsThis package also provides an option to extract raw embeddings from the model.$allin1--embedyour_audio_file.wavYou can find the embeddings in the.npyfile:./struct
└──your_audio_file1.json
└──your_audio_file1.embed.npyTo load the embeddings in Python:>>>importnumpyasnp>>>embed=np.load('your_audio_file1.embed.npy')Each model embeds for every source-separated stem per time step,
resulting in embeddings shaped as[stems=4, time_steps, embedding_size=24]:The number of source-separated stems (the order is bass, drums, other, vocals).The number of time steps (frames). The time step is 0.01 seconds (100 FPS).The embedding size of 24.Using the--embedoption with theharmonix-allensemble model will stack the embeddings,
saving them with the shape[stems=4, time_steps, embedding_size=24, models=8].PythonThe Python APIallin1.analyze()offers the same options as the CLI:>>>allin1.analyze(paths='your_audio_file.wav',include_activations=True,include_embeddings=True,)AnalysisResult(path='/path/to/your_audio_file.wav',bpm=100,beats=[...],downbeats=[...],segments=[...],activations={'beat':array(...),'downbeat':array(...),'segment':array(...),'label':array(...)},embeddings=array(...),)Concerning MP3 FilesDue to variations in decoders, MP3 files can have slight offset differences.
I recommend you to first convert your audio files to WAV format using FFmpeg (as shown below),
and use the WAV files for all your data processing pipelines.ffmpeg-iyour_audio_file.mp3your_audio_file.wavIn this package, audio files are read usingDemucs.
To my understanding, Demucs converts MP3 files to WAV using FFmpeg before reading them.
However, using a different MP3 decoder can yield different offsets.
I've observed variations of about 20~40ms, which is problematic for tasks requiring precise timing like beat tracking,
where the conventional tolerance is just 70ms.
Hence, I advise standardizing inputs to the WAV format for all data processing,
ensuring straightforward decoding.TrainingPlease refer toTRAINING.md.CitationIf you use this package for your research, please cite the following paper:@inproceedings{taejun2023allinone,title={All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audio},author={Kim, Taejun and Nam, Juhan},booktitle={IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)},year={2023}}