Package: edgel3
edgel3Look, Listen, and Learn (L3) [4] Audio subnetwork produces generic audio representations that can be used for myriad downstream tasks. However, L3-Net Audio requires 18 MB and 12 MB of static and dynamic memory respectively, making it infeasible for small edge devices with a single microcontroller.EdgeL3[2] is competetive with L3 Audio while being 95.45% sparse. However, it still has a high activation memory requirement.To jointly handle both static and dynamic memory, we introduceSpecialized Embedding Approximation[1], a teacher-student learning paradigm where the student audio embedding model is trained to approximate only the part of the teacher's embedding manifold which is relevant to the target data-domain. Notice the difference between data-domain and dataset. Restricting the specialization on a particular downstream dataset would compromise intra-domain generalizability.edgel3is an open-source Python library for downloading the smaller versions of L3 models and computing deep audio embeddings from such models.Theseamodels are specialized forSONYC-UST[5] data domain. Training pipelines can be found [here].Thesparsemodels provided have been re-trained using two different mechanisms: fine-tuningftand knowledge distillationkd. Training pipelines can be found [here].For non-compressed L3-Net, please refer toOpenL3[3]Installing edgel3DependenciesTensorflowedgel3has been tested with Tensorflow 2.0 and Keras 2.3.1.pip install tensorflow==2.0.0libsndfileedgel3depends on thepysoundfilemodule to load audio files, which depends on the non-Python librarylibsndfile. On Windows and macOS, these will be installed viapipand you can therefore skip this step.
However, on Linux this must be installed manually via your platform's package manager.
For Debian-based distributions (such as Ubuntu), this can be done by simply runningapt-get install libsndfile1For more detailed information, please consult thepysoundfileinstallation documentation.Installing edgel3The simplest way to install edgel3 is by usingpip, which will also install the additional required dependencies
if needed. To install edgel3 usingpip, simply runpip install edgel3To install the latest version of edgel3 from source:Clone or pull the lastest version:git clone https://github.com/ksangeeta2429/edgel3.gitInstall using pip to handle python dependencies:
cd edgel3
pip install -e .Getting started with edgel3Load a SONYC-UST specialized L3 audio (reduced input represenation and reduced architecture) that outputs an embedding of length 128model=edgel3.models.load_embedding_model(model_type='sea',emb_dim=128)Load a 95.45% sparse L3 audio re-trained with fine-tuningmodel=edgel3.models.load_embedding_model(model_type='sparse',retrain_type='ft',sparsity=95.45)Load a 87.0% sparse L3 audio re-trained with knowledge distillationmodel=edgel3.models.load_embedding_model(model_type='sparse',retrain_type='kd',sparsity=87.0)For more examples, please see thetutorialandmodule usage.ReferencesIf you use the SEA/EdgeL3 Github repos or the pre-trained models, please cite the relevant work:[1]Specialized Embedding Approximation for Edge Intelligence: A case study in Urban Sound ClassificationSangeeta Srivastava, Dhrubojyoti Roy, Mark Cartwright, Juan Pablo Bello, and Anish Arora.To be published in IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Toronto, Canada, June 2021.[2]EdgeL3: Compressing L3-Net for Mote-Scale Urban Noise MonitoringSangeeta Kumari, Dhrubojyoti Roy, Mark Cartwright, Juan Pablo Bello, and Anish Arora.Parallel AI and Systems for the Edge (PAISE), Rio de Janeiro, Brazil, May 2019.[3]Look, Listen and Learn More: Design Choices for Deep Audio EmbeddingsJason Cramer, Ho-Hsiang Wu, Justin Salamon, and Juan Pablo Bello.IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pages 3852–3856, Brighton, UK, May 2019.[4]Look, Listen and LearnRelja Arandjelović and Andrew ZissermanIEEE International Conference on Computer Vision (ICCV), Venice, Italy, Oct. 2017.[5]SONYC Urban Sound Tagging (SONYC-UST): a multilabel dataset from an urban acoustic sensor networkMark Cartwright, Ana Elisa Mendez Mendez, Graham Dove, Jason Cramer et al. 2019.