Using pytorch vggish for audio classification tasks

username: kepler62f
cooked: <p>I am researching on using pretrained <a href="https://github.com/tensorflow/models/tree/master/research/audioset/vggish" rel="noopener nofollow ugc">VGGish</a> model for audio classification tasks, ideally I could have a model classifying any of the classes defined in the google audioset. I came across a nice pytorch <a href="https://github.com/harritaylor/torchvggish" rel="noopener nofollow ugc">port</a> for generating audio features. The original model generates only audio features as well. The original team suggests generally the following way to proceed:</p>
<blockquote>
<p><em>As a feature extractor</em> : VGGish converts audio input features into a semantically meaningful, high-level 128-D embedding which can be fed as input to a downstream classification model. The downstream model can be shallower than usual because the VGGish embedding is more semantically compact than raw audio features. So, for example, you could train a classifier for 10 of the AudioSet classes by using the released embeddings as features. Then, you could use that trained classifier with any arbitrary audio input by running the audio through the audio feature extractor and VGGish model provided here, passing the resulting embedding features as input to your trained model. <code>vggish_inference_demo.py</code> shows how to produce VGGish embeddings from arbitrary audio</p>
</blockquote>
<p>I’m not sure how to go about using getting the released embeddings and using them for training in pytorch. I’m also not sure how to translate the embeddings into classification. Could any one kindly share some pointers? Thanks!</p>

username: ptrblck
cooked: <p>Based on the description you’ve posted it seems the authors call the output features embeddings.<br>
This might be a bit confusing, as there are <code>nn.Embedding</code> layers, which are apparenrently not meant here.</p>
<p>If I understand the use case correctly, you could store each output feature of the <code>VGGish</code> model with its corresponding target, create a new classification model, and use these output features + targets to train this new classifier.</p>
<p>According to their claim, this classifier can be “shallower”, as the “embeddings” are so great. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Let me know, if that makes sense.</p>

username: kepler62f
cooked: <blockquote>
<p>you could store each output feature of the  <code>VGGish</code>  model with its corresponding target, create a new classification model, and use these output features + targets to train this new classifier.</p>
</blockquote>
<p>Cool. Thanks for the tip! Would this seem like reasonable steps to train a new model?</p>
<ol>
<li>Download audio wav samples from <a href="https://research.google.com/audioset/dataset/index.html" rel="noopener nofollow ugc">audioset</a>  as training data</li>
<li>Send each audio wav sample thru the VGGish to get a corresponding 128-dimension vector (output feature)</li>
<li>Define a Dataset comprising the VGGish output feature as input (x) and the corresponding target (y)</li>
<li>Using nn.module, define a “shallow” model using single layer…say Linear() .</li>
<li>train</li>
</ol>

username: ptrblck
cooked: <p>Yes, your approach seems reasonable!<br>
Let us know, how your experiments went. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: STU
cooked: <p>Hi, I want to get a 128-dimension feature of my own video data, what should I do?</p>

username: kepler62f
cooked: <p>For audio classification of video data, perhaps you could first extract the audio into wav file and slice them into short audio clips using a tool like FFMPEG with filename corresponding to the time segment of the video. The audio clips can be sent into the network to extract features.</p>

username: antonelse
cooked: <p>Hi <a class="mention" href="/u/kepler62f">@kepler62f</a>. How was the model?  I am working with the VGGish on a similar topic but I’m having some issued. Could you please share the code? I’m new in PyTorch and some example makes me more confident. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>