Package: pamai
DENet training codeAuthorArthur ZuckerChris RauchInstallationClone this repositoryModelThe model is available at the original github repository :git@github.com:MiviaLab/DENet.gitInformationThe original DENET code is in tensorflow, but I need to use both YOLOR and DENET, and given that YOLOR is in pytorch, I have to convert the code to pytorch. That's also a good exercise, thus I will do it.Moreover, I am using my template implementation which allows to easily switch between model and data type, integrating this also requires a bit of work.TODOCreate datasetAddget_model,get_loss,get_optimizerfrom the namesCreate dataloadersConvert bothinit.pyandlayers.pyto pytorch (since its now in tensorflow)Create modelCreate training agentAdd utils and choose metricSample training on subsetClean hparamsUse profiler to check training time and dataleaksUse time distributed forvisualizefunction (process full audio)Use fp16 and mixed precision with apex!Usehttps://tut-arg.github.io/sed_eval/sound_event.htmlvar validation metricsCreate DockerDetails on the readmeAdd to my websiteIssue :output of sincnet and denet is the label at each window of 50ms. Thus the labels that I use should also be of the same format. Which means that when I select an audio file to use, randomly select a part of it? then extract the label from the sample.
Sample available labels : {500->550},{...},{...} a list of ranges of samples where we know that a flying foxe shouted!DENET's input : 1 sequence of 10 frames of 50ms audio (thus 500ms sequences). Per-frame labels.
SincNet's input : 1 chunk of 200ms, 10ms overlap, 128 batch size. Labels are for each wav file, a single label, which is a bit strange.AnnotationsRealized that the annotations were a bit... Strange?
In theFileInfo.csvfile, each call's start sample and end sample is labeled. For example, for the following recording (visualized using sonic Visualiser), 2 calls are visible, the label is the following:File nameVoice start sample (1)Voice end sample (1)Voice start sample (2)Voice end sample (2)120601011047196133.WAV4755395094298537349426But in theannotation.csv, the corresponding labels are the following:FileIDContextStart sampleEnd sample301298537607056Not only is a single call labeled, but the tail of it is also marked ascontext = 1. This makes it a bit strange. Though we can actually see that there is some noise/signal at the end of the recording, which annotation should we trust.I decided to merge the to files, ignoring the files without labels. This allows for a subsampling of the already huge dataset.DataloaderThis has a huge impact on the way I will chose to label the data from a given random begin and end sample.
I don't have a lot of options.Use call annotations, without processing it. That's the simplest solution but not very intelligent. Though given that the GRU uses previous samples, it should be able to automatically know tht this or this is not a call anymore (tail of a call).Look into unsupervised learning, wince some of the calls are labeled, others are not. In this case I should use the per-label calls. Then, process themusing the phoneme annotationto at least remove the silence parts. Since the GRU uses previous samples, I guess it makes sense to say "this is still part of the audio event" and process it as a whole.Solution: Will simply process the labels beforehand. Dataloader will load the random sample's label.
I should also subsample the dataset to keep mostly calls that have a class or emitter label. This will both reduce the database size and allow for easier data augmentation (adding other animals)One thing that can be said is that bat calls (and here I don't mean phonemes I mean fully "calls") can be easily grouped together based on the separation. YOLOR's results give phonemes, but we can simply infer the actual call. This is not very accurate, but can work.In process :Given the previous section, I will write a data-processing function in/utilswhich will write the.csvlabel file to the/assetsrepository.
Then, labels are given to a random audio sequence based on its position. If it is between the miniumstart_sampleand maximumend_sampleI gave it a the label1for now, which is that of a bat call. Giving the context or emitter label is straightforward but need to be done@TODOVisualizationI create scripts to add the visualization of the detection on validation data at val time. Again, since the input of the network is 200ms (or more)I can't feed it with the full audio can I?If i can't, I need to write a script to process the full audio, and give individual labels.More ideas:Add FF call to background sounds and use it as training. Hashizume's FF already have Background noises.Add other animal calls, rain, wind and other degradations in the transformations.Improving the datasetOur dataset only has positive instances -- bat calls are almost always present in recordings -- thus we need to augment it with more recordings, were other events happen. We have a huge enough dataset, which means that we can subsample it, and combine it with random noises from famous benchmark dataset? The audio conditions will be different :confused:Dataset description :293 235 total number of different audio files. Each containing at least 1 call.91 081 total number of labeled audio. Labels are :Emitter, Addressee, Context, Emitter pre-vocalization action,	Addressee pre-vocalization action,	Emitter post-vocalization action,	Addressee post-vocalization actionIn the case of emitter, 7858 and 44075 calls have no labels. ("-" sign in label means that the bat is either emitter or addressee)39147 is the number of audio files for which the emitter is known.51 different individuals44 different individuals emitted calls60813 is the number of audio files for which the context of the call is known.  (General and Unknown labels excluded)31922 is the number of audio files for which the gender of the emitter is known. (puppies without a known gender excluded)0UnknownUnknown context6401SeparationEmitted (rarely) by adults when separated from the group.5042BitingEmitted by a bat after being bitten by another.17883FeedingThe interaction involves food.66834FightingThe interaction involves intense aggressive physical contact.79635GroomingThe interaction involves one bat grooming another.3836IsolationEmitted by young pups.57147KissingThe interaction involves one bat licking another's mouth.3628LandingThe interaction involves one bat landing on top of another.169Mating protestEmitted by a female protesting a mating attempt.233810Threat-likeThe interaction involves contactless aggressive displays.106511GeneralUnspecified context. The interacting bats are usually 10-20cm apart (in other interactions the bats are usually closer).2962712SleepingThe interaction occurs in the sleep cluster.DENetImplemented it, but for now I can't use the attention because the MLP takes too much memory.Dockerdockerbuild.--network=host-tpytorch-latest-pamai
dockerrun-it--rm--namepytorch-container--network=hostpytorch-latest-pamaibash