What is Machine Listening? (Part 3)
By: Yoonchang Han
Tags: Machine Learning, Audio, Machine Listening, Music, Artificial Intelligence

What is Machine Listening? (Part 3)

“I saw the news about security camera with scream detection 10 years ago.”

It’s like speech recognition in the early 2000s.

When people first encounter machine listening technology, some remember the news articles about smart security camera with scream and gunshot detection functionality, published more than 10 years ago. We can indeed find quite many articles about it on media, but it is very hard to find an actual working one in real-life.

Many technologies are getting polished over time, but sometimes new technology is based on a fundamentally different methodology with much more potential, coverage, advantage, and functionality, and we call it “next-generation”. It’s like speech recognition in the early 2000s. We all remember that speech recognition tech was there since a long time ago, but it was only able to understand a few words or very simple sentences with low accuracy. Not machine is adapting to human, but the human needed to talk like a robot for better recognition rate. This could make a wow-effect on people at that time, but it is hard to say that it was widely used in our daily life.

However, speech recognition is now a part of our daily life. It now understands much more natural real-world pronunciations, integrated into many smart devices, transforms how we interact with machines. I wouldn’t say it is perfect because it still has many points to be improved. For instance, you can just try “Hey, woo-woo” instead of “Hey, Google”. Or you can try “Aleka” instead of “Alexa”. I’m sure you will get what I mean by many points to be improved.

Then, next-generation is end-to-end deep learning.

Similar things are happening to modern machine listening technology. Conventional sound recognition is limited to identifying a few sounds with frequent false alarms from similar sounds. It is based on the rule-based method which uses handcrafted spectral features which are fully manually engineered by human knowledge and observation. This doesn’t mean that Machine Learning(ML) techniques are totally new in this field. ML has been widely used and the state-of-the-art algorithm in this field until 2016 was actually a combination of handcrafted features with ML classifiers.

Modern machine listening can now do much more things. First, it provides significantly better performance in the real-world. Previous approaches were not adaptive to various environment and situations whereas end-to-end deep learning(DL) can be trained with various real-world cases, which significantly improves robustness and reducing false-positive rate.

Second, it has great potential to support a huge number of target sounds. For the conventional method, the performance drop is kind of unavoidable problem because all the rules should be changed according to new target classes. But for the deep learning-based algorithm, more target sounds means that it can extract better feature from the input data, hence performance can be even more improved and it can remove false alarm caused by similar sounds. It is easy if you think detecting certain sound is equal to detecting all other sounds to be negative.

Third, and most importantly, it can be generalised much more than before. If we use deep learning for a specific device or microphone only, it worth nothing compared to the conventional approaches but introducing heavy computations only (Probably it worth in marketing perspective though).

Most importantly, with the proper use of DL and audio signal processing techniques, we can achieve an extremely generalised system which can be used any kind devices, microphones, and environments. It is highly critical for the future of machine listening because we will be surrounded by thousands of IoT devices soon, all running in different environments with various microphones, also we can never obtain the recording information of all videos on Youtube.

What makes it so difficult?

Making a generalised solution requires deep understanding in audio signal processing. Just name a few, there are sampling rate, codec, types of microphone, analysis window size, and so on. It is not simply about applying deep learning techniques on the data, because it will only work well for demo.

Next, there are millions of different sounds. For instance, dog barking might look simple, but there are thousands of different species with different ages. Also, describing sounds into human language can be sometimes ambiguous, or the same sounds can be described in different words. Key jingling sound can be described as a key jingling but it is not wrong to say metal jingling or keychain sound. And there are many different sounds in a different culture like the acoustic scene of the metro in Seoul would not be the same as the underground in London. Plus, some sounds are really difficult to collect as well. We might not be able to collect real-world car crash sound in a short time, because it is hard to guess when it happens where.

A common myth about the amount of data

The amount of data is indeed highly important for deep learning. That’s why many people say data is the new oil. However, I would like to emphasise that more data does not necessarily mean better accuracy. In fact, there are lots of other things that are related to the final performance of the system.

Amount of data needed is different for different sounds. For instance, machine-generated sound like a siren is always the same, so we wouldn't need lots of data for it. On the other hand, ambulance siren in a tunnel with 60km/h would be really difficult to obtain. In this case, simulating it with signal processing technique will be a much better way.

Various quality rather than high, correct data rather than more

Also, performance is saturated once it reaches a certain amount of input data from the same audio source. It also depends on the target sound, but it is clear that performance does not increase just because of more data is used for the training. A more important factor for performance, especially for the generalised system, is the diversity of data. Finding patterns from the various condition will allow the DL model to learn the unique pattern of the target sound better rather than overfitted to the recording device or environment.

Finally, the quality of the data is extremely important. High quality here doesn't mean studio-recorded clean audio because we will never face a studio-like clean environment in a real-life. It means appropriate audio samples with correct labels. If there are lots of wrong or noisy data in the training set, it will be harmful to the system performance.

To summarise, simply pouring time and money on data is not a way to make a DL system stronger. Making an efficient, appropriate, and sustainable data collection pipeline associated with DL architecture and data augmentation technique, and focusing on the quality of data rather than quantity will be a key for successful machine listening system.

Where it is heading towards

Deep learning-based machine listening is already powerful by its superior performance. But apart from performance, it has even greater potential which can make our life much more convenient by providing generalised sound cognition ability. This will make audio information to be used not only for a simple trigger-action but also for context-aware autonomous systems and humanoid robots.

In my opinion, modern machine listening will begin with replacing existing sound recognition systems first, then move towards supporting more target sounds (wider width) with more detailed information (deeper depth). Now, the term machine listening is used mostly for environmental sound detection, but it will gradually evolve to integrate speech analysis and music information retrieval as well under one umbrella to ultimately achieve a human-like understanding of acoustic information.