Correlation between Machine Learning and Signal Processing | Towards Data Science

As a design engineer, I am increasingly exposed to more complex engineering challenges — some of which require a blend of multidisciplinary expertise. The Advanced Machine Learning and Signal Processing course provided me with the window to understand how machine learning and signal processing can be integrated and applied together.
Although the title of the course sounded daunting at first, it is not difficult to follow. However, prior knowledge of statistics and calculus will come in handy during the later parts of the course. This course is delivered by IBM Chief Data Scientist Romeo Kienzler and Nickolay Manchev and is offered by Coursera as part of a 4-course IBM Advanced Data Science Specialization.
The course is structured into 4 weeks, whereby Week 1 to Week 3 covers machine learning concepts and algorithms. On Week 4, the signal processing part is covered and we are shown how to integrate signal processing and machine learning.
This week sets the stage for the upcoming weeks and we started off exploring linear algebra used in machine learning. We are first introduced to different data objects: scalar, vector, matrix, and tensor; as well as the mathematical operations that can be applied to them (dot product, vector-matrix multiplication).
Data objectsA scalar is one dimensional and can be any integer, ie:
1, 5, -18
A vector is a group of scalars and can only contain one datatype, ie:
(0,11,135)
A tuple is similar to a vector, but it can contain multiple data types, ie:
(0,1,7.8) #notice the integer and float datatypes
Matrices are a list of equal-sized vectors and tensors are matrixes in 3 or higher dimensions. Tensors are useful for image processing (for example, 1 dimension is used for width, another for color, and another for height).
We then learned about higher-dimensional vector spaces and the intuition to picture and split them.
To split in 1-dimension, we use a point.
To split in 2-dimensions, we use a line.
To split in 3-dimensions, we use a plane.
To split higher than 3-dimensions, we use a hyperplane.
There are 2 types of machine learning:
Supervised Learning: Trained on datasets with output labels, i.e. a flag is assigned to a data point. The algorithm aims to solve the Y = f(x) equation by predicting what Y will be when we change x. Supervised learning is broken down into:
Supervised Learning: Trained on datasets with output labels, i.e. a flag is assigned to a data point. The algorithm aims to solve the Y = f(x) equation by predicting what Y will be when we change x. Supervised learning is broken down into:
Classification — predicts a discrete value i.e.: “True/False”
Regression — predicts a continuous value ie: “Price of mobile phones in the future”
2. Unsupervised Learning: Trained on datasets without output labels. The algorithm attempts to discover patterns from the input data by grouping similar data together. An example of unsupervised learning is Clustering where data are segregated into clusters (More on this during Week 3).
PySpark allows us to modularize data pipelines. By using the Spark libraries of StringIndexer, OneHotEncoder, VectorAssembler, and Normalizer, we can ensure all the pre-processing is fully aligned and taken care of before the ML model takes place. The idea is to condense these pre-processing parts into “Pipelines” and take advantage of this flexibility to quickly switch between different training parameters to optimize for the best machine learning model.
from pyspark.ml import Pipelinepipeline = Pipeline(stages = (string_indexer, encoder, vectorAssembler, normaliser))model = pipeline.fit(df)                    prediction = model.transform(df)
All the coding assignments are completed in IBM Watson Studio and this week’s hands-on approach builds the foundation for the steps required to set up and run a machine learning project.
Common machine learning models and concepts are presented this week and the lectures are a mixture of explaining theories and demonstrating how to use an algorithm with Apache SparkML. The course presenters did a great job to explain the following concepts that are crucial for a solid machine learning foundation.
Linear Regression and Batch Gradient Descent A Linear Regression model is a model based on supervised learning and can be described by the following function:
y = w0 + w1x1 + w2x2 +w3x3...+wnxn ; whereby w is the feature weight
Its main function is to predict a dependent variable (y) based on the independent variables (x) that are supplied. Its counterpart, Logistic Regression is used to predict a discrete value. The feature weights are determined using the Gradient Descent strategy and Batch Gradient Descent happens to be one of these optimization strategies.
Batch Gradient Descent is guaranteed to find the global minimum of the cost function — given there is enough time and the learning rate is not too high. (imagine trying to reach the bottom of a valley). It is slow over a large dataset as the model is only updated after evaluating the error for every sample in the dataset.
We then briefly learned about splitting our dataset into training and validation datasets, as well as the method to evaluate if the prediction results are underfitting or overfitting.
The Naive Bayes TheoremEven though this topic is challenging, Nickolay managed to summarize it into a digestible yet, in-depth manner. We were shown the intuitions and mathematics behind the Naive Bayes theorem such as:
Sum Rule and Product Rule — allows us to solve most probabilities problems
The Gaussian Distribution, or more commonly known as the Normal Distribution — data that is symmetrically distributed by the mean.
The Central Limit Theorem — all the means of a sufficiently large enough sample will end up in a Gaussian Distribution
The Bayesian Interference — the probability of a hypothesis is updated as new evidence becomes available
Up until this point, there is already a lot of information for Week 2, but we’re not quite done yet!
Support Vector MachinesWe moved on to a quick introduction of Support Vector Machines (SVMs) and learned how they lost ground to Gradient Boosting in terms of popularity. SVMs are linear classifiers and to turn them into a non-linear classifier, the “kernel” method can be deployed. This method, or “trick” that Romeo calls it, transforms training data onto the “feature” space and allows a hyperplane to cleanly separate data points. The examples in this part of the course were intuitive, beginner-friendly, and easy to remember.
This week was nicely wrapped up with Decision Trees, RandomForests, and Gradient Boosted Trees. We were shown the methods of sampling and the pros and cons of each model. There is also a coding exercise that ties up what we learned in Week 1 and Week 2 which demonstrates the application and prediction accuracy of a GBTClassifier using SparkML.
Ifelt stretched the most this week with the introduction of more advanced topics such as Clustering and Principal Component Analysis (PCA). We learn about unsupervised machine learning where we try to fathom the distances between point clouds. There are various methods to measure distance i.e. subtracting between 2 points; Pythagoras’ Theorem, Euclidean Distance, and Manhattan Distance.
We learned about clustering using K-means algorithm (the ‘Hello World’ clustering algorithm) and the Hierarchical algorithm. K-means works in a way that we specify the number of expected clusters, then the algorithm will draw a hypersphere in the 3D space and tries to group each data point from the point clouds into clusters based on the nearest center.
The Curse of Dimensionality We often have to choose the trade-off between having faster computing time and having more dimensions (features) while maintaining a certain level of accuracy. The number of samples becomes increasingly sparse as we add more dimensions and distances between the data points lose meaning. To solve this problem, PCA is used to reduce the number of dimensions while maintaining the distances between the original data points as much as possible. These distances are crucial in classification as they allow us to predict data-points into different classes.
There are a couple of quizzes this week, but they are sufficient to test my understanding of the material. Similar to the previous weeks, there is a coding exercise to consolidate the material learned so far.
I was looking forward to this week the most as we finally cover the signal processing part — a topic that I am very keen to explore.
SignalsThere are 3 variables to describe a signal: 1. Frequency — how many occurrences of the signal2. Amplitude — the strength of the signal 3. Phase shift — how much horizontal offset a signal moved from its original position.
Most signals can be generated or described using the following formula:
y(t) = Asin(2*pi*ft + φ) whereby, A = amplitude, f = frequency, t = time, φ = phase shift
Fourier transform Fourier Transform allows us to decompose a complex signal — be they music, speech, or an image into their constituent signals via a series of sinusoids.
In terms of images, we can decompose an image’s colors into its RGB constituents.
Nickolay used the example of a piano chord to identify its individual notes using Fourier transform. Sound waves are generated when vibrations travel across a medium. When we have different levels of vibrations, we will obtain different waveform amplitudes. These amplitudes can be summed, or superpositioned to obtain the net response of each individual amplitude.
We can study waveforms from two domains —the time domain and the frequency domain. Fourier Transform and Inverse Fourier Transform translates between these domains. (The former from time to frequency domain and the latter from frequency to time domain).
The chapter is concluded by linking signal processing and machine learning through the Wavelets topic. Fourier Transform works well on stationary signals but in real life, we constantly work with non-stationary signals. Fourier transform cannot provide information on when specific frequencies occur in these signals. Hence, wavelet transform is required to overcome this limitation using a scaleogram, which is a visual representation of a wavelet transform. To fully understand wavelet and Fourier transform, Nickolay explained the maths behind them.
The final programming assignment is an exercise on how to classify a signal using machine learning. For someone looking into how sensor data and machine learning works, the topics and exercises for this week are truly fascinating and interesting.
I find the products I am working on are getting smarter and fitted with more sensors. Often, having multi-disciplinary skills is advantageous to deliver a project. This course “kills 2 birds” for me by providing some of the important topics of signal processing and machine learning.
The delivery pace is good, and the explanations are decent enough for a professional or a curious casual learner. For some of the topics that are taught — especially the Bayesian theorem, PCA, and Wavelet transform parts — further study and research are important to gain a deeper understanding.
There are other courses that cover more in-depth about machine learning and signal processing but what I truly enjoyed about this course was the connection and ‘blending’ of these 2 topics that are so interrelated to each other.
With IBM Watson Studio being used in the assignments of this course, there is another added advantage to be exposed to IBM’s cloud tools and services. The course seems to be providing the learner to choose their own difficulty for the assignments. It is not challenging to pass the assignments as there is a lot of hand-holding and guidance. But there is always an option to really understand the code that is shown and that takes time for beginner programmers.
I am looking forward to applying what I learned from this course and I hope by writing this summary, I solidify my learning and you have learned something as well.
Some rights reserved