Lessons Learned “Analysing IOT Sensor Data”
Tags: Pandas, Apache Spark, Big Data, Pyspark, Data Analytics

Introduction

I analysed sensor data of an automated hardware quality testing system.

We are talking about millions of rows cut in to 124MB files.

Tech-stack

I was using Apache Spark, PySpark and Python for transforming and analysing the data. For the visualisation i used pandas and seaborn library.

Tasks

Set up environment (Apache Spark, PySpark, ..) #BigPart

Load data (load and convert the data into a Spark DataFrame)#SmallPart

Get Insights (speak with business/operator about the data in order to detect hidden treasure)#BigPart

Transform data (bring the data in a usable format cast values text to int or extract information from strings with regex,…)#BigPart

Analyse Data (get statistical metrics and create diagrams)#SmallPart

Lessons Learned

Sensor data are usually log based and already in pretty god quality meaning, no missing values or format changes since the entries were done by machine and not by human interactions.

beside the ID it’s important to have a date timestamp for each entry in your log files to get a correct timeline of the data.

Since we are talking about hardware components the environment can play a big role in the durability of components or the fluctuation from certain input values.

I enriched the data with temperature values and found out that the temperature never changes or only +- 0.01°C.

The reason for this was, the sensors where mounted in an air-conditioned room that was already sensor monitored.

This brings me to another important key point.

Speak with the Business/Operators to get background information about the data to find hidden treasure, otherwise data can look really ode.

I found out that they included in some channel input names %IW1.1.2.1 really important information like the second last number 2 indicates the module where I then could sort by later one.

If you look at something and don’t know what to do with it, usually they packed some information together. Once extracted, the new information can be of great value.

In order to visualise data it makes sense to sample the data first with:

df.select([‘column’]).sample(False,fraction=0.1, seed=3).

Not only because visualisation can be done only in memory also if you have to many data points the diagram gets overloaded and doesn't look pretty anymore.

For visualisation of grouped values, one of the best diagrams is:

sns.jointplot:

To get meaningful insights it’s also important to get data where the process/test failed. Since most of the runs are perfectly fine there only a few where a product did not pass the test. Gathering data can be a long process and needs to be done upfront, it’s important to make it clear to the business/operator to save explicitly the data where a product did not pass the test (especially if it’s done manually).

Keep in mind, to process such amount of data, a dedicated computer is needed.

Even it’s possible to run everything on a private computer, it’s not recommended to do so since you will be blocked for hours or even days.

Just save the struggle of wasting your time and rent some dedicated cloud instance/cluster.