One Class Learning in Manufacturing: Autoencoder and Golden Units Baselining
By: Edward Elson Kosasih
Tags: Machine Learning, Keras, Manufacturing Analytics, Data Science, Industry 4 0

Recently I’ve been working with manufacturing customers (both OEM and CM) who want to jump on the bandwagon of machine learning. One common use case is to better detect products (or Device Under Test/DUT) that are defective in their production line. Using machine learning’s terminology, this falls under the problem of binary classification as a DUT can only pass or fail.

However, training a binomial classifier that requires samples from both cases of pass and fail turns out to be impractical. There are two reasons why.

Imbalanced data. Manufacturing processes are optimized to produce devices with high yield i.e. as little defect as possible. A typical scenario that happens when historical production data was collected is this: huge amount of good/pass data is obtained yet bad/fail data is hardly present. Underrepresented defective samples. There are many different type of defects that could result in failure. To enumerate each one of them is already difficult, let alone collecting enough samples from all. On the other hand, good data are typically alike and easier to define.

A slight philosophical discussion (Feel free to share your opinion, I could be wrong about this): Borrowing the idea from optimization theory, if we define goodness as a global optima that we are striving for and defects as sub-optimal samples, then there should only be one definition of goodness and infinitely many defectiveness (assuming a continuous space). Of course in practice, sub-optimal samples that are tolerantly close to global optima are accepted as good (due to imperfect processes in the world). But those that are extremely far away will be deemed as defective.

Our depiction of good and bad samples in manufacturing. Good samples are dense, defective samples are sparse.

Therefore, we need to find another training paradigm that is more suited for this scenario. Ideally, that new model must be able to learn only from good samples and reject defective ones that look immensely different with the former. One Class Learning is a good candidate.

“Wait that sounds familiar… Isn’t it just Baselining with Golden Units?”

said a six sigma expert during one fine afternoon in a meeting room. Yes indeed! It turns out that One Class Learning is not a foreign concept in manufacturing. It’s just that the terminology differs. In quality control, engineers will typically collect Golden Units, or known good components and perform some measurements on them (a.k.a. feature engineering). Next, tolerance limits will be calculated based on the distribution of measurements. One statistical method that can be used to calculate such limits is called Part Average Testing (published by the Automotive Electronics Council). During production, devices whose measurement values are within the limits will be passed while those outside are failed.

Part Average Testing Limit Calculation

Part Average Limits applied to distribution of measurement values

The whole baselining process can be visualized in the following diagram.

Golden Unit Baselining

Feature Engineering

The tricky part is deciding what features to extract. Traditionally, this requires a deep domain expertise. As a result, it is expected to see hundreds of algorithms deployed in a production line, each has been carefully designed to identify the presence of certain defects. The idea of machine learning is to remove this manual intervention all along, and let the model discovers commonalities and differences from the humongous data that it has been provided with.

Nonetheless, don’t get me wrong, such “traditional” methods are actually quite accurate. I’ve worked with domain experts who could visually identify defects in certain devices in less than a second (I’m still amazed until now) — they have written algorithms that are very good at identifying those flaws. In my view, machine learning could act as a complement to existing algorithms.

For the rest of the article, I’d like to give a brief overview of how one class learning is done. I will go through a particular algorithmic implementation of one class learning called autoencoder neural network. However, I will not dive deep into the theories of autoencoder. I’ll leave it to another post.

We shall also assume that the collected measurements are in the form of images (e.g. a visual inspection system). A dummy dataset from MNIST handwritten digits will be used.

Calculating Part Average Limits based on Autoencoder Reconstruction Error from Golden Units

Our experiment setup goes this way: Assume that images from good devices look like the digit 1. Therefore, during training we will only input images of digit 1s. To test the efficacy of this model, we will test it against an unseen set of digit 1s (good samples) and digit 2s (bad samples). The hypothesis is that reconstruction error for digit 1s will be low and digit 2s high. If this is true, we can then use reconstruction error as a feature. The whole pipeline is drawn in the following figure (notice the similarity with Golden Unit Baselining).

One Class Learning

Most of the code in this tutorial comes from Keras’ basic tutorial on autoencoder.

First, let’s limit the amount of GPU resource that tensorflow-keras will consume. By default it will attempt to reserve all available memory. In this case, we’ll only take 40% of the available memory.

import tensorflow as tf

import tensorflow.keras.backend as K """

Limit GPU memory consumption

""" tfgraph = tf.get_default_graph() with tfgraph.as_default():

config = tf.ConfigProto(

intra_op_parallelism_threads=12

)

# 0.4 means only 40% of GPU memory will be used

config.gpu_options.per_process_gpu_memory_fraction = 0.4

tfSess = tf.Session(config=config)

K.set_session(tfSess)

After that we can load the MNIST handwritten digit datasets from Keras. Subsequently, we normalize the image intensity values from 0–255 to 0–1. This allows us to use a sigmoid activation function at the last layer of the autoencoder (since the output domain will be between 0 and 1 too). If we stick to 0–255 at the input, we will have to scale the autoencoder’s output accordingly as by default there’s no activation function that has an output range of 0–255.

Tips: Make sure that your neural network’s input and output has the same domain. Otherwise your training loss will not converge properly, no matter how many epochs are used.

# import library dependencies

import numpy as np

import matplotlib.pyplot as plt

from keras.datasets import mnist # import mnist dataset

(x_train, y_train), (x_test, y_test) = mnist.load_data() # normalization

x_train = x_train.astype('float32') / 255.

x_test = x_test.astype('float32') / 255.

x_train = np.reshape(x_train, (len(x_train), 28, 28, 1)) # adapt this if using `channels_first` image data format

x_test = np.reshape(x_test, (len(x_test), 28, 28, 1))

We then split the data into good samples (images of 1s) and bad samples (images of 2s).

# get trainData of digit 1 only

trainX, trainY = zip(*filter(lambda x: x[1] == 1, zip(x_train, y_train)))

trainX = np.array(trainX) # get testData of digit 1 only

testGoodX, testGoodY = zip(*filter(lambda x: x[1] == 1, zip(x_test, y_test)))

testGoodX = np.array(testGoodX)

Good Samples i.e. images of 1s

# get testData of digit 2 only

testBadX, testBadY = zip(*filter(lambda x: x[1] == 2, zip(x_test, y_test)))

testBadX = np.array(testBadX)

Bad Samples i.e. images of 2s

To keep things simple, we’ll use the default definition of autoencoder from Keras. The only additional thing that we did here is to flatten the encoded layer into a 2-neuron bottleneck layer in the middle. Note that this choice is arbitrary. I didn’t perform any hyperparameter search in this experiment as I intended this to merely be a simple introduction. As for the choice of 2-neuron, this will be useful later on when I try to visualize the encoded embeddings in a 2D space.

https://blog.keras.io/building-autoencoders-in-keras.html from keras.layers import Input, Flatten, Reshape, Dense, Conv2D, MaxPooling2D, UpSampling2D

from keras.models import Model

from keras import backend as K input_img = Input(shape=(28, 28, 1)) # adapt this if using `channels_first` image data format # decode

x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)

x = MaxPooling2D((2, 2), padding='same')(x)

x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)

x = MaxPooling2D((2, 2), padding='same')(x)

x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)

# at this point the representation is (4, 4, 8) i.e. 128-dimensional

encoded = MaxPooling2D((2, 2), padding='same')(x) # flatten and compress

shape = K.int_shape(encoded)

bn = Flatten()(encoded)

bn = Dense(2)(bn)

bnRec = Dense(shape[1] * shape[2] * shape[3], activation='relu')(bn)

encoded = Reshape((shape[1], shape[2], shape[3]))(bnRec) # encode

x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)

x = UpSampling2D((2, 2))(x)

x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)

x = UpSampling2D((2, 2))(x)

x = Conv2D(16, (3, 3), activation='relu')(x)

x = UpSampling2D((2, 2))(x)

decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x) # create encoder

encoder = Model(input_img, bn)

# create autoencoder

autoencoder = Model(input_img, decoded)

autoencoder.compile(optimizer='adadelta', loss='mse')

autoencoder.summary()

Autoencoder Summary

The next step is to train the autoencoder model. We call the .fit() function, and supply it with images of 1s. As for validation, we put images of 2s. Note that validation dataset will not be taken into account during backpropagation. I put it there so that we can see how the reconstruction errors between images of 1s and 2s diverge as training goes on.

history = autoencoder.fit(trainX, trainX,

epochs=30,

batch_size=128,

shuffle=True,

validation_data=(testBadX, testBadX))

We can then plot the reconstruction error over epochs for both images of 1s (good) and 2s (bad). It can be observed that error for 1s decreases over time while 2s stays high. This means that the autoencoder has learned to reconstruct good images that it has been trained with (1s), but not the bad ones (2s).

# summarize history for loss

plt.figure(figsize=(10, 3))

plt.plot(history.history['loss'])

plt.plot(history.history['val_loss'])

plt.title('model loss')

plt.ylabel('loss')

plt.xlabel('epoch')

plt.legend(['good', 'bad'], loc='upper left')

plt.show()

Reconstruction Error for Good and Bad Samples over Time

The errors above are only point estimates. We need to see the whole distribution of reconstruction error in order to calculate the tolerance limits (e.g. Part Average Testing Limits). From the chart below, we can see that indeed bad images have higher reconstruction error than good images.

# predict

predGood = autoencoder.predict(testGoodX)

predBad = autoencoder.predict(testBadX) # reconstruction error

rec1 = np.sum((predGood - testGoodX)**2, axis=(1, 2, 3))

rec2 = np.sum((predBad - testBadX)**2, axis=(1, 2, 3)) # histogram

plt.figure(figsize=(10, 5))

plt.subplot(2, 1, 1)

plt.hist(rec1, bins=30, range=(0,300), color='g')

plt.tick_params(axis='x', which='both', bottom=False, top=False, labelbottom=False)

plt.ylabel("count of good images") plt.subplot(2, 1, 2)

plt.hist(rec2, bins=30, range=(0,300), color='r')

plt.xlabel("reconstruction error")

plt.ylabel("count of bad images") plt.show()

Reconstruction Error for good (Top) and bad (Bottom) images

We can now calculate the Part Average Limit. Following the golden units’ baselining experiment, we will only use errors from good samples and calculate the limits accordingly. The resulting threshold is visualized below.

# Part Average Limit

robustMean = np.median(rec1)

robustStd = (np.quantile(rec1, 0.75) - np.quantile(rec1, 0.25)) / 1.35

PAUL = robustMean + 6 * robustStd

PALL = robustMean - 6 * robustStd # histogram

plt.figure(figsize=(10, 4))

plt.hist(rec1, bins=30, range=(0,300), color='g', alpha=0.5, label='good samples')

plt.hist(rec2, bins=30, range=(0,300), color='r', alpha=0.5, label='bad samples')

plt.xlabel("reconstruction error")

plt.ylabel("count of images")

plt.axvline(x=PAUL, c='b')

plt.axvline(x=PALL, c='b')

plt.legend()

plt.show()

Part Average Limit on Reconstruction Error

The following are some samples of the autoencoder’s outputs for both good and bad images. We could visually see that images of 1s can be well reconstructed, while 2s are not. Interestingly, the autoencoder’s outputs on images of 2s look oddly similar to 1s. I don’t have a theoretical guarantee on this, but empirically it seems that the decoder part of the model has only learned to construct 1s with certain degrees of variety (e.g. angle and strokes), but not other digits.

Autoencoder’s reconstruction of good images

Autoencoder’s reconstruction of bad images

To further visualize how the decoder constructs images of 1s with different strokes and angles, we could look at the encoding produced in the bottleneck layer of the autoencoder model. We would like to see if there are any structures there. Could there be clusters of images with certain strokes / angles?

Since we have compressed the information into just a 2-neuron layer, we can easily represent each image as a 2-dimension vector and visualize all of them in a 2D plot.

testGoodEncodedX = encoder.predict(testGoodX)

np.shape(testGoodEncodedX) plt.figure(figsize=(8, 8))

plt.scatter(testGoodEncodedX[:,0], testGoodEncodedX[:, 1])

plt.show()

2D Scatter Plot of Encoded Images

Looking at the distribution, it seems that our initial suspicion that clusters of digits with certain strokes/angles exist is incorrect. Perhaps the pattern looks more like a spectrum instead of cluster? To visualize this, we can replace every dot in the plot above with its’ original image.

from matplotlib.offsetbox import AnnotationBbox, OffsetImage fig, ax = plt.subplots(figsize=(20, 20)) for x, xEnc in zip(testGoodX, testGoodEncodedX):

imagebox = OffsetImage(x[:,:,0], cmap=plt.cm.gray)

ab = AnnotationBbox(imagebox, [xEnc[0], xEnc[1]], frameon=False)

ax.add_artist(ab)



plt.xlim((testGoodEncodedX[:,0].min(), testGoodEncodedX[:,0].max()))

plt.ylim((testGoodEncodedX[:,1].min(), testGoodEncodedX[:,1].max()))

plt.show()

Indeed, we found that there are varying degrees of images of 1s in the embedding space. Going diagonally from the upper left to lower right quadrant, we see the various spectrum of angle. Meanwhile in the off-diagonal direction, we see thicker strokes as we get closer to the lower left quadrant.

Conclusion

I must first apologize that I couldn’t use dataset from a real production line. Nevertheless, I hope that this simple tutorial gives you a better insight of how one class learning and golden unit baselining with Part Average limit are quite similar in practice. I have provided a simple python code walkthrough in order to show aspects of one class learning that we usually analyze in practice.

Disclaimer: Having worked with data from real factory floors, I can testify that the reality is much messier than this simplistic sample of image of 1s and 2s. There are a lot of preprocessing that needs to be done. Moreover, not all of the data labels could be trusted — as these are annotated by humans, there could be errors due to inconsistency. Nonetheless, the fundamental concept stays the same.