Building an End-to-End Defect Classifier Application for Printed Circuit Boards | by Sean McClure | Towards Data Science

The Printed Circuit Board (PCB) is in most electronic products, mechanically supporting and connecting components along conductive tracks. Their prevalence underlies the modern electronics industry, with a world market exceeding $60 billion1.
PCBs are prone to a variety of defects that impede proper manufacturing, costing companies money. Defects such as shorts, spurs, mousebites, and pinholes cause issues like current leakage and open circuits, quickly degrading performance or rendering PCBs useless. PCB manufacturers must strive to ship defect-free units to remain competitive and ensure customer satisfaction.
Failure analysts are employed to ensure defects are kept to a minimum. But inspecting and diagnosing defects manually is challenging. There are multiple defects to detect, an extremely low tolerance for errors, and significant expertise required to reliably recognize and handle flawed units. Considering the time to train new analysts, and the cognitive load required to ensure reliability, an automated solution to detecting and classifying defects in PCBs is in demand.
In this article we will build a full PCB Defect Classifier that automates the task of detecting and classifying defects in printed circuit boards. I always encourage AI practitioners to build full products rather than throwing models over the fence to developers and other engineers. Your machine learning must exist in concert with a full application; thus only through its survival inside a holistic product can you be certain you’re building for users, not engineers.
Play the following video to see the application we will be building in this article:
In this article I will go over the application’s back-end and front-end code. I will walk through all of the code related to defect detection, extraction, and prediction. Thus the machine learning aspects will be covered in full. However I will only review the high-level aspects of the actual application pipeline code and front-end code, in order to keep this article to a reasonable size. Note that the entire application and its code are available on GitHub, thus readers have access to everything. I encourage others to clone and play with the project as they see fit.
This is always the best way to learn. I welcome questions in the comments section for those needing support (you can also @ me on Twitter so the support benefits others more publicly).
Let’s get started.
Google Colab
Flask
Deep Learning: fastai
Image Processing: tetryonai
User Interface: Azle
Notebook: PCB_Defect_Extraction_and_Classification.ipynb
Defect Turker: (for creating labelled image data)
PCB Defect Classifier: (automated visual inspection system and classifier)
There are a number of challenges we face when attempting to automate defect detection and classification2. A few of these were already mentioned in the introduction. The fuller list includes:
low tolerance for defects in units;
multiple defect types to account for;
low inference times required for automated solutions;
training data for ML models limited (usually narrow, domain-specific, and small);
traditional systems rely on hand-crafted features and significant expertise;
methods used by others often private.
We will need an approach that addresses these challenges while supporting a real-world application that effectively detects, extracts and predicts defect types in PCBs.
Given the challenges we face it makes sense to use a hybrid approach to build our classifier application, which combines the reference method and non-reference method.
Reference Method: the inspected PCB board (test board) is compared to a reference template image (template board) that has no defects, in order to allow analysts to identify differences.
Non-Reference Method: the inspected PCB board is checked for compliance against design principles, without relying on template images.
The reference method is relatively straightforward to implement but has large storage requirements (holding all the images). The non-reference method doesn’t suffer from storage issues, but can be limited in the number of defects found.
Building a product that automates both reference and non-reference methods may not cut down on storage and computational requirements, but if the manual detection and classification errors are reduced we gain massively from the increased and reliable throughput .
While we could attempt to train our deep learning model to classify the defects present on an entire board, this would be challenging. Take a look at the following example test and template PCB boards:
The board on the left is what our model needs to classify. We can see the individual defects by comparing to the template. But how likely is it that our model will be able to reliably classify a combination of defects? What would the class be called? How many unique defect combinations exist?
To lower the burden placed on our model we will first extract the individual defects from the test image. This can be accomplished by creating a differenced image and extracting the parts of the image that are different.
The following is the kind of differenced image we can produce using image processing in Python:
Notice the defects are highlighted in red. The above image doesn’t show all the defects since in this article we are only concerned with extracting 3 types. Capturing the other defect types of fairly trivial, and I will discuss this later.
Since we have a differenced image that highlights the defects in the test image we can also use image processing to extract the colored parts of the image:
This dramatically lowers the burden placed on our deep learning classifier. By extracting individual defects we convert the problem from a complex recognition problem into something akin to digit recognition (a problem that already works well with deep learning). This also means we are more likely to be successful using transfer learning, since there are numerous successful models in the OCR space. I will discuss transfer learning in more detail later on.
With this approach we can outline 5 major efforts:
DEFECT EXTRACTIONDEFECT LABELLINGDEFECT CLASSIFICATIONBUILDING A PCB DEFECT CLASSIFIER APPLICATIONRUNNING/TESTING THE APPLICATION
DEFECT EXTRACTION
DEFECT LABELLING
DEFECT CLASSIFICATION
BUILDING A PCB DEFECT CLASSIFIER APPLICATION
RUNNING/TESTING THE APPLICATION
Head on over to Google Colab and start a new Python 3 notebook. This will be our modeling environment. This is a Jupyter notebook environment that runs entirely in the cloud.
Google Colab allows access to an environment capable of running GPUs at no cost. Colab also makes installing and importing libraries a breeze. This is back-end as a service (BaaS), which helps (immensely) with rapid prototyping.
You have 2 options for getting the notebook into your own Google Colab environment.
You can simply upload the notebook contained in the GitHub project:
You can simply upload the notebook contained in the GitHub project:
2. Or you can mount a drive in Google Colab, clone the defect_classifier repo, and open the notebook from your Google drive.
2.1 Open New Python3 Notebook
2.2 Mount a Drive on Google Colab
Run the following in the first cell (play button, or shift-return on Mac):
You will be presented with a URL. Click it, get the auth code, return to your notebook and enter it. You now have access to your Google Drive inside Colab:
2.3 Clone Repo
...then copy the cloned notebook into your mounted drive:
2.4 Open Notebook from Drive
Now find the notebook inside your Google drive, as you would any other file, right-click, and choose open with Google Colab.
We will require Graphical Processing Units (GPU) to train our Deep Learning model in reasonable time. We can set the hardware acceleration in Colab by clicking on Runtime at the top, choosing Change runtime type, selecting GPU under hardware accelerator, and clicking SAVE.
First we need to build the defect extraction pipeline. To use less code I’ve created a small library that will abstract much of the image processing. I plan on continuing to add to this library going forward to stay tuned. For an intuitive yet detailed tutorial on creating libraries in R and Python see here.
Let’s import the library now:
The library is freely available on GitHub. We can install libraries from GitHub using pip as follows:
!pip install git+https://github.com/sean-mcclure/tetryon_ai.git
Note the ! before the usual pip command. This is the “bang” symbol commonly used when doing interactive computing. It allows our Google Colab environment to access the underlying shell.
We can now import the library:
from tetryonai.main import *
You can also click on the Files icon to the left so we can see the files we work with (click on Refresh as needed):
Let’s create folders to hold our differenced images, extracted defects, and the hold out images needed to test our model. We’ll use tetryonai’s directory function:
directory(**{    "choice" : "make",    "directory_path" : "difference_img"})directory(**{    "choice" : "make",    "directory_path" : "extracted_defects"})directory(**{    "choice" : "make",    "directory_path" : "hold_out"})
We will need access to a large number of test and template PCB board images. These are available on GitHub3 via the DeepPCB project. This provides us with a large number of images obtained from a linear scan CCD. Review this project for more details.
Clone the DeepPCB project into your Colab environment by running the following in the next cell:
!git clone https://github.com/tangsanli5201/DeepPCB.git
Hit Refresh in the Files pane to see the new DeepPCB folder. You can expand through the folder to see just how many images are available.
For this project we will use only the group00041 set of temp and test images as shown above. These are the images previously described in Figures 1-3.
Let’s write some Python for storing the group00041 temp and test image paths from our DeepPCB folder inside a dict. We’ll then use this in the next step to difference our images:
Now we will use image processing to difference each pair of temp and test images using the paths specified inside the dict we just created. We also need to hold out a set of these images to test in our application. I will combine these operations into one codeblock for convenience:
We are using tetryonai’s copy_files function to copy the necessary files into the target_directory called hold_out that we created earlier. I am using a split_percentage of 20% for the hold out set. In the code above, once 20% of the images have been held out the remaining are subtracted using tetryonai’s subtract_images function.
If we Refresh our Files pane and inspect our difference_img and hold_out folders we will see the images placed as inspected. You can also open a few of the images inside difference_img to confirm the subtraction process worked (as was shown in Figure 2):
We are now ready to extract the highlighted defects that appear in the differenced images. We will use tetryonai’s extract_contours_from_image function to read in each differenced image, extract defects per image, and save the defects to the extracted_defects folder we created towards the beginning:
Images that are corrupt/non-readable will be skipped by the extract_contours_from_image function. This still leaves us with plenty of images to train our model.
The extract_contours_from_image function uses contours to extract the highlighted parts of our differenced images. Think of contours as a curve joining all the continuous points (along the boundary) that have the same color or intensity. Contours are a useful tool for shape analysis and object detection/recognition. You can read more about contours on OpenCV.
Refresh the Files pane and inspect the extracted_defects folder. You will notice individual folders have been created for each set of contours extracted from each differenced image.
We will now combine all the extracted defects into a single directory called all_extracts. Let’s make that directory now:
directory(**{    "choice" : "make",    "directory_path" : "all_extracts"})
We write some Python to walk through the extracted_defects directory and move any PNG files using tetryonai’s move_files function:
Refreshing the Files pane and inspecting the all_extracts folder will reveal all extracted defects.
In PART 2 we will be applying labels to our extracted defects (“turking”). We thus need to download all extracted defect images to use with the labelling application we create next. So let’s zip all extracted defects for easy downloading:
!zip -r /content/extracted_defects.zip /content/all_extracts/
If you Refresh your Files pane you will see a new extracted_defects.zip file created.
Our labelling application will also require a JSON file with the filenames of the images in our zip folder. Let’s write some Python to grab all filenames in the all_extracts directory:
Save the JSON file we just created:
with open('extracted_defects.json', 'w') as outfile:    json.dump(file_names_and_sizes, outfile)
Refresh the File pane and download both the extracted_defects.zip and extracted_defects.json files:
Save these files to your Desktop for now.
In PART I we differenced a large set of test and template PCB images, extracted defects in the differenced images using contours, and created the 2 files we need to use the labelling application we create next. Let’s do that now.
With our set of extracted defect images in hand we have a good-sized dataset to use for training a Deep Learning model for image classification. But like any supervised machine learning approach our training data requires labels. Since we extracted our images we obviously don’t have these labels available.
Image classification requires large amounts of labelled data, making hand-crafted labeling an arduous task. But the colossal data prerequisite to image classification often isn’t a show stopper. This is because there are a wealth of publicly-available models that have already been trained on large benchmark sets of images. As long as the domain isn’t too different we can use much less data to train a pre-trained model. Can we get away with successfully classifying our extracted defects using much less labelled data, paired with an existing image classifier?
My bet is yes, but there is only one way to find out. This is a good opportunity to build a new, general-purpose product. This product will allow us to create labels for a set of images.
TI won’t go through the steps for creating the Defect Turker since this article is already long enough. It’s a simple application that allows us to apply labels to any images that are inside its img folder.
Too often people rush to find existing solutions, with products that don’t address their true requirements. A lot of time (and money) ends up being wasted trying to force existing products to their specific set of challenges. Bespoke software development is immensely powerful, but only when it’s not inhibited by over-staffed projects and over-engineered solutions. Rapid prototyping is a massively beneficial skill.
We won’t do our labelling on Colab since it currently has no obvious way to host web applications inside its environment. We will thus run the Defect Turker locally.
The Defect Turker is available on GitHub. Clone the application onto your local machine by running the following in a terminal session:
git clone https://github.com/sean-mcclure/defect_turker.git
I also start a simple web service using the following line:
python3 -m http.server
Before opening our browser to the application let’s add the extracted defect images (extracted_defects.zip) and JSON file (extracted_defects.json) we created earlier to the application. These files are on my Desktop, so I’ll double-click the ZIP file to decompress it contents:
..and run the following line in a new tab in terminal, from inside the defect_turker directory:
mv ../content/all_extracts/* ../defect_turker/img/
Confirm our extracted defect images are inside the application’s img folder:
We also need the JSON file containing all image names. Move these now:
mv ../extracted_defects.json ../defect_turker/img/
We’re now ready to label our images. Let’s start the application. Since our web server is already up-and-running we simply point our browser to http://localhost:8000/.
If you need to change the port # on the web server you simply add the port to the end of the full command (e.g. python3 -m http.server 5555)
You should see the following:
Your first image will likely be something other than the one shown here. We can see there are 1333 total images. That’s a lot of labelling. I decided to break this up throughout the day and do a few hundred at a time. Ideally we would split this task among a few people on the team.
Product Idea: Create a version of this application that allows labelling to be done by multiple people, then recombines the results into a single CSV file.
The first icon is a legend showing the labels for each defect type. Refer to this to learn which defect has what label.
It turns out our image differencing only extracted 3 types of defects:
spur
spurious copper
short
The reason only 3 types were extracted has to do with how our image differencing was set up. We can easily detect the other types of defects as well, but I will keep our application to these 3 defect types for this article. See NOTES at the end for details.
Now we can start labelling:
Simply click on the button that corresponds to the label for the presented image. Once we have labelled all the images (or as many as you wish) we click on the upper right icon to download a CSV file that can be used to train our Deep Learning model.
The CSV file has each image name and the label we applied. Save the file as image_labels.csv and keep it on your Desktop for Part 3.
We’re now ready to train our Deep Learning PCB Classifier.
We move back to the Colab notebook to train our Deep Learning model using the CSV file we created with the Defect Turker. We will use fastai to train our Deep Learning model.
fast.ai, the research institute, is dedicated to making deep learning more accessible, via their courses, software, research and community. Their library fastai is a powerful open source API sitting on top of PyTorch. Recent research breakthroughs in Deep Learning are embedded into the fastai library meaning we get access to cutting-edge Deep Learning models that are more accurate and faster than many other DL libraries; all while writing much less code.
!pip install http://download.pytorch.org/whl/cpu/torch-1.0.0-cp36-cp36m-linux_x86_64.whl!pip install fastai
and import:
from fastai.vision import *
We will upload the image_labels.csv file created with the Defect Turker in Part 2. Run the following line in Colab to upload the file:
from google.colab import filesfiles.upload()
You may have to run this twice as the first run often fails.
We’ll now convert image_labels.csv to a data frame using tetryonai’s csv_to_dataframe function:
image_labels_frame = csv_to_dataframe(**{    "path_to_csv" : "image_labels.csv"})
Let’s check the class balance between our labels. First, we’ll get the number of examples of each label using tetryonai’s get_feature_counts function:
labelled_counts = get_feature_counts(**{    "data_frame" : image_labels_frame,    "feature" : "label"})
...then use Matplotlib to create our plot:
...which gives:
Note: If “label” appears in the chart, run del labelled_counts[‘label’] in the cell prior to creating the visual.
We have a “majority class” of NOT, while the 3 defect types are fairly balanced. We could downsample the majority class but it may not be needed. We’ll go ahead and use our labelled data as is, and readjust the balance in our training data only if needed.
The fastai training approach we will use requires individual labelled CSV files for each class we want to predict. Let’s create a folder called training_csvs to hold these individual files:
directory(**{    "choice" : "make",    "directory_path" : "training_csvs"})
...now let’s create individual CSV files for each class using the image_labels_frame dataframe we created from our uploaded image_labels.csv file.
First we create individual lists for each class:
...then we convert those lists into data frames for each class:
Now let’s create individual folders for the CSV files:
Now we can move the individual CSV files to their appropriate folders:
We obviously need images to train our image classifier. These will be the defect images we extracted in PART 1, zipped into the file called extracted_defects.zip, and added to our Defect Turker for labelling.
We could just use the extracted_defects.zip file that is currently in our Colab environment, but we may have restarted and rerun our pipeline from PART 1 if we are doing this project over a few days. This will obviously recreate our zipped image file making the filenames in our previously labelled CSV file (image_labels.csv) mismatch our zipped images filenames.
Given the above, let’s remove the extracted_defects.zip file currently in Colab and upload the extracted_defects.zip file we know we used with the Defect Turker.
Remove zipped images from Colab:
!rm extracted_defects.zip
Upload used for labelling:
files.upload()
Move extracted_defects.zip into training_csvs folder:
move_files(**{    "file_paths" : ["extracted_defects.zip"],    "target_directory" : "training_csvs"})
Extract all images from extracted_defects.zip into the training_csvs folder:
import zipfilewith zipfile.ZipFile("training_csvs/extracted_defects.zip", 'r') as zip_ref:    zip_ref.extractall("training_csvs/")
Now we can move all extracted defect images into their proper training folder, using the image_labels.csv file:
To clean up, let’s remove any remaining PNG files (defect images) that were not labelled:
directory(**{    "choice" : "remove",    "directory_path" : "training_csvs/content",    "force" : True})
We should inspect the image data to confirm they were read-in properly:
data.show_batch(rows=4, figsize=(7, 8))
We can see the labels match the images (the spur, spurious, short, and not_defect images all match the labels shown).
We are now ready to train our classifier. We need a fairly deep architecture if we are to get good enough results to create a product people use. Deep networks (large number of stacked layers) allow for deeper representations than their shallow counterparts and have been shown to perform very well on image recognition tasks.
We will download a pre-trained PyTorch model from torchvision. We want to use a pre-trained model to cut down on training time and CPU requirements. Pre-trained models have been trained on large benchmark datasets to solve problems similar to ours. This is called transfer learning.
The following figure shows the ResNet34 architecture we will use. This architecture is based off research discussed in this paper.
To download the pre-trained ResNet34 model from fastai we run the following:
from fastai.metrics import error_rate # 1 - accuracylearn = create_cnn(data, models.resnet34, metrics=error_rate)
Let’s train our model on our defect data.
By using a pre-trained model it means all the weights are already trained prior to our implementation. We refer to this kind of setup as a frozen model, which be default has “all” layers frozen (except the very last layers). Freezing prevents well-trained weights from being modified, which is what we want since are looking to implement transfer learning.
We are using the default values across a large number of hyper-parameters. It makes sense to see how good our model can perform with the least amount of work possible.
We need to make sure our notebook is utilizing GPU hardware acceleration. We did this at the beginning when we set the hardware accelerator to GPU. If you restarted your notebook since the beginning of this article ensure this is still set.
We access GPU hardware via Nvidia’s CUDA API. This enables us to tap into the parallel computing needed to train our model is reasonable time.
fastai’s fit_one_cycle method uses Leslie Smith’s 1cycle policy, which gives us a faster way to train complex models. You can learn more about this policy Sylvain Gugger’s post here.
We will train our model for 10 epochs. An “epoch” is the number of times our learning algorithm will work through the entire training dataset (each example in the training data had an opportunity to update the model parameters).
Epoch is different than the batch size, which is the number of samples processed before the model is updated.
We begin by training only the last layers of our model, by calling fastai’s fit_one_cycle method, specifying 10 epochs:
defaults.device = torch.device('cuda')learn.fit_one_cycle(10)
Below shows the first 3 epochs during training:
Total training will take around 40 mins. Keep in mind this training would be performed offline, thus training time isn’t that important for our product (whereas inference time is).
Your final result should look similar to this:
We have fairly low error rates, which is good. Not bad for a frozen model using default hyper-parameters. But let’s see if we can improve the results.
Now we will unfreeze the network, which means all the weights in our model can now be updated. We can call fastai’s unfreeze() method in our learn object:
learn.unfreeze()
Since we have “thawed” our model it no longer makes sense to use the same learning rate across all layers. Different layers in deep models benefit from so-called “differential” learning rates, where various rates are used depending on where in the network they are applied. To use differential learning rates we first need to find a suitable range.
To find a suitable range of learning rates we will plot our model’s loss against increasing rates. We can do this by first running:
learn.lr_find()
We are attempting to observe and understand clues that sometimes appear earl-on in training. By doing so, we can potentially tune our architecture via its hyper-parameters in such a way that less epochs are required. This enables us to avoid running complete grid and random searches in our efforts to find good hyper-parameters.
We can plot the learning rates against loss by running the following:
learn.recorder.plot()
The typical approach is to take the values right below the lowest point, prior the error becoming worse. We will use the range between 3 x 10^–5 and 3 x 10^e-4.
We will now run fit_one_cycle again, this using the max_lr argument to specify the range of learning rates. Let’s train our thawed model across 4 epochs using differential learning rates:
learn.fit_one_cycle(4, max_lr=slice(3e-5, 3e-4))
Note we achieved much lower error. Of course this is near-meaningless without first examining the quality of predictions.
As usual we examine the confusion matrix (CM) to see if there are any red flags with respect to misclassification. There might be certain defect types that our model struggles with.
We see most values sitting across the characteristic diagonal of the CM. Looks like our model is quite effective at predicting defects. We will use this model for our product.
Now that we have a trained model with reasonable accuracy we want to download it as an object that can be used for inference (predictions on real data):
learn.save('defect_classifier')learn.load('defect_classifier')learn.export('defect_classifier.pkl')
The .pkl file should appear in the training_csvs folder. Download this:
It’s about 84MB in size.
Recall in PART 1 we created the extracted_defects.json file and uploaded this to the Defect Turker for labelling. We will now create a JSON file with filenames for the actual application we build Part 4:
...and save this file test_temps.json:
with open('test_temps.json', 'w') as outfile:    json.dump(test_temps, outfile)
and download the file:
Finally, let’s zip and download the hold out images to use with PCB Defect Classifier:
.., and download from Colab as usual:
We will use these files in PART 4.
We’re now ready to build our PCB Defect Classifier application.
We’ve arrived at the ultimate goal of this project (and of any real-world project), which is to create a real application for end users. As I stated in the introduction, creating products around our machine learning is the only true test our work has utility beyond a group of engineers.
I also stated that I will not cover all pipeline and front-end coding in order to keep this article a reasonable size.
TThis is the main application we want to create in this article. We looked at the reasons why a product like this would prove useful to end users in the printed circuit board domain. Let’s go over the major parts needed to craft this product, building out both the back-end and front-end components.
User fetches next set of PCB images (test and template);
User inspects PCB images;
User runs differencing/extraction/prediction pipeline;
User inspects predictions;
User reviews tallied costs.
The user can also set the defect costs based on their organization’s manufacturing.
We will use Flask to create a lightweight web service. Between Flask and Azle it will be easy to call our Python functions in the back-end. There are many tutorials on how to create a Flask web service so I won’t detail those steps here. Again, all code is available on GitHub.
We will use the major pipeline functions used in our Colab environment to create the back-end of our product pipeline.
From the figures above we know we need:
image differencing
image extraction
defect prediction
We will use Azle to call the functions (via app.py) contained in classifier.py, image_differencing.py and utility.py.
Here are the major functions needed:
Image Differencing:
subtract_images(**{    "image_path_1": image_path_1,    "image_path_2": image_path_2,    "write_path": "diff_img/diff_" + id + ".png"})
Defect Extraction:
extract_contours_from_image(**{    "image_path": path_to_diff,    "write_path": "contours/" + id + "/",    "hsv_lower": [0, 150, 50],    "hsv_upper": [10, 255, 255]})
Defect Prediction:
def classifier(image_path):    learn = load_learner('model')    img = open_image(image_path)    prediction = learn.predict(img)    return(str(prediction))def predict(image_path):    prediction = classifier(image_path)    return(prediction)
The first 2 are the same tetryonai functions we used in our Colab environment for processing the test and template images.
I will create a directory called api that holds the main Flask file (app.py) and the files pertaining to our pipeline:
api└── app.py└── classifier.py└── image_differencing.py└── utility.py
The utility functions inside utility.py just help with some basic operations needed for the operation of the application.
Creating the application in Azle enables rapid prototyping. Here the major pieces:
Add buttons:
az.add_button("target_class", target_instance, {    "this_class": "my_button",    "text": "FETCH BOARDS"})
Show images:
az.add_image("target_class", target_instance, {    "this_class": "my_image",    "image_path": "https://bit.ly/1L574po"})
Call APIs
Sliders
Clone the application from GitHub by running the following in terminal:
git clone https://github.com/sean-mcclure/defect_classifier.git
We need to add the following files:
new PCB board images not used during model training;
JSON file containing all new board names;
Trained PCB defect classifier model.
Recall in PART 3 we created and downloaded the files test_temps.json and hold_out.zip. Place these files on your Desktop if they’re not already and unzip the hold_out.zip folder.
Now we can add all the hold out images to our application by moving them into the test_temps folder. From inside the cloned application (assuming application was cloned to Desktop):
mv ../Desktop/hold_out/* test_temps/
...and the test_temps.json file:
mv ../Desktop/test_temps.json test_temps/
Now we have:
test_temps    └── 000410009_temp.jpg    └── 000410009_test.jpg    └── 00041012_temp.jpg    └── 00041012_test.jpg    ...    └── test_temps.json
Our application now has access to the kind of images we would expect in our PCB board manufacturing scenario.
In PART 3 we saved the .pkl file. This is our trained defect classifier model. Let’s add this to the model folder of the application:
mv ../Desktop/defect_classifier.pkl model/
We need to rename our model object since fastai’s load_learner method expects the object to be called “export.pkl”:
mv model/defect_classifier.pkl model/export.pkl
Let’s go ahead and see how our PCB Defect Classifier works!
We built the web service in Flask. We call this as usual:
python api/app.py
You should see the following in your terminal:
* Serving Flask app "app" (lazy loading)* Environment: production   WARNING: This is a development server. * Debug mode: on * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) * Restarting with stat * Debugger is active! * Debugger PIN: 622-284-701
We can see that we are running in debugger mode. This is ideal since any change we make to the Python will cause the service to automatically restart. This is very beneficial for prototyping.
Now we need to start a web service to show the front-end of the application. In your terminal run the following:
python3 -m http.server
Then open your browser to:
http://localhost:8000/app/
Our first screen looks like this:
The first step for the user is to run FETCH BOARDS, which will bring in our PCB images stored in test_temps (our hold out images).
This fetches both the TEST and TEMPLATE images. These are the PCB boards that manufactures need to test to see if any defects exist.
Now the user runs the pipeline, which will:
subtract the images to create a differenced image;
extract the defects contained in the differenced image;
predict the defect class for each extracted defect;
add the predicted cost to the cost summary.
Here’s the fuller video again we saw at the beginning, showing how we can make multiple runs against our pipeline, getting accurate predictions each time:
Beautiful. We can see that differencing and defect extraction is almost instantaneous. The prediction is set to 1000ms for each extracted defects, which seems reasonable for our demonstration. Is this fast enough for a PCB manufacturer?
FFeature Idea: Users could be presented a slider to set the prediction time. Based on manufacturing throughput, setting a lower prediction time might make sense. Of course this has to be balanced with how fast the prediction can be made with our pre-trained model (inference time). Setting up a chart showing best inference times would be a good experiment to run and show clients.
There are a number of things we could do to improve our product. While model accuracy is already quite good we didn’t try the other defect types. Will model performance be as good on these defects? What if this product needs to predict at a much higher rate? Would it be worth using shallower networks that run faster inference, while sacrificing some of our hard-won accuracy?
Here are some things we could try:
Data Augmentation
Other hyper-parameter settings
More data
Different architectures
There are also various user experiences we might build for, by way of different/additional product features. Would it make sense for users to have an option to mark bad predictions as incorrect (feed this info back to model). What about tallying additional information about the predicted defects, other than cost? Perhaps creating interesting summarizing visualizations. By embracing rapid prototyping we don’t have to know the answers to these questions upfront; we can build, showcase, and change as needed, with real products.
In this article we created an end-to-end PCB Defect Classifier application. We used image processing to difference sets of test and template PCB images, extract defects using contours, and classify the extracted defects using Deep Learning. We then created a front-end application that allows users to run our machine learning pipeline and review model predictions.
Applications like this help automate tasks that are manual, time-consuming, and prone to human error. The purpose is not to replace human experts, but rather augment their efforts so they may focus on more strategic and creative aspects of their jobs.
If you have any questions / need support please use the comments section. I can help others modify these applications for their specific purpose. Azle questions can be asked on Stack Overflow.
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
You’ll notice that we only get 3 of the defect types. This is because of the way the differencing works. It only picks up the black-differenced defects. The other defects, like mousebites, etc. need to be white-differenced. This can be achieved by simply changing the order of the image paths in the differencing function. Thus, if you want to detect, extract and predict all defects run both versions.If you want to remove the folders in contours run rm -r contours/* in terminal. If you want to remove all images in diff_img run rm diff_img/*.
You’ll notice that we only get 3 of the defect types. This is because of the way the differencing works. It only picks up the black-differenced defects. The other defects, like mousebites, etc. need to be white-differenced. This can be achieved by simply changing the order of the image paths in the differencing function. Thus, if you want to detect, extract and predict all defects run both versions.
If you want to remove the folders in contours run rm -r contours/* in terminal. If you want to remove all images in diff_img run rm diff_img/*.
World PCB Production in 2014 Estimated at $60.2BA PCB Dataset for Defects Detection and Classificationtangsanli5201/DeepPCB on GitHub
World PCB Production in 2014 Estimated at $60.2B
A PCB Dataset for Defects Detection and Classification
tangsanli5201/DeepPCB on GitHub