Classify images with CNN. Getting young children to tidy up their... | by 1Regina | Towards Data Science

Getting young children to tidy up their rooms is often challenging. What I insist is messy, they will insist is clean enough. After all, all adjectives are subjective and I want my children to grow up respecting others’ opinion in our inclusive society. How do you put some definition around differences in opinion? An objective way to achieve this distinction is using image classification to differentiate between a clean versus a messy room.
This application can also be extended to cleaning services agencies or Airbnb especially so with more than 10% of its user complaints pertaining to dirty and messy conditions. (see source)
The process to building my application comprises of 5 stages: 1) Web-scraping 2) Dataset preparation 3)Models Building 4) Model Testing and 5)Front end Flask App.
1. Web-Scraping
There was no dataset of the size I want, so I had to turn to web-scraping (such is life!) with keywords on google_images_download to obtain images from Google.
For the “Clean” category, words like “clean bedroom”, “tidy room”, “ hotel room”, “clean and neat” and “declutter” were used.
For the “Messy” category, words used include “clutter”, “mess” and “disorganized room”.
I created a third category for fun to cater to the snapshots of excuse notes my kids will leave me to escape their chores. (Yup, inertia actually begins at a tender age.) For this category/class, I used words like “kids’ handwritten note”, “love letters”, “handwritten notes” etc to scrape google images.
2. Dataset Preparation
After obtaining more than 3,000 images, it is important to ensure the images are correct. Going into each category, I eye-balled the pictures, removing i) conflicting pictures, ii) contrasting pictures showing before-and-after, iii) duplicates which won’t help in training, iv) cleaning agents pictures, v) cartoon images, vi) pictures of the words, vii) company logos.
This dataset spring-cleaning is essential. Remember the principle: garbage in, garbage out.
At the end of the day, I was left with more than 600 pictures for ”Messy” and “Excuse” and more than 900 for “Clean”.
3. Models Building
Starting with machine-learning, I used 8 different algorithms.
K-Nearest NeighborsGaussian Naive BayesLogistic RegressionSupport Vector ClassifierDecision TreeMulti-Layer PerceptronRandom ForestXGBoost
K-Nearest Neighbors
Gaussian Naive Bayes
Logistic Regression
Support Vector Classifier
Decision Tree
Multi-Layer Perceptron
Random Forest
XGBoost
Using the F1 score as the evaluation metric for all the models, XGBoost is the best performer at 0.76. (For those with an appetite for technicals, I will elaborate them in italics in this article. You can skip them if your interest is only on the outcome. F1 is the harmonic balance between precision and recall. i) Precision is the ratio of True Positives out of all the predicted positives (comprising of True Positives and False Positives) whereas ii) Recall is the ratio of True Positives out of True Positives and False Negatives.
Curious to know if this can be improved in a substantial manner, let’s explore Deep Learning using Convolutional Neural Network (CNN) which is reputable for working with images.
The CNN architecture comprises of many layers. Each layer will extract certain features (e.g contrast, shapes, edges, texture) from the training pictures for each class . The trained model is then subsequently applied to unseen pictures which are then classified using the trained feature elements.
Indeed, the CNN model with the Adam optimizer and early stopping is the best model, beating the XGBoost with a new 0.84 F1 score associated with the trained network. The superiority of the Adam optimizer lies in its adaptive learning rate and favoured due to its relatively less parameters tuning. Below are the codes for my model and classification report for the associated F1 score.
# compile & train model# initialize # of epochs to train for, and batch sizeEPOCHS = 30BS = 128# initialize the model and optimizer model.compile(loss=”categorical_crossentropy”, optimizer=’adam’,metrics=[“accuracy”])# train the networkimport kerasH = model.fit_generator(aug.flow(trainX, trainY, batch_size=BS), validation_data=(testX, testY), steps_per_epoch=len(trainX) // BS, epochs=EPOCHS,# inject callbacks with best weights restorationcallbacks=[ keras.callbacks.EarlyStopping(patience=8, verbose=1, restore_best_weights=True), keras.callbacks.ReduceLROnPlateau(factor=.5, patience=3, verbose=1), ])
“restore_best_weight” ensures that the model will retrieve the best weights achieved through the epoch runs.
4. Model Testing
Putting the model to the test with 3 unseen pictures, one from each class, here are the results:
Taking a step further, even if my creative kids were to someday write me a note on a decorated piece of paper or couple a “gift” with their notes, the model can still confidently classify these as excuses.
5. Front End Application
Image classification is a subset of image recognition which has widespread use in the security industry (facial recognition), virtual search engine (object finder in stores), healthcare (emotion detection in patients) and gaming and augmented reality. In a way, smartphone cameras have made all these advances possible with multitudes of pictures that can be easily created. This project demonstrates an application of image classification in our homes :)