Fast cloth color identification from customer images in pure Go
Tags: Machine Learning, Leboncoin, Algorithms, Go, Golang

The goal was to detect the primary color of an ad by extracting it from the images. This is a known problem, but they were not any solution out of the box that met the following criteria:

Fast — a few milliseconds — since it was going to be applied to ~2 million ads, and it makes it easier to deploy.

Easily translatable to Go, since this is the core language at leboncoin and it had to be integrated into the submission process.

Accurate, since consumers expect good results when they use a filter on leboncoin.

All available solutions either took a few seconds to run or heavily relied on big scientific libraries like sci-kit-learn which is far too heavy for this task. A homemade algorithm had to be created.

Choosing the colors

Color filter on the leboncoin website

Colors were determined from the filter, by merging certain colors (like “cream” and “white”) and not trying to tag non-colors like “multi-color” or “prints”.

In the end, the list is composed of 12 distinct classes of colors.

Gray isn’t a part of the list, because customers don’t agree at all on what is gray, making it impossible to check if a model is performing well or not.

(Adding gray to the list resulted in a ~95% error rate and added confusion to the other classes)

8 random samples classified as “gray” by customers, notice the diversity.

Getting the data

To train and evaluate our model, examples were needed from all classes in a balanced way. Luckily leboncoin has a very big dataset of ~3 million ads in the clothing category with images associated, about half of which has a color in the description.

First, 1000 ads of each class were queried where the color appeared uniquely in the title. It can be done directly from the website with a query of the form “green NOT red NOT blue NOT yellow…”, but it was easier to plug directly into Elasticsearch and download the image references.

Then, all the images were downloaded (about 600 Mo) and split in training/test datasets with 90% going for training. No validation dataset was made as the model is not really prone to overfitting because of its simplicity.

48 random samples from the dataset

The model

Preprocessing

Preprocessing is done in 3 steps.

Resizing the image to 256x256px. Cropping the image, by cutting a quarter of each side. Augmenting saturation.

As the cloth is most often in the middle of the image, cropping helps a lot to separate the background from the object of interest. Augmenting saturation also helps by making colors more vivid.

To do the image processing, the Go library “ imaging “ was used which is fast enough and effortless. It does not have any heavy dependency such as opencv or vips.

Extracting the primary color

To extract the primary color of an ad, all the images should be used and their information combined. The assumption is made that all images are of the same object; Albeit this is not always the case, as leboncoin allows packs to be sold as one ad. However, there was no time to handle this correctly.

The first step is to detect in each image the dominant colors. The famous K-Means clustering technique was used, as it is known to converge fast and give good results.

To resume how K-Means works, it starts with k random clusters. At each update, points are attached to their closest cluster. Then the new clusters are the means of the points which are attached to them. This process is iterated until no points change side — convergence.

The clusters here are clusters of pixels, each pixel in the image is treated as a point in (Red, Green, Blue) space.

Here is K-Means run on the 2 images using k=4:

The first figure is the image itself.

The second figure is the pixels plotted in (R, G, B) space.

The third figure is a visualization of which pixel belongs to which cluster after convergence.

The fourth figure is the mapping of pixels to their corresponding cluster.