Building a Useful Dataset for Spam Detection
By: Data
Tags: Ecommerce, Machine Learning, Classification, NLP, Data Science

The diagram above demonstrates the concept of sample value. In the leftmost diagram, we are shown the total population of samples and their known label, denoted by the shape and colour of the marker. This is representative of our classification problem. In the middle diagram, we see the scenario where the greyed-out markers are unlabelled samples and the shaped/coloured in samples have been selected and labelled using PL. We can see that the decision boundary discovered using PL is sub-optimal as it would misclassify a significant proportion of the green/square samples. Many of the samples that have been labelled are located in uninformative points in the feature space. In contrast, the rightmost diagram shows samples that have been selected using AL. Here, we see that the labelled samples are more densely concentrated around the point where the two classes meet in the feature space. This allows our model to find a more optimal decision boundary.

Even though AL can help reduce the impact of class imbalances in our data, the skew in our distribution was so large that we decided to implement a sampling strategy, based on domain knowledge, to create our initial pool of unlabelled data. This strategy was successful in creating a better representation of both classes.

Having created a large pool of unlabelled samples to select from, we now required a querying strategy to assign a value score to each unlabelled sample. This will enable us to discriminate samples by value when selecting unlabelled samples to be labelled as part of the next iteration. As this is a Binary Classification task, we used the Least Confidence (LC) querying strategy. LC entailed training a model on the set of labelled data, then applying the model over the pool of unlabelled data and assigning each unlabelled sample a value based on the probability of the highest likelihood class. In LC, the samples are ranked by 1 minus the likelihood, so the lower the likelihood the higher the ranking.

For Example:

In the diagram to the left, we can see that sample C is ranked highest because its score is 1–0.6=0.4, which is higher than sample A (1–0.8=0.2) and B (1–0.9=0.1).

Our Active Learning Process

Putting this all together, we can now define our dataset creation process as the following:

Step 1: Randomly select a subset of samples from the pool of unlabelled samples and manually label them. These will be our Test Set.

Step 2: Again, randomly select a small subset of samples from the pool of unlabelled samples and manually label them. This will be the seed to the Labelled Set.

Step 3: Train the model on the Labelled Set and generate class probabilities for the pool of remaining unlabelled samples.

Step 4: Apply our querying strategy, LC, to these probabilities to obtain a ranking of samples. We take the top N samples from this ranking and manually label them. These are then added to the Labelled Set.

Step 5: Repeat steps 3 & 4 until a stopping criteria has been met. In our scenario, the stopping criteria was to produce a fixed number of labelled data samples, unless the task exceeded a predefined deadline.

Labelling Process

The labelling process itself also required some consideration as there were a couple of potential pitfalls. One problem is that the process is vulnerable to human error as a person may mislabel a sample. This was of particular concern in our situation because of the nature of AL.

In AL, samples that are highly ranked for labelling also tend to be ambiguous as they lie close to the decision boundary. This means that the descriptions may be difficult for a human to judge as well. To manage this, we implemented a majority vote system where each sample required 3 votes by 3 separate individuals before it was considered labelled. We also provided the labeller with supplementary information about the product to aid their decision. Each sample consisted of the description text and also an image of the product for context.