6 Reasons to Spend More Time Thinking About Labels
By: Jean Voigt
Tags: Feature Engineering, Data Science, Data Cleaning, Machine Learning, Mlops

6 Reasons to Spend More Time Thinking About Labels

by Chikilino via pixabay

Imagine you are in a forest and can only get home by looking at markers labeled “may be closer to home”. You probably would take a while to get out of the forest.

That is, in essence how an artificial intelligence project may start out. Often labels are inaccurate, incomplete, unreliable, unsteady or insufficient in number. Even if you are far from the only one walking in the woods!

In the initial example one could decide not to use supervised learning at all and only consider reinforcement or semi-supervised learning. Alternatively, incomplete records could be removed. But what if removed records actually are the best labels, concern most of the records, or worse, someone requests that this one particular point in the forest must be noticed by the model, or it isn’t useful at all?

Oftentimes, when people start with a new problem it is about trying to get results quickly. This is only natural and, as I have written before, often a function of the organizational setup. In this article I want to outline a few experiences as a practitioner walking down the path between data lakes, decision trees and transformation pipelines. My observations are as a humble spectator. They are done in a way that is a rather casual approach and by no means hard fact. It is the start of a conversation. So, if you disagree or spot errors, please let me know.

Before diving into the details, below is a quick summary of the six reasons labels should be given more careful consideration:

Engineered and collected labels: Engineered labels are created because for some reasons the collected labels are poor or don’t exist, using techniques similar to those employed for feature engineering. The collected labels are direct observations from primary data collection. Compare Cassi’s data provenance types.

Noisy and clean labels: While the data is accurately collected, labels are not. Noisy labels show a disproportionate amount of imbalance. Perhaps even to the degree that they reduce the training set to something that cannot reasonably be expected to build a model for actual use.

Many and few labels: The usual complaint is to not have sufficient labels. However, if the few labels that are available are all really good, it’s much less of a problem. However, having many labels with a number of bad ones is a problem. In such a situation, the first order of business is the daunting task to find out which labels are actually good.

Initial and updated labels: Most models have some kind of initial train/test data. In the event of engineered labels, one actually may decide, for good reason, not to trust these labels quite yet. It is best to collect updated labels. In essence, your labels will be learned through some kind of feedback loop.

Definitive and approximate labels: Approximate labels are a difficult type of label. These are different from engineered labels, in that it is not clear whether they actually are describing a positive or negative outcome.

Consistent and inconsistent labels: One of the key points addressed by Andrew Ng in his MLOps discussion has been on the labels that have been marked subtly different in two distinct datasets. If one is not careful, training before correction may have unintended consequences.

Quite a few of the issues should be addressed as part of an established machine learning operations. Some issues may be resolved through support functions such as legal, people, general data management and smart procedure design — more on that at a later post. For now, let’s focus on the all important labels, as opposed to the features.

Engineered and collected labels

When creating new labels from partial features, who decides how and what is a good label? (image by author)

Collecting primary data is time-consuming and very expensive, so people often engage in, to borrow a term from the feature space, “feature engineering”. While the collected data can be problematic in its own way, such as having lots of noise or a huge quantity of data records, usually the quality is excellent.

Engineered labels, however, present many more challenges. For label engineering beyond established semi-supervised techniques, deciding how to select and which labels to use is only the beginning of a very long journey. How the selection should be influenced by the feature quality of that record is one aspect. Possible imbalances formed by removing or selecting labels present another challenge. That is assuming the prevention of data-leakage from features into the labels can be assured. Finally, all those rules should be well-maintained and consistently applied. This may be feasible for a small individual project. However, for larger industrial scale label engineering this may well lead to automatic labelling or broader adoption of emerging self-supervised learning techniques.

Noisy and clean labels

Where would you have a better chance to listen to the violin player? (image by author)

Clearly, label errors come in different degrees. In the recent public case of ImageNet and others, it was an error spotted almost by chance. Some errors never get observed and the model just performs subpar. But errors are not the only noise in the label data. Noise comes in many forms and by the time model evaluation metrics are looked at it’s pretty hard to tell a label error from a model issue. So apart from going back and collecting primary data, what can realistically been done to spot this?

First, it would be good to detect that there is a noisy label problem. One approach would be, assuming known clean and known noisy labels, to run the model against partitions of that data, record outcome variance and compare that against partitions of your target labels — running a model against corner-case data is a practice in finance deserving wider consideration. Of course, a sample of the labels can be manually rated, with the advantage of also correcting the labels in one go. That is expensive and may not even be required. The less thought of option is to change your model to something less sensitive to errors. A perfectly labeled dataset may not be required for every use case or there may be a semi-supervised solutions to the problem. Finally, perhaps a smaller dataset would do equally as good or even better?

Only when other options have been exhausted and it has been established that there are bad labels preventing resolution of the problem at hand, cleaning comes into play. To correct automatically or, especially for smaller data sets, clean labels manually are promising options.

Many and few labels

It can actually be a blessing to have fewer labels, especially in the event of low volume high quality data. This is precisely what Andrew Ng recently spoke about. Small, well curated data may be much more valuable, especially because label issues can be fixed relatively inexpensively. Perhaps the label problem is only in a subset of the data. Thinking about labels and data in terms of smaller, more easily managed collections could help solve issues.

Similar to the few label problem the situation may not be so dire with a second look, unless the labels show signs of noise, inconsistency or accuracy issues. Even in the event of these problems, a large label population provides options that are not afforded by datasets with fewer labels. Specifically, the actual label ground truth can be learned if the label population is large enough.

Initial and updated labels

Labels used for initial training, assuming they don’t have any problems, are usually fewer and less specific than those collected over time after the model has been deployed. If the data collection and management does not consider the effects of new labels through the feedback loop, these label updates can have unintended consequences and even introduce bias.

However, since the generation of initial labels is costly and highly constrained, labels from the production deployment are highly beneficial. Considering the data that should flow back early in the model design does not sound like a bad idea. Including some procedure to validate the retrieved label updates and how the model performance changes given these updates might be worthwhile as well.

Definitive and approximate labels

How to label a case where the defendant is found guilty but the appeal is still pending? (image by author)

A label that clearly conveys that a record belongs to one or the other class is great. In many instances however, that information is not available yet or won’t ever be. It may not be possible to wait for the label to become accurate for any practical application. The results are approximate labels, generated by systems or processes that you can only trust to a certain degree.

There are two basic choices: assume approximate labels are accurate and correct any uncertainty with labels collected during production deployment, or incorporate the uncertainty into the model. The latter approach provides some relief even in the case where production models would never produce accurate labels — which is the case for some legal and regulatory use cases. The more important observation in my experience, is to recognize the limitation of the system in the event of approximate labels. If you cannot observe something, you just cannot learn well from it. This truth holds for humans and it does apply to machines. I reckon that for many approximate labels situation, supervised machine learning may just not be the best tool — semi-supervised, specifically soft-labels might work better.

Consistent and inconsistent labels

Clearly, consistent labels are preferred. In practice this is rarely the case. Inconsistency gets introduced by combining two datasets, because the system generating the data had been updated or because the process generating the data had been changed over time. Someone may have changed the labeling procedures over time and nobody thought about re-labeling all the old data according to the new procedures. The source of inconsistency is so large that there are many techniques (some referenced above) to correct for this issue if a large number of labels are available.

Well, the trouble starts when labels are both inconsistent and low in number. I am not familiar with any technique that helps — if you do know any I would love to learn about them.

The next best thing is to build a rules based system and learn from the production labels generated by the system, essentially a reinforcement approach. For that to work, it is crucial for the organization to be prepared to stick through the learning cycle of the model. The rule based system can become quite good. As a result, the organization may no longer be prepared to switch back to the initially less accurate AI model once good label data collection has completed.

Now what?

It seems as the machine learning field is advancing tools such as MLFlow will help with managing the model and data versions, including the label data, more effectively. Hence allowing to manage each individual label issue more directly. Further, the advances in automatic labelling and self-supervised learning are promising for industrial applications, but the necessary data collection may not be feasible in all situations.

I hope to have illustrated a few interesting ideas for a better chance to find a path out of the forest. I would welcome any further ideas on the matter.

Further reading