A Free And Powerful Labelling Tool Every Data Scientist Should Know | by Lai Woen Yon | Towards Data Science

As a data scientist, you will definitely need to train models to meet your organization’s needs. Most of the time, you require labelled data from within your company in order to build a customized solution.
You’re approached by a product manager one day who wants you to build a named entity recognition model to improve the quality of the downstream data science product. Due to the short time frame this product needs to be launched, he hired a bunch of human labellers to assist you. What should you do next?
“ Give us the tools, and we will finish the job “
by Winston Churchill
The task objective is clear, manpower is ample, and a deadline is set. What’s next? A useful labelling tool. Please note the emphasis on “useful”. But what is it that I mean by “useful”? The tool should at least give you the ability to:
Track the labelling quality, on-premises or in the cloud, at any time. The annotation staff can be informed if they are not following the instructions before the entire labelling project is completed.Keep track of the progress of every annotator. The deadline is given to you, so it is imperative that every annotator be given a specific deadline for finishing their labelling. As a project leader, you should be able to track the progress to ensure everyone stays on track.Allow multiple annotations to work together. Bias is one of the most problematic topics in machine learning. As a data scientist, you do not want annotation bias to affect your model after spending several days or weeks training your model, only to find out that it is biased. As a possible solution, multiple authors could collaborate on the same task/article and only approve labels that are unanimously accepted.Post-process your labelling result with the least amount of effort. One particular aspect you will appreciate is the ease with which you can handle the labelled data. In my experience, the JSON format is the easiest to work with when I am working on labelling projects for clients. It would be great if the tool could support multiple types of formats for importing and exporting data.Provide a user-friendly interface to your human annotators. Developing a good data labelling tool for me is based primarily on this factor. Not only was I responsible for preparing the client documentation, but I also had to train the human annotations on how to use the labelling tool with as little human error as possible. You can therefore save yourself from a great deal of hassle if you use an easy-to-use labelling tool.Code your labelling interface to suit your needs. I’m sure, as a programmer, you’d like your tool to be controlled by code. Coding to make the interface customized would allow the time to be dedicated to other important tasks.
Track the labelling quality, on-premises or in the cloud, at any time. The annotation staff can be informed if they are not following the instructions before the entire labelling project is completed.
Keep track of the progress of every annotator. The deadline is given to you, so it is imperative that every annotator be given a specific deadline for finishing their labelling. As a project leader, you should be able to track the progress to ensure everyone stays on track.
Allow multiple annotations to work together. Bias is one of the most problematic topics in machine learning. As a data scientist, you do not want annotation bias to affect your model after spending several days or weeks training your model, only to find out that it is biased. As a possible solution, multiple authors could collaborate on the same task/article and only approve labels that are unanimously accepted.
Post-process your labelling result with the least amount of effort. One particular aspect you will appreciate is the ease with which you can handle the labelled data. In my experience, the JSON format is the easiest to work with when I am working on labelling projects for clients. It would be great if the tool could support multiple types of formats for importing and exporting data.
Provide a user-friendly interface to your human annotators. Developing a good data labelling tool for me is based primarily on this factor. Not only was I responsible for preparing the client documentation, but I also had to train the human annotations on how to use the labelling tool with as little human error as possible. You can therefore save yourself from a great deal of hassle if you use an easy-to-use labelling tool.
Code your labelling interface to suit your needs. I’m sure, as a programmer, you’d like your tool to be controlled by code. Coding to make the interface customized would allow the time to be dedicated to other important tasks.
These are the features I would at least like my labelling tool to have. A tool like this could cost you thousands of dollars and countless hours of labour to develop. What is the best way to find such an off-the-shelf tool ... for free?
Before I begin, let me disclaim the following: I do not work for Label Studio and I am not affiliated with Label Studio. It is simply my personal experience from working on a client’s project.
github.com
As I was developing a system to label data on a client’s behalf, I ran into this tool. In light of the fact that it is an open-source and free data labelling tool, I am quite surprised at its flexibility and functionality. Using a few simple steps, I will show you how you can construct a simple NER labelling interface.
You can easily set up Label Studio by following the instructions in the Github repository. It supports both installations on the local machine and deployment in the cloud. As part of my work for a client, I built and deployed the tool using Docker. It allows him to store the data locally since credentials are the main requirement for his business. To demonstrate the tool, I am going to use Heroku Buttons. You may deploy it in any manner you deem acceptable for your business.
You will be able to see the project page after you log in. The process of creating a project is illustrated in the gif above. For this project, I call it NER.
We will prepare a sample project with only two articles (in Label Studio, they call these "tasks"). You are free add more if you want to.
The sample articles are copied from these two articles:

https://www.reuters.com/world/asia-pacific/japans-mount-aso-erupts-alert-level-raised-2021-10-20/
https://www.reuters.com/technology/bitcoin-sits-below-all-time-high-after-us-etf-debut-2021-10-20/

https://www.reuters.com/world/asia-pacific/japans-mount-aso-erupts-alert-level-raised-2021-10-20/
https://www.reuters.com/technology/bitcoin-sits-below-all-time-high-after-us-etf-debut-2021-10-20/
import requests
from bs4 import BeautifulSoup
import json
from google.colab import files

Using Beautifulsoup to crawl the article body automatically
%time
task_texts = []

task_links = ["https://www.reuters.com/world/asia-pacific/japans-mount-aso-erupts-alert-level-raised-2021-10-20/", 
              "https://www.reuters.com/technology/bitcoin-sits-below-all-time-high-after-us-etf-debut-2021-10-20/"]

for task_link in task_links:
    page = requests.get(task_link)
    if page.status_code == 200:
        soup = BeautifulSoup(page.content, 'html.parser')
        task_texts.append(''.join([p.getText() for p in soup.find_all('p', {'data-testid': lambda value: value and "paragraph" in value})]))
    else:
        print('Website returns non-parsable content')
        raise

CPU times: user 2 μs, sys: 0 ns, total: 2 μs
Wall time: 5.48 μs

Populate the crawled articles into a list format accepted by label studio
%time
label_studio_input = []

for task_text in task_texts:
    label_studio_input.append({"data":{"text":task_text}})

CPU times: user 3 μs, sys: 0 ns, total: 3 μs
Wall time: 7.39 μs

print(json.dumps(label_studio_input[0], indent=4, sort_keys=True, ensure_ascii=False))

{
    "data": {
        "text": "TOKYO, Oct 20 (Reuters) - A volcano erupted in Japan on Wednesday, blasting ash several miles into the sky and prompting officials to warn against the threat of lava flows and falling rocks, but there were no immediate reports of casualties or damage.Mount Aso, a tourist destination on the main southern island of Kyushu, sent plumes of ash 3.5 km (2.2 miles) high when it erupted at about 11:43 a.m. (0243 GMT), the Japan Meteorological Agency said.It raised the alert level for the volcano to 3 on a scale of 5, telling people not to approach, and warned of a risk of  large falling rocks and pyroclastic flows within a radius of about 1 km (0.6 mile) around the mountain's Nakadake crater.The government is checking to determine the status of a number of climbers on the mountain at the time, Chief Cabinet Secretary Hirokazu Matsuno told reporters, but added that there were no reports of casualties.Television networks broadcast images of a dark cloud of ash looming over the volcano that swiftly obscured large swathes of the mountain.Ash falls from the 1,592-metre (5,222-foot) mountain in the prefecture of Kumamoto are expected to shower nearby towns until late afternoon, the weather agency added.Mount Aso had a small eruption in 2019, while Japan's worst volcanic disaster in nearly 90 years killed 63 people on Mount Ontake in September 2014."
    }
}

Save file to temporary folder and automatically download the file
with open("label_studio_input.json",'w') as j:
    json.dump(label_studio_input, j)

files.download("label_studio_input.json")

Our next step is to upload our own data to prepare for labelling. For your convenience, I have created a script to help you get started. As a starting point, I used two articles from Reuters.com. Run the script above in Colab, and you should be able to view a list of tasks in JSON format like this:
[ {    “data”: {         “text”: “TOKYO, Oct 20 (Reuters) - A volcano ...”    } }, {    “data”: {         “text”: “HONG KONG, Oct 20 (Reuters) - Bitcoin ...”    } }]
You can import data by uploading the label_studio_input.json file. After that, you may select from a large variety of templates to customize your labelling interface in the last tab. The default NER template can be used to start off our first NER project. Here are the steps to show you the first project page.
Label Studio is by far the simplest tool I have ever used. This tool’s user interface makes it very easy for me to use. As you can see in the above video, you simply select the label button and highlight the word associated with it. It is possible to activate the label button with the hotkeys if you want to do it more quickly. By pressing the “3” key, you can activate the label button for the “LOC” label. You can confirm your labelling by clicking “Submit” on the right panel after you have finished. Simply edit the text, then update it as needed.
You can see how intuitive the entire labelling process is. What about exporting your result to future processing?
Once you have completed labelling, export it in a different format so you can post-process the result. In this demo, I will export it in JSON format and further clean the result into a Panda Dataframe.
Here is a script that shows you a very simple step to clean the label studio results:
I have stored the label studio result in my google drive.
For demo purpose, I will download it from the google drive and use it to do simple post-processing.
import pandas as pd
import json
from IPython import display

!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1IqO4-bTCa12IW5paSfQsNoTziCXY304e' -O label_studio_output.json

--2021-10-20 11:54:30--  https://docs.google.com/uc?export=download&id=1IqO4-bTCa12IW5paSfQsNoTziCXY304e
Resolving docs.google.com (docs.google.com)... 172.217.15.110, 2607:f8b0:4004:811::200e
Connecting to docs.google.com (docs.google.com)|172.217.15.110|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://doc-0o-bg-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/8ct4776jhmqlnu6o2lt742603fr01lqr/1634730825000/05232633842228891353/*/1IqO4-bTCa12IW5paSfQsNoTziCXY304e?e=download [following]
Warning: wildcards not supported in HTTP.
--2021-10-20 11:54:30--  https://doc-0o-bg-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/8ct4776jhmqlnu6o2lt742603fr01lqr/1634730825000/05232633842228891353/*/1IqO4-bTCa12IW5paSfQsNoTziCXY304e?e=download
Resolving doc-0o-bg-docs.googleusercontent.com (doc-0o-bg-docs.googleusercontent.com)... 142.251.33.193, 2607:f8b0:4004:837::2001
Connecting to doc-0o-bg-docs.googleusercontent.com (doc-0o-bg-docs.googleusercontent.com)|142.251.33.193|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5949 (5.8K) [application/json]
Saving to: ‘label_studio_output.json’

label_studio_output 100%[===================>]   5.81K  --.-KB/s    in 0s      

2021-10-20 11:54:30 (36.0 MB/s) - ‘label_studio_output.json’ saved [5949/5949]


with open("label_studio_output.json",'r') as j:
    raw_result = json.load(j)

print(json.dumps(raw_result[0], indent=4, sort_keys=True))

{
    "annotations": [
        {
            "completed_by": {
                "email": "followalotofdata@gmail.com",
                "first_name": "",
                "id": 1,
                "last_name": ""
            },
            "created_at": "2021-10-20T08:08:52.329065Z",
            "ground_truth": false,
            "id": 1,
            "lead_time": 101.718,
            "parent_annotation": null,
            "parent_prediction": null,
            "prediction": {},
            "result": [
                {
                    "from_name": "label",
                    "id": "2c-MOFaI_1",
                    "origin": "manual",
                    "to_name": "text",
                    "type": "labels",
                    "value": {
                        "end": 5,
                        "labels": [
                            "PER"
                        ],
                        "start": 0,
                        "text": "TOKYO"
                    }
                },
                {
                    "from_name": "label",
                    "id": "CQuF--hNqt",
                    "origin": "manual",
                    "to_name": "text",
                    "type": "labels",
                    "value": {
                        "end": 52,
                        "labels": [
                            "LOC"
                        ],
                        "start": 47,
                        "text": "Japan"
                    }
                },
                {
                    "from_name": "label",
                    "id": "SsOEHB7pb8",
                    "origin": "manual",
                    "to_name": "text",
                    "type": "labels",
                    "value": {
                        "end": 321,
                        "labels": [
                            "LOC"
                        ],
                        "start": 315,
                        "text": "Kyushu"
                    }
                },
                {
                    "from_name": "label",
                    "id": "7eWePA7S4O",
                    "origin": "manual",
                    "to_name": "text",
                    "type": "labels",
                    "value": {
                        "end": 445,
                        "labels": [
                            "ORG"
                        ],
                        "start": 418,
                        "text": "Japan Meteorological Agency"
                    }
                },
                {
                    "from_name": "label",
                    "id": "hLIKJlIXzN",
                    "origin": "manual",
                    "to_name": "text",
                    "type": "labels",
                    "value": {
                        "end": 837,
                        "labels": [
                            "PER"
                        ],
                        "start": 821,
                        "text": "Hirokazu Matsuno"
                    }
                }
            ],
            "result_count": 0,
            "task": 1,
            "updated_at": "2021-10-20T08:09:19.767039Z",
            "was_cancelled": false
        }
    ],
    "created_at": "2021-10-20T08:08:06.826941Z",
    "data": {
        "text": "TOKYO, Oct 20 (Reuters) - A volcano erupted in Japan on Wednesday, blasting ash several miles into the sky and prompting officials to warn against the threat of lava flows and falling rocks, but there were no immediate reports of casualties or damage.Mount Aso, a tourist destination on the main southern island of Kyushu, sent plumes of ash 3.5 km (2.2 miles) high when it erupted at about 11:43 a.m. (0243 GMT), the Japan Meteorological Agency said.It raised the alert level for the volcano to 3 on a scale of 5, telling people not to approach, and warned of a risk of  large falling rocks and pyroclastic flows within a radius of about 1 km (0.6 mile) around the mountain's Nakadake crater.The government is checking to determine the status of a number of climbers on the mountain at the time, Chief Cabinet Secretary Hirokazu Matsuno told reporters, but added that there were no reports of casualties.Television networks broadcast images of a dark cloud of ash looming over the volcano that swiftly obscured large swathes of the mountain.Ash falls from the 1,592-metre (5,222-foot) mountain in the prefecture of Kumamoto are expected to shower nearby towns until late afternoon, the weather agency added.Mount Aso had a small eruption in 2019, while Japan's worst volcanic disaster in nearly 90 years killed 63 people on Mount Ontake in September 2014."
    },
    "file_upload": "80025440-label_studio_input.json",
    "id": 1,
    "meta": {},
    "predictions": [],
    "project": 1,
    "updated_at": "2021-10-20T08:09:19.734419Z"
}

Now we have the JSON formatted result, what we will do next is to convert it into panda format. Feel free to convert it to any format you like. The important point here is to show you how easy it is to work with the label studio output.
post_processed_result = []

for task in raw_result:
    for label in task['annotations'][0]['result']:
        task_row = {}
        task_row['task_id']     = task['annotations'][0]['id']
        task_row['token']       = label['value']['text']
        task_row['token_start'] = label['value']['start']
        task_row['token_end']   = label['value']['end']
        task_row['label']       = label['value']['labels'][0]
        post_processed_result.append(task_row)

pd.DataFrame(post_processed_result)



Certainly not! Earlier in the article, we discussed the defining characteristics of a good data labelling tool. Let’s go over some of the cool features you can customize for different business scenarios.
# Code for the default NER template<View>    <Labels name=”label” toName=”text”>        <Label value=”PER” background=”red”/>        <Label value=”ORG” background=”darkorange”/>        <Label value=”LOC” background=”orange”/>        <Label value=”MISC” background=”green”/>    </Labels>    <Text name=”text” value=”$text”/></View>
I showed you only how to start NER projects using the freely available template in the case study above. Additionally, Label Studio gives you access to a code editor that you can use to customize your labelling templates. Code is in XML format, so you can add custom label configurations to your dataset using tags. The following are two examples to allow you to further customize your dataset.
NER’s default template assigns ORG and LOC labels very similar colours. It can cause confusion when annotations are checked after labelling. By changing the colour of the LOC label to blue, we can avoid this human error. To make the task more complete, we will also add more labels.
# Modify the LOC background color to blue<View>    <Labels name=”label” toName=”text”>        <Label value=”PER” background=”red”/>        <Label value=”ORG” background=”darkorange”/>        <Label value=”LOC” background=”blue”/>        <Label value=”MISC” background=”green”/>        <Label value=”BRAND” background=”yellow”/>        <Label value=”TIME” background=”purple”/>    </Labels>    <Text name=”text” value=”$text”/></View>
At the time of setting up the project, we only had the option of choosing one template. Even in the case of a single project, you may need to perform multi-task inference. In one of my client’s projects, I was required to design a multi-label classification model that also performed NER. Adding a new labelling job to our existing tasks is simply a matter of adding a few lines to the script.
<View>    <Labels name=”label” toName=”text”>        <Label value=”PER” background=”red”/>        <Label value=”ORG” background=”darkorange”/>        <Label value=”LOC” background=”blue”/>        <Label value=”MISC” background=”green”/>        <Label value=”BRAND” background=”yellow”/>        <Label value=”TIME” background=”purple”/>    </Labels>    <Text name=”text” value=”$text”/>    <Taxonomy name=”article_class” toName=”text”>        <Choice value=”world”>            <Choice value=”africa”/>            <Choice value=”america”/>        </Choice>        <Choice value=”business”>            <Choice value=”environment”/>            <Choice value=”finance”/>        </Choice>    </Taxonomy></View>
We can see that adding a new tag called Taxonomy generates a multi-label classification job right away for the existing task.
Now that you know how easy Label Studio is to use, you can now use it on your next labelling project. In closing, I’d like to point out one more important feature.
There is no doubt that labelling jobs are labour-intensive, which means that we need to spread the tasks among multiple annotators to complete them as quickly as possible. To reduce human bias, we should assign more than one human to each labelling task. The good news is that Label Studio allows annotators to sign up and they can work on the same task if needed.
To start annotating the same articles as the existing annotator, all they need to do is create a new tab under their account:
Our goal in this article is to show you how to build a labelling tool to support your next labelling project. In addition to its array of free features, Label Studio also allows you to customize your labelling interface with just a few lines of code. The platform supports not only natural language processing task but also computer vision and speech recognition tasks. The team edition would be ideal if you intend to use this tool for a long time.
Label Studio really did a great job of making this labelling tool as easy to use as possible. No matter if you are a startup or a small business, labelling can sometimes be a problem when you are developing a new machine learning project. Costs and time can add up when designing a new labelling tool. In this regard, Label Studio is a viable option.
https://labelstud.io/
https://labelstud.io/
Woen Yon is a Data Scientist based in Singapore. His experience includes developing advanced artificial intelligence products for several multinational enterprises.
Woen Yon works with a handful of smart people to offer web solutions including web crawling services and website development for local and international start-up business owners. They are well aware of the challenges of building quality software. Please do not hesitate to drop him an email at wushulai@live.com if you need assistance.
He loves making friends! Feel free to connect with him on LinkedIn and Medium