GitHub issue #383: Adding the Linguistic Code-switching Evaluation (LinCE) benchmark
State: closed | Created: 2020-07-11T22:35:20Z

Hi,

First of all, this library is really cool! Thanks for putting all of this together! 

This PR contains the [Linguistic Code-switching Evaluation (LinCE) benchmark](https://ritual.uh.edu/lince). As described in the official website (FAQ):

> 1. Why do we need LinCE?
>LinCE brings 10 code-switching datasets together for 4 tasks and 4 language pairs with 5 leaderboards in a single evaluation platform. We examined each dataset and fixed major issues on the partitions (or even define official partitions) with a comprehensive stratification method (see our paper for more details).
>Besides, we believe that online benchmarks like LinCE bring steady research progress and allow to compare state-of-the-art models at the pace of the progress in NLP. We expect to benefit greatly the code-switching community with this benchmark.


The data comes from social media and here's the summary table of tasks per language pair:

| Language Pairs                         | LID | POS | NER | SA |
|----------------------------------------|-----|-----|-----|----|
| Spanish-English                        | ✅  | ✅  | ✅  | ✅ |
| Hindi-English                          | ✅  | ✅  | ✅  |    |
| Modern Standard Arabic-Egyptian Arabic | ✅  |     | ✅  |    |
| Nepali-English                         | ✅  |     |     |    |

The tasks are as follows:
* LID: token-level language identification
* POS: part-of-speech tagging
* NER: named entity recognition
* SA: sentiment analysis

With the exception of MSA-EA, the rest of the datasets contain token-level LID labels.

## Usage

For Spanish-English LID, we can load the data as follows:
```
import nlp

data = nlp.load_dataset('./datasets/lince/lince.py', 'lid_spaeng')

for split in data:
    print(data[split])
```

Here's the output:
```
Dataset(schema: {'idx': 'int32', 'tokens': 'list<item: string>', 'lid': 'list<item: string>'}, num_rows: 21030)
Dataset(schema: {'idx': 'int32', 'tokens': 'list<item: string>', 'lid': 'list<item: string>'}, num_rows: 3332)
Dataset(schema: {'idx': 'int32', 'tokens': 'list<item: string>', 'lid': 'list<item: string>'}, num_rows: 8289)
```

Here's the list of shortcut names for every dataset available in LinCE:
* `lid_spaeng`
* `lid_hineng`
* `lid_nepeng`
* `lid_msaea`
* `pos_spaeng`
* `pos_hineng`
* `ner_spaeng`
* `ner_hineng`
* `ner_msaea`
* `sa_spaeng`


All the numbers match with Table 3 in the LinCE [paper](https://www.aclweb.org/anthology/2020.lrec-1.223.pdf). Also, note that the MSA-EA datasets use the Persian script while the other datasets use the Roman script.


## Features

Here is how the features look in the case of language identification (LID) tasks:

| LID Feature          | Type          | Description                               |
|----------------------|---------------|-------------------------------------------|
| `idx`                | `int`       | Dataset index of current sentence         |
| `tokens`             | `list<str>` | List of tokens (string) of a sentence     |
| `lid`                | `list<str>` | List of LID labels (string) of a sentence |

For part-of-speech (POS) tagging:

| POS Feature          | Type          | Description                               |
|----------------------|---------------|-------------------------------------------|
| `idx`                | `int`       | Dataset index of current sentence         |
| `tokens`             | `list<str>` | List of tokens (string) of a sentence     |
| `lid`                | `list<str>` | List of LID labels (string) of a sentence |
| `pos`                | `list<str>` | List of POS tags (string) of a sentence   |

For named entity recognition (NER):

| NER Feature          | Type          | Description                               |
|----------------------|---------------|-------------------------------------------|
| `idx`                | `int`       | Dataset index of current sentence         |
| `tokens`             | `list<str>` | List of tokens (string) of a sentence     |
| `lid`                | `list<str>` | List of LID labels (string) of a sentence |
| `ner`                | `list<str>` | List of NER labels (string) of a sentence |

**NOTE**: the MSA-EA NER dataset does not contain the `lid` feature.

For sentiment analysis (SA):

| SA Feature          | Type        | Description                               |
|---------------------|-------------|-------------------------------------------|
| `idx`               | `int`       | Dataset index of current sentence         |
| `tokens`            | `list<str>` | List of tokens (string) of a sentence     |
| `lid`               | `list<str>` | List of LID labels (string) of a sentence |
| `sa`                | `str`       | Sentiment label (string) of a sentence    |