Release of FairLex dataset
State: closed | Author: iliaschalkidis (CONTRIBUTOR)


**FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing**

We present a benchmark suite of four datasets for evaluating the fairness of pre-trained legal language models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (European Council, USA, Swiss, and Chinese), five languages (English, German, French, Italian, and Chinese), and fairness across five attributes (gender, age, nationality/region, language, and legal area). In our experiments, we evaluate pre-trained language models using several group-robust fine-tuning techniques and show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Furthermore, we provide a quantitative and qualitative analysis of our results, highlighting open challenges in the development of robustness methods in legal NLP.

*Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Letizia, Sebastian Felix Schwemer, Anders Søgaard. FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing. 2022. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland.*

Note: Please review this initial commit, and I'll update the publication link, once I'll have the ArXived version. Thanks!


Comments: This is awesome ! The dataset card and the dataset script look amazing :)

I wanted to ask you if you'd be interested to have this dataset under the namespace of you research group at https://huggingface.co/coastalcph ? If yes, then you can actually create a dataset repository under your research group name and upload the files from this PR there

Hi @lhoestq,

Yeah, I could do that. I see that people do that a lot of models, but not for datasets. 

Is there any good reason to have it under the organization domain instead of the general domain?

 Thanks!

It's nice to have it under your namespace:
- it will appear on your research group page, along with your models
- you can edit or create datasets at any time - you don't need to open PRs on GitHub

All the datasets that are not under a namespace are this way because we started adding datasets from GitHub. Now we encourage users to upload them directly to make things simpler, and aligned with the workflow for models

(the documentation will be updated in the following days)

Note that we will keep accepting PRs here though when there is no clear namespace under which a dataset should be, or for users that want a review from us

Ok, I'll do that. So, I'll just have to upload all the files under the `/fairlex` directory in my PR, right?

Yes exactly !

Ok, I uploaded most of them from the UI environment (https://huggingface.co/datasets/coastalcph/fairlex). Can I possibly upload the dummy data as well from the UI environment. I really want to avoid the CLI right now 😄 

Yea sure, feel free to use the UI of the website, even for the dummy data ^^

Did you upload them yourself? Because I see the data preview, and I'm pretty sure, I didn't do that 😄 

The preview is computed from the real data ;)

The dummy data are used for testing only

Haha, ok I was shocked! Cool, I close this PR, then. Thanks, again! 

Thank you 🤗