add more complete fairness tables
Browse files
README.md
CHANGED
|
@@ -358,12 +358,52 @@ The [notebook](https://huggingface.co/spaces/HuggingFaceM4/m4-bias-eval/blob/mai
|
|
| 358 |
|
| 359 |
Besides, we also computed the classification accuracy on FairFace for both the base and instructed models:
|
| 360 |
|
| 361 |
-
| Model | Shots | <nobr>FairFace - Gender<br>acc.</nobr> | <nobr>FairFace - Race<br>acc.</nobr> | <nobr>FairFace - Age<br>acc.</nobr> |
|
| 362 |
-
|
| 363 |
-
| IDEFICS 80B | 0 | 95.8 | 64.1 | 51.0 |
|
| 364 |
-
| IDEFICS 9B | 0 | 94.4 | 55.3 | 45.1 |
|
| 365 |
-
| IDEFICS 80B Instruct | 0 | 95.7 | 63.4 | 47.1 |
|
| 366 |
-
| IDEFICS 9B Instruct | 0 | 92.7 | 59.6 | 43.9 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 367 |
|
| 368 |
## Other limitations
|
| 369 |
|
|
|
|
| 358 |
|
| 359 |
Besides, we also computed the classification accuracy on FairFace for both the base and instructed models:
|
| 360 |
|
| 361 |
+
| Model | Shots | <nobr>FairFace - Gender<br>acc.</nobr> | <nobr>FairFace - Gender<br>per bucket std.</nobr> | <nobr>FairFace - Race<br>acc.</nobr> | <nobr>FairFace - Race<br>per bucket std.</nobr> | <nobr>FairFace - Age<br>acc.</nobr> | <nobr>FairFace - Age<br>per bucket std.</nobr> |
|
| 362 |
+
|:---------------------|--------:|----------------------------:|-----------------------:|--------------------------:|---------------------:|-------------------------:|--------------------:|
|
| 363 |
+
| IDEFICS 80B | 0 | 95.8 | 1.0 | 64.1 | 16.1 | 51.0 | 2.9 |
|
| 364 |
+
| IDEFICS 9B | 0 | 94.4 | 2.2 | 55.3 | 13.0 | 45.1 | 2.9 |
|
| 365 |
+
| IDEFICS 80B Instruct | 0 | 95.7 | 2.4 | 63.4 | 25.6 | 47.1 | 2.9 |
|
| 366 |
+
| IDEFICS 9B Instruct | 0 | 92.7 | 6.3 | 59.6 | 22.2 | 43.9 | 3.9 |
|
| 367 |
+
|
| 368 |
+
|
| 369 |
+
| Model | Shots | Task | Gender | White | Black | Latino_Hispanic | East | Southeast Asian | Indian | Middle Eastern | Average |
|
| 370 |
+
|:---------------------|--------:|:----------------|:---------|--------:|--------:|------------------:|-------:|------------------:|---------:|-----------------:|----------:|
|
| 371 |
+
| IDEFICS 80B | 0 | FairFace Gender | Male | 95.4 | 96.1 | 94.4 | 96.3 | 97.2 | 94.8 | 94.4 | 95.5 |
|
| 372 |
+
| | 0 | | Female | 94.8 | 97.8 | 95.8 | 95.1 | 96.5 | 97.3 | 95.7 | 96.1 |
|
| 373 |
+
| | 0 | | Average | 95.1 | 97.0 | 95.1 | 95.7 | 96.8 | 96.1 | 95.1 | 95.8 |
|
| 374 |
+
| | 0 | FairFace Race | Male | 73.4 | 71.4 | 87.9 | 73.2 | 52.3 | 35.6 | 63.0 | 65.2 |
|
| 375 |
+
| | 0 | | Female | 63.8 | 73.7 | 82.7 | 68.6 | 33.6 | 42.4 | 58.2 | 60.4 |
|
| 376 |
+
| | 0 | | Average | 68.6 | 72.6 | 85.3 | 70.9 | 42.9 | 39.0 | 60.6 | 62.8 |
|
| 377 |
+
| | 0 | FairFace Age | Male | 53.3 | 51.9 | 50.7 | 54.7 | 48.3 | 48.4 | 52.4 | 51.4 |
|
| 378 |
+
| | 0 | | Female | 55.0 | 47.4 | 45.3 | 53.5 | 53.8 | 48.1 | 49.9 | 50.4 |
|
| 379 |
+
| | 0 | | Average | 54.1 | 49.7 | 48.0 | 54.1 | 51.1 | 48.2 | 51.1 | 50.9 |
|
| 380 |
+
| IDEFICS 9B | 0 | FairFace Gender | Male | 92.9 | 94.6 | 91.5 | 94.7 | 95.2 | 91.6 | 90.2 | 93.0 |
|
| 381 |
+
| | 0 | | Female | 94.6 | 97.5 | 95.6 | 94.1 | 96.5 | 97.5 | 96.2 | 96.0 |
|
| 382 |
+
| | 0 | | Average | 93.7 | 96.0 | 93.6 | 94.4 | 95.8 | 94.5 | 93.2 | 94.5 |
|
| 383 |
+
| | 0 | FairFace Race | Male | 54.2 | 55.5 | 70.0 | 41.3 | 50.3 | 50.6 | 75.0 | 56.7 |
|
| 384 |
+
| | 0 | | Female | 44.1 | 65.3 | 67.4 | 42.1 | 29.5 | 56.0 | 73.1 | 53.9 |
|
| 385 |
+
| | 0 | | Average | 49.1 | 60.4 | 68.7 | 41.7 | 39.9 | 53.3 | 74.0 | 55.3 |
|
| 386 |
+
| | 0 | FairFace Age | Male | 50.1 | 45.6 | 46.8 | 44.5 | 39.2 | 42.2 | 45.4 | 44.8 |
|
| 387 |
+
| | 0 | | Female | 48.1 | 43.9 | 41.3 | 48.9 | 46.7 | 42.9 | 45.7 | 45.4 |
|
| 388 |
+
| | 0 | | Average | 49.1 | 44.7 | 44.1 | 46.7 | 43.0 | 42.6 | 45.6 | 45.1 |
|
| 389 |
+
| IDEFICS 80B Instruct | 0 | FairFace Gender | Male | 94.0 | 95.9 | 92.5 | 93.0 | 96.4 | 91.9 | 92.8 | 93.8 |
|
| 390 |
+
| | 0 | | Female | 98.2 | 98.2 | 97.6 | 96.9 | 98.5 | 98.6 | 98.2 | 98.0 |
|
| 391 |
+
| | 0 | | Average | 96.1 | 97.0 | 95.1 | 95.0 | 97.5 | 95.2 | 95.5 | 95.9 |
|
| 392 |
+
| | 0 | FairFace Race | Male | 85.2 | 72.9 | 93.7 | 82.7 | 43.9 | 19.2 | 38.9 | 62.4 |
|
| 393 |
+
| | 0 | | Female | 77.5 | 80.7 | 88.8 | 79.2 | 29.0 | 28.2 | 37.9 | 60.2 |
|
| 394 |
+
| | 0 | | Average | 81.3 | 76.8 | 91.3 | 81.0 | 36.5 | 23.7 | 38.4 | 61.3 |
|
| 395 |
+
| | 0 | FairFace Age | Male | 51.2 | 48.3 | 47.6 | 50.6 | 45.8 | 44.1 | 46.8 | 47.8 |
|
| 396 |
+
| | 0 | | Female | 48.1 | 44.4 | 42.0 | 49.3 | 44.9 | 42.7 | 50.6 | 46.0 |
|
| 397 |
+
| | 0 | | Average | 49.7 | 46.4 | 44.8 | 50.0 | 45.4 | 43.4 | 48.7 | 46.9 |
|
| 398 |
+
| IDEFICS 9B Instruct | 0 | FairFace Gender | Male | 87.1 | 89.6 | 84.6 | 87.6 | 90.3 | 85.8 | 83.3 | 86.9 |
|
| 399 |
+
| | 0 | | Female | 98.7 | 99.6 | 98.8 | 98.7 | 99.0 | 99.8 | 99.0 | 99.1 |
|
| 400 |
+
| | 0 | | Average | 92.9 | 94.6 | 91.7 | 93.1 | 94.6 | 92.8 | 91.1 | 93.0 |
|
| 401 |
+
| | 0 | FairFace Race | Male | 67.1 | 62.3 | 92.5 | 77.4 | 41.1 | 24.0 | 43.5 | 58.3 |
|
| 402 |
+
| | 0 | | Female | 61.1 | 73.0 | 87.2 | 77.9 | 22.5 | 33.1 | 42.5 | 56.7 |
|
| 403 |
+
| | 0 | | Average | 64.1 | 67.6 | 89.8 | 77.6 | 31.8 | 28.5 | 43.0 | 57.5 |
|
| 404 |
+
| | 0 | FairFace Age | Male | 51.4 | 44.2 | 43.4 | 45.6 | 40.1 | 40.2 | 44.1 | 44.1 |
|
| 405 |
+
| | 0 | | Female | 51.2 | 39.3 | 38.2 | 46.7 | 44.9 | 41.1 | 43.4 | 43.6 |
|
| 406 |
+
| | 0 | | Average | 51.3 | 41.8 | 40.8 | 46.2 | 42.5 | 40.7 | 43.7 | 43.9 |
|
| 407 |
|
| 408 |
## Other limitations
|
| 409 |
|