take off detailed fairness table
Browse files
README.md
CHANGED
|
@@ -365,46 +365,6 @@ Besides, we also computed the classification accuracy on FairFace for both the b
|
|
| 365 |
| IDEFICS 80B Instruct | 0 | 95.7 (2.4) | 63.4 (25.6) | 47.1 (2.9) |
|
| 366 |
| IDEFICS 9B Instruct | 0 | 92.7 (6.3) | 59.6 (22.2) | 43.9 (3.9) |
|
| 367 |
|
| 368 |
-
|
| 369 |
-
| Model | Shots | <nobr>Task<br> | <nobr>Gender<br> | <nobr>White<br> | <nobr>Black<br> | <nobr>Latino_Hispanic<br> | <nobr>East<br> | <nobr>Southeast Asian<br> | <nobr>Indian<br> | <nobr>Middle Eastern<br> | <nobr>Average<br> |
|
| 370 |
-
|:---------------------|------------------:|:------------------|:-------------------|------------------:|------------------:|----------------------------:|-----------------:|----------------------------:|-------------------:|---------------------------:|--------------------:|
|
| 371 |
-
| IDEFICS 80B | 0 | FairFace - Gender | Male | 95.4 | 96.1 | 94.4 | 96.3 | 97.2 | 94.8 | 94.4 | 95.5 |
|
| 372 |
-
| | 0 | | Female | 94.8 | 97.8 | 95.8 | 95.1 | 96.5 | 97.3 | 95.7 | 96.1 |
|
| 373 |
-
| | 0 | | Average | 95.1 | 97.0 | 95.1 | 95.7 | 96.8 | 96.1 | 95.1 | 95.8 |
|
| 374 |
-
| | 0 | FairFace - Race | Male | 73.4 | 71.4 | 87.9 | 73.2 | 52.3 | 35.6 | 63.0 | 65.2 |
|
| 375 |
-
| | 0 | | Female | 63.8 | 73.7 | 82.7 | 68.6 | 33.6 | 42.4 | 58.2 | 60.4 |
|
| 376 |
-
| | 0 | | Average | 68.6 | 72.6 | 85.3 | 70.9 | 42.9 | 39.0 | 60.6 | 62.8 |
|
| 377 |
-
| | 0 | FairFace - Age | Male | 53.3 | 51.9 | 50.7 | 54.7 | 48.3 | 48.4 | 52.4 | 51.4 |
|
| 378 |
-
| | 0 | | Female | 55.0 | 47.4 | 45.3 | 53.5 | 53.8 | 48.1 | 49.9 | 50.4 |
|
| 379 |
-
| | 0 | | Average | 54.1 | 49.7 | 48.0 | 54.1 | 51.1 | 48.2 | 51.1 | 50.9 |
|
| 380 |
-
| IDEFICS 9B | 0 | FairFace - Gender | Male | 92.9 | 94.6 | 91.5 | 94.7 | 95.2 | 91.6 | 90.2 | 93.0 |
|
| 381 |
-
| | 0 | | Female | 94.6 | 97.5 | 95.6 | 94.1 | 96.5 | 97.5 | 96.2 | 96.0 |
|
| 382 |
-
| | 0 | | Average | 93.7 | 96.0 | 93.6 | 94.4 | 95.8 | 94.5 | 93.2 | 94.5 |
|
| 383 |
-
| | 0 | FairFace - Race | Male | 54.2 | 55.5 | 70.0 | 41.3 | 50.3 | 50.6 | 75.0 | 56.7 |
|
| 384 |
-
| | 0 | | Female | 44.1 | 65.3 | 67.4 | 42.1 | 29.5 | 56.0 | 73.1 | 53.9 |
|
| 385 |
-
| | 0 | | Average | 49.1 | 60.4 | 68.7 | 41.7 | 39.9 | 53.3 | 74.0 | 55.3 |
|
| 386 |
-
| | 0 | FairFace - Age | Male | 50.1 | 45.6 | 46.8 | 44.5 | 39.2 | 42.2 | 45.4 | 44.8 |
|
| 387 |
-
| | 0 | | Female | 48.1 | 43.9 | 41.3 | 48.9 | 46.7 | 42.9 | 45.7 | 45.4 |
|
| 388 |
-
| | 0 | | Average | 49.1 | 44.7 | 44.1 | 46.7 | 43.0 | 42.6 | 45.6 | 45.1 |
|
| 389 |
-
| IDEFICS 80B Instruct | 0 | FairFace - Gender | Male | 94.0 | 95.9 | 92.5 | 93.0 | 96.4 | 91.9 | 92.8 | 93.8 |
|
| 390 |
-
| | 0 | | Female | 98.2 | 98.2 | 97.6 | 96.9 | 98.5 | 98.6 | 98.2 | 98.0 |
|
| 391 |
-
| | 0 | | Average | 96.1 | 97.0 | 95.1 | 95.0 | 97.5 | 95.2 | 95.5 | 95.9 |
|
| 392 |
-
| | 0 | FairFace - Race | Male | 85.2 | 72.9 | 93.7 | 82.7 | 43.9 | 19.2 | 38.9 | 62.4 |
|
| 393 |
-
| | 0 | | Female | 77.5 | 80.7 | 88.8 | 79.2 | 29.0 | 28.2 | 37.9 | 60.2 |
|
| 394 |
-
| | 0 | | Average | 81.3 | 76.8 | 91.3 | 81.0 | 36.5 | 23.7 | 38.4 | 61.3 |
|
| 395 |
-
| | 0 | FairFace - Age | Male | 51.2 | 48.3 | 47.6 | 50.6 | 45.8 | 44.1 | 46.8 | 47.8 |
|
| 396 |
-
| | 0 | | Female | 48.1 | 44.4 | 42.0 | 49.3 | 44.9 | 42.7 | 50.6 | 46.0 |
|
| 397 |
-
| | 0 | | Average | 49.7 | 46.4 | 44.8 | 50.0 | 45.4 | 43.4 | 48.7 | 46.9 |
|
| 398 |
-
| IDEFICS 9B Instruct | 0 | FairFace - Gender | Male | 87.1 | 89.6 | 84.6 | 87.6 | 90.3 | 85.8 | 83.3 | 86.9 |
|
| 399 |
-
| | 0 | | Female | 98.7 | 99.6 | 98.8 | 98.7 | 99.0 | 99.8 | 99.0 | 99.1 |
|
| 400 |
-
| | 0 | | Average | 92.9 | 94.6 | 91.7 | 93.1 | 94.6 | 92.8 | 91.1 | 93.0 |
|
| 401 |
-
| | 0 | FairFace - Race | Male | 67.1 | 62.3 | 92.5 | 77.4 | 41.1 | 24.0 | 43.5 | 58.3 |
|
| 402 |
-
| | 0 | | Female | 61.1 | 73.0 | 87.2 | 77.9 | 22.5 | 33.1 | 42.5 | 56.7 |
|
| 403 |
-
| | 0 | | Average | 64.1 | 67.6 | 89.8 | 77.6 | 31.8 | 28.5 | 43.0 | 57.5 |
|
| 404 |
-
| | 0 | FairFace - Age | Male | 51.4 | 44.2 | 43.4 | 45.6 | 40.1 | 40.2 | 44.1 | 44.1 |
|
| 405 |
-
| | 0 | | Female | 51.2 | 39.3 | 38.2 | 46.7 | 44.9 | 41.1 | 43.4 | 43.6 |
|
| 406 |
-
| | 0 | | Average | 51.3 | 41.8 | 40.8 | 46.2 | 42.5 | 40.7 |
|
| 407 |
-
|
| 408 |
## Other limitations
|
| 409 |
|
| 410 |
- The model currently will offer medical diagnosis when prompted to do so. For example, the prompt `Does this X-ray show any medical problems?` along with an image of a chest X-ray returns `Yes, the X-ray shows a medical problem, which appears to be a collapsed lung.`
|
|
|
|
| 365 |
| IDEFICS 80B Instruct | 0 | 95.7 (2.4) | 63.4 (25.6) | 47.1 (2.9) |
|
| 366 |
| IDEFICS 9B Instruct | 0 | 92.7 (6.3) | 59.6 (22.2) | 43.9 (3.9) |
|
| 367 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 368 |
## Other limitations
|
| 369 |
|
| 370 |
- The model currently will offer medical diagnosis when prompted to do so. For example, the prompt `Does this X-ray show any medical problems?` along with an image of a chest X-ray returns `Yes, the X-ray shows a medical problem, which appears to be a collapsed lung.`
|