Question Answering
Transformers
PyTorch
Safetensors
French
camembert
Carbon Emissions
Inference Endpoints
bourdoiscatie commited on
Commit
aba60a4
1 Parent(s): ee45ad0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +15 -21
README.md CHANGED
@@ -30,7 +30,7 @@ co2_eq_emissions: 100
30
 
31
  ## Model Description
32
 
33
- We present **QAmemBERT**, which is a [CamemBERT base](https://huggingface.co/camembert-base) fine-tuned for the Question-Answering task for the French language on four French Q&A datasets composed of contexts and questions with their answers inside the context (= SQuAD v1 format) but also contexts and questions with their answers not inside the context (= SQuAD v2 format).
34
  All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
35
  This represents a total of over **221,348 questions/answers pairs used to finetune this model and 6,376 to test it**.
36
 
@@ -39,14 +39,14 @@ This represents a total of over **221,348 questions/answers pairs used to finetu
39
 
40
  | Dataset | Format | Train split | Dev split | Test split |
41
  | ----------- | ----------- | ----------- | ----------- | ----------- |
42
- | [piaf](https://www.data.gouv.fr/en/datasets/piaf-le-dataset-francophone-de-questions-reponses/)| SQuAD v1 | 9 224 Q & A | X | X |
43
- | piaf_v2| SQuAD v2 | 9 224 Q & A | X | X |
44
- | [fquad](https://fquad.illuin.tech/)| SQuAD v1 | 20 731 Q & A | 3 188 Q & A (not used in training because it serves as a test dataset) | 2 189 Q & A (not used in our work because not freely available)|
45
- | fquad_v2 | SQuAD v2 | 20 731 Q & A | 3 188 Q & A (not used in training because it serves as a test dataset) | X |
46
- | [lincoln/newsquadfr](https://huggingface.co/datasets/lincoln/newsquadfr) | SQuAD v1 | 1 650 Q & A | 455 Q & A (not used in our work) | X |
47
- | lincoln/newsquadfr_v2 | SQuAD v2 | 1 650 Q & A | 455 Q & A (not used in our work) | X |
48
- | [pragnakalp/squad_v2_french_translated](https://huggingface.co/datasets/pragnakalp/squad_v2_french_translated)| SQuAD v2 | 79 069 Q & A | X | X |
49
- | pragnakalp/squad_v2_french_translated_v2| SQuAD v2 | 79 069 Q & A | X | X |
50
 
51
  All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
52
 
@@ -58,44 +58,38 @@ The evaluation was carried out using the [**evaluate**](https://pypi.org/project
58
 
59
  ### FQuaD 1.0 (validation)
60
 
61
- The metric used is Squad v1.
62
 
63
  | Model | Exact_match | F1-score |
64
  | ----------- | ----------- | ----------- |
65
  | [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 53.60 | 78.09 |
66
  | QAmembert (previous version) | 54.26 | 77.87 |
67
  | QAmembert (**this version**) | 53.98 | 78.00 |
68
- | QAmembert-large | **55.95** | **81.05** |
69
- | [fT0](https://huggingface.co/CATIE-AQ/frenchT0) | 41.15 | 65.79 |
70
 
71
- ♪ this model is available on request only
72
 
73
  ### qwant/squad_fr (validation)
74
 
75
- The metric used is Squad v1.
76
 
77
  | Model | Exact_match | F1-score |
78
  | ----------- | ----------- | ----------- |
79
  | [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 60.17 | 78.27 |
80
  | QAmembert (previous version) | 60.40 | 77.27 |
81
  | QAmembert (**this version**) | 60.95 | 77.30 |
82
- | QAmembert-large| **65.58** | **81.74** |
83
- | [fT0](https://huggingface.co/CATIE-AQ/frenchT0) | 41.05 | 56.14 |
84
 
85
- ♪ this model is available on request only.
86
 
87
  ### frenchQA
88
 
89
- This dataset includes question with no answers in the context. The metric used is Squad v2.
90
 
91
  | Model | Exact_match | F1-score | Answer_f1 | NoAnswer_f1 |
92
  | ----------- | ----------- | ----------- | ----------- | ----------- |
93
  | [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | n/a | n/a | n/a | n/a |
94
  | QAmembert (previous version) | 60.28 | 71.29 | 75.92 | 66.65
95
  | QAmembert (**this version**) | **77.14** | 86.88 | 75.66 | 98.11
96
- | QAmembert-large| **77.14** | **88.74** | **78.83** | **98.65**
97
-
98
- ♪ this model is available on request only.
99
 
100
 
101
 
 
30
 
31
  ## Model Description
32
 
33
+ We present **QAmemBERT**, which is a [CamemBERT base](https://huggingface.co/camembert-base) fine-tuned for the Question-Answering task for the French language on four French Q&A datasets composed of contexts and questions with their answers inside the context (= SQuAD 1.0 format) but also contexts and questions with their answers not inside the context (= SQuAD 2.0 format).
34
  All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
35
  This represents a total of over **221,348 questions/answers pairs used to finetune this model and 6,376 to test it**.
36
 
 
39
 
40
  | Dataset | Format | Train split | Dev split | Test split |
41
  | ----------- | ----------- | ----------- | ----------- | ----------- |
42
+ | [piaf](https://www.data.gouv.fr/en/datasets/piaf-le-dataset-francophone-de-questions-reponses/)| SQuAD 1.0 | 9 224 Q & A | X | X |
43
+ | piaf_v2| SQuAD 2.0 | 9 224 Q & A | X | X |
44
+ | [fquad](https://fquad.illuin.tech/)| SQuAD 1.0 | 20 731 Q & A | 3 188 Q & A (not used in training because it serves as a test dataset) | 2 189 Q & A (not used in our work because not freely available)|
45
+ | fquad_v2 | SQuAD 2.0 | 20 731 Q & A | 3 188 Q & A (not used in training because it serves as a test dataset) | X |
46
+ | [lincoln/newsquadfr](https://huggingface.co/datasets/lincoln/newsquadfr) | SQuAD 1.0 | 1 650 Q & A | 455 Q & A (not used in our work) | X |
47
+ | lincoln/newsquadfr_v2 | SQuAD 2.0 | 1 650 Q & A | 455 Q & A (not used in our work) | X |
48
+ | [pragnakalp/squad_v2_french_translated](https://huggingface.co/datasets/pragnakalp/squad_v2_french_translated)| SQuAD 2.0 | 79 069 Q & A | X | X |
49
+ | pragnakalp/squad_v2_french_translated_v2| SQuAD 2.0 | 79 069 Q & A | X | X |
50
 
51
  All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
52
 
 
58
 
59
  ### FQuaD 1.0 (validation)
60
 
61
+ The metric used is SQuAD 1.0.
62
 
63
  | Model | Exact_match | F1-score |
64
  | ----------- | ----------- | ----------- |
65
  | [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 53.60 | 78.09 |
66
  | QAmembert (previous version) | 54.26 | 77.87 |
67
  | QAmembert (**this version**) | 53.98 | 78.00 |
68
+ | [QAmembert-large](https://huggingface.co/CATIE-AQ/QAmembert-large) | **55.95** | **81.05** |
 
69
 
 
70
 
71
  ### qwant/squad_fr (validation)
72
 
73
+ The metric used is SQuAD 1.0.
74
 
75
  | Model | Exact_match | F1-score |
76
  | ----------- | ----------- | ----------- |
77
  | [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 60.17 | 78.27 |
78
  | QAmembert (previous version) | 60.40 | 77.27 |
79
  | QAmembert (**this version**) | 60.95 | 77.30 |
80
+ | [QAmembert-large](https://huggingface.co/CATIE-AQ/QAmembert-large) | **65.58** | **81.74** |
 
81
 
 
82
 
83
  ### frenchQA
84
 
85
+ This dataset includes question with no answers in the context. The metric used is SQuAD 2.0.
86
 
87
  | Model | Exact_match | F1-score | Answer_f1 | NoAnswer_f1 |
88
  | ----------- | ----------- | ----------- | ----------- | ----------- |
89
  | [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | n/a | n/a | n/a | n/a |
90
  | QAmembert (previous version) | 60.28 | 71.29 | 75.92 | 66.65
91
  | QAmembert (**this version**) | **77.14** | 86.88 | 75.66 | 98.11
92
+ | [QAmembert-large](https://huggingface.co/CATIE-AQ/QAmembert-large) | **77.14** | **88.74** | **78.83** | **98.65**
 
 
93
 
94
 
95