Repository: embeddings-benchmark/mteb
Pull request #2645: Added SpeechCommands Dataset (Subset)
Author: login: AdnanElAssadi56
Created: 2025-05-03T20:32:16Z | Merged: 2025-05-06T19:52:23Z

<!-- If you are submitting a dataset or a model for the model registry please use the corresponding checklists below otherwise feel free to remove them. -->

<!-- add additional description, question etc. related to the new dataset -->


### Code Quality
<!-- Please do not delete this -->
- [ ] **Code Formatted**: Format the code using `make lint` to maintain consistent style.

### Documentation
<!-- Please do not delete this -->
- [ ] **Updated Documentation**: Add or update documentation to reflect the changes introduced in this PR.

### Testing
<!-- Please do not delete this -->
- [ ] **New Tests Added**: Write tests to cover new functionality. Validate with `make test-with-coverage`.
- [ ] **Tests Passed**: Run tests locally using `make test` or `make test-with-coverage` to ensure no existing functionality is broken.


### Adding datasets checklist
<!-- see also https://github.com/embeddings-benchmark/mteb/blob/main/docs/adding_a_dataset.md -->

**Reason for dataset addition**: ... <!-- Add reason for adding dataset here. E.g. it covers task/language/domain previously not covered -->

- [ ] I have run the following models on the task (adding the results to the pr). These can be run using the `mteb -m {model_name} -t {task_name}` command.
  - [ ] `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`
  - [ ] `intfloat/multilingual-e5-small`
- [ ] I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
- [ ] If the dataset is too big (e.g. >2048 examples), considering using `self.stratified_subsampling() under dataset_transform()`
- [ ] I have filled out the metadata object in the dataset file (find documentation on it [here](https://github.com/embeddings-benchmark/mteb/blob/main/docs/adding_a_dataset.md#2-creating-the-metadata-object)).
- [ ] Run tests locally to make sure nothing is broken using `make test`.
- [ ] Run the formatter to format the code using `make lint`.


### Adding a model checklist
<!--
When adding a model to the model registry
see also https://github.com/embeddings-benchmark/mteb/blob/main/docs/reproducible_workflow.md
-->

 - [ ] I have filled out the ModelMeta object to the extent possible
 - [ ] I have ensured that my model can be loaded using
   - [ ] `mteb.get_model(model_name, revision)` and
   - [ ] `mteb.get_model_meta(model_name, revision)`
 - [ ] I have tested the implementation works on a representative set of tasks.



Diff:
diff --git a/mteb/abstasks/TaskMetadata.py b/mteb/abstasks/TaskMetadata.py
index 70cd17a829..4f5453ee16 100644
--- a/mteb/abstasks/TaskMetadata.py
+++ b/mteb/abstasks/TaskMetadata.py
@@ -21,6 +21,7 @@
 
 TASK_SUBTYPE = Literal[
     "Article retrieval",
+    "Accent identification",
     "Conversational retrieval",
     "Dialect pairing",
     "Dialog Systems",
@@ -61,6 +62,7 @@
     "Stroke Classification of Musical Instrument",
     "Tonic Classification of Musical Instrument",
     "Speaker Count Identification",
+    "Species Classification",
     "Spoken Digit Classification",
     "Gender Clustering",
     "Music Clustering",
@@ -68,6 +70,7 @@
     "Sentiment Analysis",
     "Intent Classification",
     "Vehicle Clustering",
+    "Environment Sound Clustering",
     "Rendered semantic textual similarity",
     "Gender Classification",
     "Age Classification",
@@ -75,6 +78,7 @@
 
 TASK_DOMAIN = Literal[
     "Academic",
+    "Bioacoustics",
     "Blog",
     "Constructed",
     "Encyclopaedic",
diff --git a/mteb/tasks/Audio/AudioClassification/__init__.py b/mteb/tasks/Audio/AudioClassification/__init__.py
index fe2257b321..0cf7ed89d8 100644
--- a/mteb/tasks/Audio/AudioClassification/__init__.py
+++ b/mteb/tasks/Audio/AudioClassification/__init__.py
@@ -14,6 +14,7 @@
 from .eng.MridinghamTonic import *
 from .eng.NSynth import *
 from .eng.SIBFLEURS import *
+from .eng.SpeechCommands import *
 from .eng.SpokenQAforIC import *
 from .eng.VoxCelebSA import *
 from .eng.VoxLingua107Top10 import *
diff --git a/mteb/tasks/Audio/AudioClassification/eng/SpeechCommands.py b/mteb/tasks/Audio/AudioClassification/eng/SpeechCommands.py
new file mode 100644
index 0000000000..40f680a49d
--- /dev/null
+++ b/mteb/tasks/Audio/AudioClassification/eng/SpeechCommands.py
@@ -0,0 +1,86 @@
+from __future__ import annotations
+
+from mteb.abstasks.Audio.AbsTaskAudioClassification import (
+    AbsTaskAudioClassification,
+)
+from mteb.abstasks.TaskMetadata import TaskMetadata
+
+
+class SpeechCommandsClassification(AbsTaskAudioClassification):
+    metadata = TaskMetadata(
+        name="SpeechCommands",
+        description="A set of one-second .wav audio files, each containing a single spoken English word or background noise. To keep evaluation fast, we use a downsampled version of the original dataset by keeping ~50 samples per class for training.",
+        reference="https://arxiv.org/abs/1804.03209",
+        dataset={
+            "path": "AdnanElAssadi/speech_commands_small",
+            "revision": "a59564b91bf0cfcf587e11c2603fe42bae21e5f0",
+        },
+        type="AudioClassification",
+        category="a2t",
+        eval_splits=["test"],
+        eval_langs=["eng-Latn"],
+        main_score="accuracy",
+        date=("2018-04-11", "2018-04-11"),  # v0.02 release date
+        domains=["Speech"],
+        task_subtypes=["Spoken Language Identification"],
+        license="cc-by-4.0",
+        annotations_creators="human-annotated",
+        dialect=[],
+        modalities=["audio"],
+        sample_creation="found",
+        bibtex_citation=r"""
+@article{speechcommands2018,
+  author = {Pete Warden},
+  journal = {arXiv preprint arXiv:1804.03209},
+  title = {Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition},
+  year = {2018},
+}
+""",
+        descriptive_stats={
+            "n_samples": {"train": 1755, "validation": 9982, "test": 4890},
+            "n_classes": 36,
+            "classes": [
+                "yes",
+                "no",
+                "up",
+                "down",
+                "left",
+                "right",
+                "on",
+                "off",
+                "stop",
+                "go",
+                "zero",
+                "one",
+                "two",
+                "three",
+                "four",
+                "five",
+                "six",
+                "seven",
+                "eight",
+                "nine",
+                "bed",
+                "bird",
+                "cat",
+                "dog",
+                "happy",
+                "house",
+                "marvin",
+                "sheila",
+                "tree",
+                "wow",
+                "backward",
+                "forward",
+                "follow",
+                "learn",
+                "visual",
+                "_unknown_",  # (likely background noise or silent segments)
+            ],
+        },
+    )
+
+    audio_column_name: str = "audio"
+    label_column_name: str = "label"
+    samples_per_label: int = 50
+    is_cross_validation: bool = False


Reviews:
user: login: KennethEnevoldsen
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T09:46:33Z

Review comments:
diff_hunk: @@ -0,0 +1,84 @@
+from __future__ import annotations
+
+from mteb.abstasks.Audio.AbsTaskAudioClassification import (
+    AbsTaskAudioClassification,
+)
+from mteb.abstasks.TaskMetadata import TaskMetadata
+
+
+class SpeechCommandsClassification(AbsTaskAudioClassification):
+    metadata = TaskMetadata(
+        name="SpeechCommands",
+        description="A set of one-second .wav audio files, each containing a single spoken English word or background noise.",
+        reference="https://arxiv.org/abs/1804.03209",
+        dataset={
+            "path": "AdnanElAssadi/speech_commands_small",
+            "revision": "a59564b91bf0cfcf587e11c2603fe42bae21e5f0",  # Using downsampled version of v0.02
+        },
+        type="AudioClassification",
+        category="a2t",
+        eval_splits=["validation", "test"],
path: mteb/tasks/Audio/AudioClassification/eng/SpeechCommands.py
user: login: KennethEnevoldsen
body: ```suggestion
        eval_splits=["test"],
```

Probably just default to testing on the test set
created_at: 2025-05-05T09:44:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,84 @@
+from __future__ import annotations
+
+from mteb.abstasks.Audio.AbsTaskAudioClassification import (
+    AbsTaskAudioClassification,
+)
+from mteb.abstasks.TaskMetadata import TaskMetadata
+
+
+class SpeechCommandsClassification(AbsTaskAudioClassification):
+    metadata = TaskMetadata(
+        name="SpeechCommands",
+        description="A set of one-second .wav audio files, each containing a single spoken English word or background noise.",
+        reference="https://arxiv.org/abs/1804.03209",
+        dataset={
+            "path": "AdnanElAssadi/speech_commands_small",
+            "revision": "a59564b91bf0cfcf587e11c2603fe42bae21e5f0",  # Using downsampled version of v0.02
path: mteb/tasks/Audio/AudioClassification/eng/SpeechCommands.py
user: login: KennethEnevoldsen
body: ```suggestion
            "revision": "a59564b91bf0cfcf587e11c2603fe42bae21e5f0",
```
How it was downsamples should be explained in the description. E.g. 

"To keep evaluations fast, this dataset was downsampled to N test samples."
created_at: 2025-05-05T09:46:07Z
author_association: CONTRIBUTOR

PR comments:
user: login: KennethEnevoldsen
created_at: 2025-05-04T10:20:01Z
body: See comment on: #2641
author_association: CONTRIBUTOR
user: login: AdnanElAssadi56
created_at: 2025-05-05T05:12:46Z
body: Classification Task Results:

**Model: "facebook/wav2vec2-base-960h"**

```python
{
  "dataset_revision": "a59564b91bf0cfcf587e11c2603fe42bae21e5f0",
  "task_name": "SpeechCommands",
  "mteb_version": "1.34.7",
  "scores": {
    "validation": [
      {
        "accuracy": 0.820016,
        "f1": 0.78786,
        "f1_weighted": 0.828848,
        "scores_per_experiment": [
          {
            "accuracy": 0.826187,
            "f1": 0.793806,
            "f1_weighted": 0.833697
          },
          {
            "accuracy": 0.814266,
            "f1": 0.783741,
            "f1_weighted": 0.824581
          },
          {
            "accuracy": 0.81657,
            "f1": 0.784775,
            "f1_weighted": 0.824561
          },
          {
            "accuracy": 0.820978,
            "f1": 0.787059,
            "f1_weighted": 0.83015
          },
          {
            "accuracy": 0.82208,
            "f1": 0.789918,
            "f1_weighted": 0.831251
          }
        ],
        "main_score": 0.820016,
        "hf_subset": "default",
        "languages": [
          "eng-Latn"
        ]
      }
    ],
    "test": [
      {
        "accuracy": 0.772761,
        "f1": 0.59604,
        "f1_weighted": 0.805765,
        "scores_per_experiment": [
          {
            "accuracy": 0.772188,
            "f1": 0.593245,
            "f1_weighted": 0.801258
          },
          {
            "accuracy": 0.786503,
            "f1": 0.611583,
            "f1_weighted": 0.818594
          },
          {
            "accuracy": 0.770348,
            "f1": 0.593791,
            "f1_weighted": 0.800519
          },
          {
            "accuracy": 0.766667,
            "f1": 0.587611,
            "f1_weighted": 0.806996
          },
          {
            "accuracy": 0.768098,
            "f1": 0.593971,
            "f1_weighted": 0.801459
          }
        ],
        "main_score": 0.772761,
        "hf_subset": "default",
        "languages": [
          "eng-Latn"
        ]
      }
    ]
  },
  "evaluation_time": 668.0435230731964,
  "kg_co2_emissions": null
}
```


**Model: "microsoft/wavlm-base"**

```python

{
  "dataset_revision": "a59564b91bf0cfcf587e11c2603fe42bae21e5f0",
  "task_name": "SpeechCommands",
  "mteb_version": "1.34.7",
  "scores": {
    "validation": [
      {
        "accuracy": 0.897556,
        "f1": 0.878752,
        "f1_weighted": 0.900454,
        "scores_per_experiment": [
          {
            "accuracy": 0.900421,
            "f1": 0.88042,
            "f1_weighted": 0.903247
          },
          {
            "accuracy": 0.89441,
            "f1": 0.873538,
            "f1_weighted": 0.89769
          },
          {
            "accuracy": 0.891405,
            "f1": 0.87075,
            "f1_weighted": 0.894241
          },
          {
            "accuracy": 0.904027,
            "f1": 0.878108,
            "f1_weighted": 0.906619
          },
          {
            "accuracy": 0.897516,
            "f1": 0.890946,
            "f1_weighted": 0.900471
          }
        ],
        "main_score": 0.897556,
        "hf_subset": "default",
        "languages": [
          "eng-Latn"
        ]
      }
    ],
    "test": [
      {
        "accuracy": 0.846094,
        "f1": 0.712584,
        "f1_weighted": 0.855561,
        "scores_per_experiment": [
          {
            "accuracy": 0.869734,
            "f1": 0.724721,
            "f1_weighted": 0.881121
          },
          {
            "accuracy": 0.839264,
            "f1": 0.707667,
            "f1_weighted": 0.852215
          },
          {
            "accuracy": 0.828425,
            "f1": 0.698054,
            "f1_weighted": 0.829972
          },
          {
            "accuracy": 0.863599,
            "f1": 0.727785,
            "f1_weighted": 0.877339
          },
          {
            "accuracy": 0.829448,
            "f1": 0.704691,
            "f1_weighted": 0.83716
          }
        ],
        "main_score": 0.846094,
        "hf_subset": "default",
        "languages": [
          "eng-Latn"
        ]
      }
    ]
  },
  "evaluation_time": 1138.9256689548492,
  "kg_co2_emissions": null
}

```


author_association: CONTRIBUTOR
user: login: KennethEnevoldsen
created_at: 2025-05-05T09:47:06Z
body: Other than that I think we just needs the test to pass and then we are golden
author_association: CONTRIBUTOR