Repository: embeddings-benchmark/mteb

Issue #1465: Can't load some datasets because of datasets 3.0
Created: 2024-11-16T19:53:11Z
When I try to download datasets, I get the error ModuleNotFoundError: No module named 'datasets.tasks' or a similar one, because in datasets version 3.0, tasks was deprecated
Tasks with errors:

IndicQARetrieval
HotelReviewSentimentClassification
TweetEmotionClassification
TenKGnadClassification
ArxivClassification
PatentClassification
FilipinoHateSpeechClassification
MyanmarNews
DutchBookReviewSentimentClassification
SwedishSentimentClassification
WisesightSentimentClassification
UrduRomanSentimentClassification
JSTS

Simple solution would be fix datasets version <3.0, but I'm not sure if it good.

Duplicate issue #1672: SwedishSentimentClassification fails
Created: 2025-01-01T17:47:54Z
Also see #1663 & #1664
2025-01-01 17:42:02.124780: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-01-01 17:42:02.136671: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-01 17:42:02.150260: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-01 17:42:02.154131: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-01 17:42:02.164452: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO:mteb.cli:Running with parameters: Namespace(model='thenlper/gte-base', task_types=None, categories=None, tasks=['SwedishSentimentClassification'], languages=None, benchmarks=None, device=None, output_folder='/data/niklas/results/results', verbosity=2, co2_tracker=True, eval_splits=None, model_revision=None, batch_size=16, overwrite=False, save_predictions=False, func=<function run at 0x7f09f9de39a0>)
WARNING:mteb.model_meta:Loader not specified for model thenlper/gte-base, loading using sentence transformers.
INFO:mteb.evaluation.MTEB:

## Evaluating 1 tasks:
─────────────────────────────── Selected tasks  ────────────────────────────────
Classification
    - SwedishSentimentClassification, s2s


INFO:mteb.evaluation.MTEB:

********************** Evaluating SwedishSentimentClassification **********************
ERROR:mteb.evaluation.MTEB:Error while evaluating SwedishSentimentClassification: No module named 'datasets.tasks'
Traceback (most recent call last):
  File "/env/lib/conda/gritkto/bin/mteb", line 8, in <module>
    sys.exit(main())
  File "/data/niklas/mteb/mteb/cli.py", line 387, in main
    args.func(args)
  File "/data/niklas/mteb/mteb/cli.py", line 145, in run
    eval.run(
  File "/data/niklas/mteb/mteb/evaluation/MTEB.py", line 623, in run
    raise e
  File "/data/niklas/mteb/mteb/evaluation/MTEB.py", line 527, in run
    task.load_data(**kwargs)
  File "/data/niklas/mteb/mteb/abstasks/AbsTask.py", line 192, in load_data
    self.dataset = datasets.load_dataset(**self.metadata_dict["dataset"])  # type: ignore
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/datasets/load.py", line 1884, in load_dataset_builder
    builder_cls = get_dataset_builder_class(dataset_module, dataset_name=dataset_name)
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/datasets/load.py", line 250, in get_dataset_builder_class
    builder_cls = import_main_class(dataset_module.module_path)
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/datasets/load.py", line 169, in import_main_class
    module = importlib.import_module(module_path)
  File "/env/lib/conda/gritkto/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/data/huggingface/modules/datasets_modules/datasets/timpal0l--swedish_reviews/b5ca53a6d7b4004000f565882a5294e9d7c9f4b341a2aa226bcf08b8762f68f5/swedish_reviews.py", line 23, in <module>
    from datasets.tasks import TextClassification
ModuleNotFoundError: No module named 'datasets.tasks'

Closing comment (2025-01-01T17:59:46Z): Duplicate of #1465. This was fixed in #1470 and datasets reuploaded in #1619 for v2.0 branch. You need to downgrade datasets version to <3.0