ValueError: Task image-classification is not compatible with this dataset! Available tasks: []
Author: dxlong2000
Pull request: no
Labels: name: bug

### System Info

Dear all,

Thank you for your great work. I have tried to run ```image-classification``` example on my simple dataset and I got this error. The version of transformers I used is the newest one. Do you have any idea what happened?

Thanks very much!

```WARNING:datasets.builder:Using custom data configuration default-9107176dcf18ce11
WARNING:datasets.builder:Found cached dataset imagefolder (/root/.cache/huggingface/datasets/imagefolder/default-9107176dcf18ce11/0.0.0/e872d3ec27c6c200a8881a4af52930df7eca3372b19aa4d0f5db74a2fded8141)
100% 1/1 [00:00<00:00, 45.80it/s]
Traceback (most recent call last):
  File "run_image_classification.py", line 388, in <module>
    main()
  File "run_image_classification.py", line 240, in main
    task="image-classification",
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 1713, in load_dataset
    ds = ds.prepare_for_task(task)
  File "/usr/local/lib/python3.7/dist-packages/datasets/dataset_dict.py", line 1272, in prepare_for_task
    return DatasetDict({k: dataset.prepare_for_task(task=task, id=id) for k, dataset in self.items()})
  File "/usr/local/lib/python3.7/dist-packages/datasets/dataset_dict.py", line 1272, in <dictcomp>
    return DatasetDict({k: dataset.prepare_for_task(task=task, id=id) for k, dataset in self.items()})
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 2172, in prepare_for_task
    f"Task {task} is not compatible with this dataset! Available tasks: {list(unique_values(tasks))}"
ValueError: Task image-classification is not compatible with this dataset! Available tasks: []
```

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

!CUDA_DIVISIBLE_DEVICES=0, python3 run_image_classification.py \
    --model_name_or_path facebook/convnext-tiny-224 \
    --train_dir $TRAIN_DIR \
    --output_dir $OUTPUT_DIR \
    --do_train \
    --do_eval \
    --learning_rate 1e-5 \
    --num_train_epochs 10 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 16 \
    --logging_strategy steps \
    --logging_steps 10 \
    --evaluation_strategy epoch \
    --save_strategy epoch \
    --load_best_model_at_end True \
    --save_total_limit 3 \
    --seed 1337 \
    --overwrite_output_dir 

### Expected behavior

ValueError: Task image-classification is not compatible with this dataset! Available tasks: []

Comments:
I have a similar situation. I had image classification working a couple of weeks ago with a local (disk) dataset, using run_image_classification.py, but now that code no longer works.

I run the script within a Jupyter Notebook with
```
%run Experimento_5/run_image_classification.py\
--train_dir {ruta_dataset}\
--output_dir Experimento_5/modelos/\
--remove_unused_columns False\
--do_train\
--do_eval
```

And after it loads the dataset I get:
```
Dataset imagefolder downloaded and prepared to /root/.cache/huggingface/datasets/imagefolder/default-b8c6bbfc7a1635cf/0.0.0/e872d3ec27c6c200a8881a4af52930df7eca3372b19aa4d0f5db74a2fded8141. Subsequent calls will reuse this data.
100%
1/1 [00:00<00:00, 14.38it/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File /drive/Experimento_5/run_image_classification.py:388, in <module>
    384         trainer.create_model_card(**kwargs)
    387 if __name__ == "__main__":
--> 388     main()

File /drive/Experimento_5/run_image_classification.py:236, in main()
    234     if data_args.validation_dir is not None:
    235         data_files["validation"] = os.path.join(data_args.validation_dir, "**")
--> 236     dataset = load_dataset(
    237         "imagefolder",
    238         data_files=data_files,
    239         cache_dir=model_args.cache_dir,
    240         task="image-classification",
    241     )
    243 # If we don't have a validation split, split off a percentage of train as validation.
    244 data_args.train_val_split = None if "validation" in dataset.keys() else data_args.train_val_split

File /usr/local/lib/python3.8/dist-packages/datasets/load.py:1713, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1711 # Rename and cast features to match task schema
   1712 if task is not None:
-> 1713     ds = ds.prepare_for_task(task)
   1714 if save_infos:
   1715     builder_instance._save_infos()

File /usr/local/lib/python3.8/dist-packages/datasets/dataset_dict.py:1272, in DatasetDict.prepare_for_task(self, task, id)
   1269 @is_documented_by(Dataset.prepare_for_task)
   1270 def prepare_for_task(self, task: Union[str, TaskTemplate], id: int = 0) -> "DatasetDict":
   1271     self._check_values_type()
-> 1272     return DatasetDict({k: dataset.prepare_for_task(task=task, id=id) for k, dataset in self.items()})

File /usr/local/lib/python3.8/dist-packages/datasets/dataset_dict.py:1272, in <dictcomp>(.0)
   1269 @is_documented_by(Dataset.prepare_for_task)
   1270 def prepare_for_task(self, task: Union[str, TaskTemplate], id: int = 0) -> "DatasetDict":
   1271     self._check_values_type()
-> 1272     return DatasetDict({k: dataset.prepare_for_task(task=task, id=id) for k, dataset in self.items()})

File /usr/local/lib/python3.8/dist-packages/datasets/arrow_dataset.py:2171, in Dataset.prepare_for_task(self, task, id)
   2169 compatible_templates = [template for template in (self.info.task_templates or []) if template.task == task]
   2170 if not compatible_templates:
-> 2171     raise ValueError(
   2172         f"Task {task} is not compatible with this dataset! Available tasks: {list(unique_values(tasks))}"
   2173     )
   2175 if not 0 <= id < len(compatible_templates):
   2176     templates_list_str = "\n".join(
   2177         f"- `{idx}` for task {template}" for idx, template in enumerate(compatible_templates)
   2178     )

ValueError: Task image-classification is not compatible with this dataset! Available tasks: []
```

Solved it by downgrading to datasets==2.4.0

Cc @mariosasko

> Solved it by downgrading to datasets==2.4.0

Hi,

After downgrading ```dataset```, I need to install ```evaluate``` again. I did it by ```pip install evaluate``` and I run the experiment. After doing so, I got this error:

```
[INFO|trainer.py:1628] 2022-09-29 17:35:38,675 >> ***** Running training *****
[INFO|trainer.py:1629] 2022-09-29 17:35:38,676 >>   Num examples = 5394
[INFO|trainer.py:1630] 2022-09-29 17:35:38,676 >>   Num Epochs = 10
[INFO|trainer.py:1631] 2022-09-29 17:35:38,676 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1632] 2022-09-29 17:35:38,676 >>   Total train batch size (w. parallel, distributed & accumulation) = 16
[INFO|trainer.py:1633] 2022-09-29 17:35:38,676 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1634] 2022-09-29 17:35:38,676 >>   Total optimization steps = 3380
  0% 0/3380 [00:00<?, ?it/s]Traceback (most recent call last):
  File "run_image_classification.py", line 388, in <module>
    main()
  File "run_image_classification.py", line 362, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1525, in train
    ignore_keys_for_eval=ignore_keys_for_eval,
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1737, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 721, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 2166, in __getitem__
    key,
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 2151, in _getitem
    pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 387, in format_row
    formatted_batch = self.format_batch(pa_table)
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 418, in format_batch
    return self.transform(batch)
  File "run_image_classification.py", line 315, in train_transforms
    _train_transforms(pil_img.convert("RGB")) for pil_img in example_batch["image"]
KeyError: 'image'
  0% 0/3380 [00:00<?, ?it/s]
```

Do you have any idea?

Thanks!

@NielsRogge @mariosasko 

Hi @NielsRogge,

I have tested this issue and it seems that the problem was still there:

```
0% 0/3380 [00:00<?, ?it/s]Traceback (most recent call last):
  File "run_image_classification.py", line 388, in <module>
    main()
  File "run_image_classification.py", line 362, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1504, in train
    ignore_keys_for_eval=ignore_keys_for_eval,
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1716, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 721, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 2166, in __getitem__
    key,
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 2151, in _getitem
    pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 387, in format_row
    formatted_batch = self.format_batch(pa_table)
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 418, in format_batch
    return self.transform(batch)
  File "run_image_classification.py", line 315, in train_transforms
    _train_transforms(pil_img.convert("RGB")) for pil_img in example_batch["image"]
KeyError: 'image'
  0% 0/3380 [00:00<?, ?it/s]
```

Do you have any suggestion?

Thanks!

Hi! We still need to make a patch release on the `datasets` side for my fix to take effect. In the meantime, you can install `datasets` directly from `main`:
```
pip install git+https://github.com/huggingface/datasets.git
```

When installing dataset directly from main, I got this error:

```
Traceback (most recent call last):
  File "run_image_classification.py", line 36, in <module>
    import evaluate
  File "/usr/local/lib/python3.7/dist-packages/evaluate/__init__.py", line 37, in <module>
    from .hub import push_to_hub
  File "/usr/local/lib/python3.7/dist-packages/evaluate/hub.py", line 4, in <module>
    from datasets.utils.metadata import known_task_ids
ImportError: cannot import name 'known_task_ids' from 'datasets.utils.metadata' (/usr/local/lib/python3.7/dist-packages/datasets/utils/metadata.py)
```

cc @lvwerra do we need to re-add known_task_ids to not break evaluate ? We moved those lists to the Hub, `datasets` doesn't contain any task list anymore

Is there a way we can get them from the Hub? Then we can replace this dependancy on `datasets`.

I don't think so, but the list is available here: https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts

Anyway for the next release I think we need to still have the known_task_ids variable, otherwise it makes evaluate crash.

Do you think you could fix this on the evaluate side and do a release ?
Otherwise we can also re-add this list temporarily

@dxlong2000 we just released `datasets` 2.5.2 to fix this issue ;)

Hi @lvwerra  @mariosasko,

I have installed 
```
!pip install git+https://github.com/huggingface/transformers
!pip install -r requirements.txt
!pip install datasets==2.5.2
!pip install evaluate
```

and it seems that the problem is still there:

```
[INFO|trainer.py:1613] 2022-10-10 04:21:42,170 >>   Total optimization steps = 3380
  0% 0/3380 [00:00<?, ?it/s]Traceback (most recent call last):
  File "run_image_classification.py", line 388, in <module>
    main()
  File "run_image_classification.py", line 362, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1504, in train
    ignore_keys_for_eval=ignore_keys_for_eval,
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1716, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 721, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 2229, in __getitem__
    key,
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 2214, in _getitem
    pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 387, in format_row
    formatted_batch = self.format_batch(pa_table)
  File "/usr/local/lib/python3.7/dist-packages/datasets/formatting/formatting.py", line 418, in format_batch
    return self.transform(batch)
  File "run_image_classification.py", line 315, in train_transforms
    _train_transforms(pil_img.convert("RGB")) for pil_img in example_batch["image"]
KeyError: 'image'
  0% 0/3380 [00:00<?, ?it/s]
```

Hi,

This is another error I think, not related to datasets or evaluate. Do you have an "image" column in your dataset? Did you specify `--remove_unused_columns False` when running the script?

This issue seems to persist in datasets 2.6.1

After loading my dataset from the Hub (which was created locally using split-folders and then uploaded to the hub), the dataset looks ok:

```
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 4797
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 1200
    })
})
```

, but then I get `ValueError: Task image-classification is not compatible with this dataset! Available tasks: []` when running `run_image_classification.py`

You can remove the `.prepare_for_task()` call in run_image_classification.py for now.

This is an issue with `datasets` not recognizing an existing task, feel free to open an issue on the `datasets` repo

The issue seems to persist. Is there any solution for this? `prepare_for_task()` is not called from run_image_classification.py (at least in the current version) @lhoestq 

```
File "/workdir/transformer-sparsity/examples/pytorch/image-classification/run_image_classification_no_check.py", line 431, in <module>
    task="image-classification",
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 1757, in load_dataset
    main()
  File "/workdir/transformer-sparsity/examples/pytorch/image-classification/run_image_classification_no_check.py", line 267, in main
    task="image-classification",
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 1757, in load_dataset
    ds = ds.prepare_for_task(task)
  File "/opt/conda/lib/python3.7/site-packages/datasets/dataset_dict.py", line 1278, in prepare_for_task
    return DatasetDict({k: dataset.prepare_for_task(task=task, id=id) for k, dataset in self.items()})
  File "/opt/conda/lib/python3.7/site-packages/datasets/dataset_dict.py", line 1278, in <dictcomp>
    ds = ds.prepare_for_task(task)
  File "/opt/conda/lib/python3.7/site-packages/datasets/dataset_dict.py", line 1278, in prepare_for_task
    return DatasetDict({k: dataset.prepare_for_task(task=task, id=id) for k, dataset in self.items()})
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2301, in prepare_for_task
    return DatasetDict({k: dataset.prepare_for_task(task=task, id=id) for k, dataset in self.items()})
  File "/opt/conda/lib/python3.7/site-packages/datasets/dataset_dict.py", line 1278, in <dictcomp>
    f"Task {task} is not compatible with this dataset! Available tasks: {list(unique_values(tasks))}"
ValueError: Task image-classification is not compatible with this dataset! Available tasks: []
```

@yazdanbakhsh what's your transformers + datasets version?

@NielsRogge Thanks for the message. I am using ViT + ImageNet-1K

https://huggingface.co/datasets/imagenet-1k
https://huggingface.co/google/vit-base-patch16-224

I am using head for installing HuggingFace.

Forgot to mention that I am using offline datasets. I pass the directory with arrow format. 

My other solution is to use "load_from_disk" (I am adding this option to `run_image_classifiction`). I will update the issue if it works. 

@NielsRogge I think the issue is that `self.info = self._load_info()` is not called when HF_DATASET_OFFLINE is true. 

https://github.com/huggingface/datasets/blob/232a43943e87dfedcc328a9a3d3b4d89ea5c6627/src/datasets/builder.py#L788