KeyError: 'tags' arised while huggingface-hub==0.23.4,transformers==4.41.2
Author: HarikrishnanK9
Pull request: no
Labels: name: bug

### System Info

@ArthurZucker @Rocketknight1 @stevhliu 

My code worked perfectly when huggingface-hub==0.27.0,transformers==4.47.1 but failed when huggingface-hub==0.23.4,transformers==4.41.2(Experimented with 2 different environments,one is production environment which caused this issue,another one is a new one)
Actually there where no issues previously.It will be much helpful if someone share a solution without upgrading the versions,as if i change any versions there it can affect the rest of the codes.In code tried model is llam3 7b instruct and a locally finetuned model inorder to generate response from llm using pipeline,the dataset wa in a private repository and tried to utilize it using from datasets import load_dataset

python version = 3.10.12,ubuntu 22.04

Detailed Error:
Traceback (most recent call last):
  File "/home/paperspace/product/backend/dataingestion/test_ingestion.py", line 245, in process_llm_dataset
    output_file_path, processed_df = Benchmark.prepare_and_generate_responses(request)
  File "/home/paperspace/product/backend/dataingestion/test_ingestion.py", line 146, in prepare_and_generate_responses
    dataset = load_dataset(
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/datasets/load.py", line 1849, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/datasets/load.py", line 1564, in dataset_module_factory
    ).get_module()
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/datasets/load.py", line 944, in get_module
    data_files = DataFilesDict.from_patterns(
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/datasets/data_files.py", line 721, in from_patterns
    else DataFilesList.from_patterns(
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/datasets/data_files.py", line 624, in from_patterns
    resolve_pattern(
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/datasets/data_files.py", line 388, in resolve_pattern
    for filepath, info in fs.glob(pattern, detail=True, **glob_kwargs).items()
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 406, in glob
    path = self.resolve_path(path, revision=kwargs.get("revision")).unresolve()
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 179, in resolve_path
    repo_and_revision_exist, err = self._repo_and_revision_exist(repo_type, repo_id, revision)
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 127, in _repo_and_revision_exist
    self._api.repo_info(repo_id, revision=revision, repo_type=repo_type, timeout=HF_HUB_ETAG_TIMEOUT)
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2491, in repo_info
    return method(
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2366, in dataset_info
    return DatasetInfo(**data)
  File "/home/paperspace/anaconda3/envs/eval_env/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 799, in __init__
    self.tags = kwargs.pop("tags")
KeyError: 'tags'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/paperspace/product/backend/dataingestion/test_ingestion.py", line 261, in <module>
    process_llm_dataset(request)
  File "/home/paperspace/product/backend/dataingestion/test_ingestion.py", line 250, in process_llm_dataset
    raise HTTPException(status_code=500, detail=str(e))
fastapi.exceptions.HTTPException: 500: 'tags'

### Who can help?

@ArthurZucker @Rocketknight1 @stevhliu 


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

use keivalya/MedQuad-MedicalQnADataset from huggingface using " from datasets import load_dataset" pass the question part only to the llm(llama3 7 b instruct or any finetuned form of it) inorder to generate answer.But remember one thing in my case i have transformed that dataset by extracting the question and answer and saved it as a jsonl file with structure like this {"question": "What is the capital of France?","answer": "The capital of France is Paris."} this transformed dataset is actually loaded using from datasets import load_dataset;then utilizing from transformers import pipeline inorder to use the llama 3 7b instruct.


### Expected behavior

Expecting the code properly works properly not only in[ huggingface-hub==0.27.0,transformers==4.47.1] but also in [huggingface-hub==0.23.4,transformers==4.41.2]

Comments:
Hi @HarikrishnanK9, we can't really fix issues that depend on interactions between older versions! In general, the `transformers` and `huggingface_hub` packages expect each other to be relatively up-to-date.

If you're still getting issues like this with the latest versions of both then feel free to reopen the issue, but we can't really help if you're stuck on older versions.

Thank You @Rocketknight1