Repository: huggingface/huggingface_hub

Issue:
`TypeError: string indices must be integers` when iterating over models from `list_models`
### Describe the bug

Hi, It seems it is an internal error. When I iterate over the whole list (now retrieved as Python `Iterators`), it fails at some point and throws a `TypeError` exception.

Also, it outputs `Invalid model-index. Not loading eval results into CardData.` for some models, which it isn't an exception I guess. I've seen this [issue](https://github.com/nateraw/modelcards/issues/48) from nateraw/modelcards archived repo. What is the meaning of this warning? It appeared 978 times before crashing (978 from 358114 iterated models, value from `i` in the reproducible code).

### Reproduction

```
from huggingface_hub import HfApi
api = HfApi()
models = api.list_models(full=True, cardData=True, fetch_config=True, sort="lastModified", direction=-1)
i = 0
for d in models:
    i +=1
print(i)
```
This code is just to iterate over all models.

### Logs

```shell
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/test_api/testAPI.ipynb Cell 16 line 2
      1 i = 0
----> 2 for d in models:
      3     i +=1
      4 print(i)

File ~/anaconda3/envs/hf/lib/python3.9/site-packages/huggingface_hub/hf_api.py:1328, in HfApi.list_models(self, filter, author, search, emissions_thresholds, sort, direction, limit, full, cardData, fetch_config, token)
   1326 if "siblings" not in item:
   1327     item["siblings"] = None
-> 1328 model_info = ModelInfo(**item)
   1329 if emissions_thresholds is None or _is_emission_within_treshold(model_info, *emissions_thresholds):
   1330     yield model_info

File ~/anaconda3/envs/hf/lib/python3.9/site-packages/huggingface_hub/hf_api.py:527, in ModelInfo.__init__(self, **kwargs)
    524 self.mask_token = kwargs.pop("mask_token", None)
    525 card_data = kwargs.pop("cardData", None) or kwargs.pop("card_data", None)
    526 self.card_data = (
--> 527     ModelCardData(**card_data, ignore_metadata_errors=True) if isinstance(card_data, dict) else card_data
    528 )
    530 self.widget_data = kwargs.pop("widget_data", None)
    531 self.model_index = kwargs.pop("model-index", None) or kwargs.pop("model_index", None)

File ~/anaconda3/envs/hf/lib/python3.9/site-packages/huggingface_hub/repocard_data.py:305, in ModelCardData.__init__(self, language, license, library_name, tags, datasets, metrics, eval_results, model_name, ignore_metadata_errors, **kwargs)
...
--> 552     name = elem["name"]
    553     results = elem["results"]
    554     for result in results:

TypeError: string indices must be integers
```


### System info

```shell
- huggingface_hub version: 0.19.0
- Platform: Linux-6.2.16-4-pve-x86_64-with-glibc2.28
- Python version: 3.9.12
- Running in iPython ?: Yes
- iPython shell: ZMQInteractiveShell
- Running in notebook ?: Yes
- Running in Google Colab ?: No
- Token path ?: /root/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: N/A
- Jinja2: 3.1.2
- Graphviz: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: N/A
- pydantic: N/A
- aiohttp: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /root/.cache/huggingface/hub
- HF_ASSETS_CACHE: /root/.cache/huggingface/assets
- HF_TOKEN_PATH: /root/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10
```



Discussion:
I checked some things. It doesn't crash on a specific repo. When sorting (`sort="lastModified", direction=-1`) it crashes with [pidanr/pegasus-bbcnews](https://huggingface.co/pidanr/pegasus-bbcnews) repo, and when not sorting with [Waynehillsdev/Wayne_NLP_mT5](https://huggingface.co/Waynehillsdev/Wayne_NLP_mT5). 

I guess it has to do with the evaluation results values, but I'm not seeing the failure point.

Resolving patch:
diff --git a/src/huggingface_hub/repocard.py b/src/huggingface_hub/repocard.py
--- a/src/huggingface_hub/repocard.py
+++ b/src/huggingface_hub/repocard.py
@@ -1,5 +1,6 @@
 import os
 import re
+import warnings
 from pathlib import Path
 from typing import Any, Dict, Literal, Optional, Type, Union
 
@@ -21,7 +22,6 @@
 
 from .constants import REPOCARD_NAME
 from .utils import EntryNotFoundError, SoftTemporaryDirectory, validate_hf_hub_args
-from .utils.logging import get_logger
 
 
 TEMPLATE_MODELCARD_PATH = Path(__file__).parent / "templates" / "modelcard_template.md"
@@ -31,8 +31,6 @@
 # See https://github.com/huggingface/moon-landing/blob/main/server/lib/ViewMarkdown.ts#L18
 REGEX_YAML_BLOCK = re.compile(r"^(\s*---[\r\n]+)([\S\s]*?)([\r\n]+---(\r\n|\n|$))")
 
-logger = get_logger(__name__)
-
 
 class RepoCard:
     card_data_class = CardData
@@ -104,7 +102,7 @@ def content(self, content: str):
                 raise ValueError("repo card metadata block should be a dict")
         else:
             # Model card without metadata... create empty metadata
-            logger.warning("Repo card metadata block was not found. Setting CardData to empty.")
+            warnings.warn("Repo card metadata block was not found. Setting CardData to empty.")
             data_dict = {}
             self.text = content
 
diff --git a/src/huggingface_hub/repocard_data.py b/src/huggingface_hub/repocard_data.py
--- a/src/huggingface_hub/repocard_data.py
+++ b/src/huggingface_hub/repocard_data.py
@@ -1,15 +1,11 @@
 import copy
+import warnings
 from collections import defaultdict
 from dataclasses import dataclass
 from typing import Any, Dict, List, Optional, Tuple, Union
 
 from huggingface_hub.utils import yaml_dump
 
-from .utils.logging import get_logger
-
-
-logger = get_logger(__name__)
-
 
 @dataclass
 class EvalResult:
@@ -305,12 +301,12 @@ def __init__(
                 model_name, eval_results = model_index_to_eval_results(model_index)
                 self.model_name = model_name
                 self.eval_results = eval_results
-            except KeyError as error:
+            except (KeyError, TypeError) as error:
                 if ignore_metadata_errors:
-                    logger.warning("Invalid model-index. Not loading eval results into CardData.")
+                    warnings.warn("Invalid model-index. Not loading eval results into CardData.")
                 else:
                     raise ValueError(
-                        f"Invalid `model_index` in metadata cannot be parsed: KeyError {error}. Pass"
+                        f"Invalid `model_index` in metadata cannot be parsed: {error.__class__} {error}. Pass"
                         " `ignore_metadata_errors=True` to ignore this error while loading a Model Card. Warning:"
                         " some information will be lost. Use it at your own risk."
                     )


Test patch:
diff --git a/tests/test_hf_api.py b/tests/test_hf_api.py
--- a/tests/test_hf_api.py
+++ b/tests/test_hf_api.py
@@ -1685,6 +1685,48 @@ def test_model_info_with_file_metadata(self):
         assert files is not None
         self._check_siblings_metadata(files)
 
+    def test_model_info_corrupted_model_index(self) -> None:
+        """Loading model info from a model with corrupted data card should still work.
+
+        Here we use a model with a "model-index" that is not an array. Git hook should prevent this from happening
+        on the server, but models uploaded before we implemented the check might have this issue.
+
+        Example data from https://huggingface.co/Waynehillsdev/Waynehills-STT-doogie-server.
+        """
+        with self.assertWarnsRegex(UserWarning, "Invalid model-index"):
+            model = ModelInfo(
+                **{
+                    "_id": "621ffdc036468d709f1751d8",
+                    "id": "Waynehillsdev/Waynehills-STT-doogie-server",
+                    "cardData": {
+                        "license": "apache-2.0",
+                        "tags": ["generated_from_trainer"],
+                        "model-index": {"name": "Waynehills-STT-doogie-server"},
+                    },
+                    "gitalyUid": "53c57f29a007fc728c968127061b7b740dcf2b1ad401d907f703b27658559413",
+                    "likes": 0,
+                    "private": False,
+                    "config": {"architectures": ["Wav2Vec2ForCTC"], "model_type": "wav2vec2"},
+                    "downloads": 1,
+                    "tags": [
+                        "transformers",
+                        "pytorch",
+                        "tensorboard",
+                        "wav2vec2",
+                        "automatic-speech-recognition",
+                        "generated_from_trainer",
+                        "license:apache-2.0",
+                        "endpoints_compatible",
+                        "region:us",
+                    ],
+                    "pipeline_tag": "automatic-speech-recognition",
+                    "createdAt": "2022-03-02T23:29:04.000Z",
+                    "modelId": "Waynehillsdev/Waynehills-STT-doogie-server",
+                    "siblings": None,
+                }
+            )
+            self.assertIsNone(model.card_data.eval_results)
+
     def test_list_repo_files(self):
         files = self._api.list_repo_files(repo_id=DUMMY_MODEL_ID)
         expected_files = [
diff --git a/tests/test_repocard.py b/tests/test_repocard.py
--- a/tests/test_repocard.py
+++ b/tests/test_repocard.py
@@ -41,7 +41,7 @@
 from huggingface_hub.hf_api import HfApi
 from huggingface_hub.repocard import REGEX_YAML_BLOCK
 from huggingface_hub.repocard_data import CardData
-from huggingface_hub.utils import EntryNotFoundError, SoftTemporaryDirectory, is_jinja_available, logging
+from huggingface_hub.utils import EntryNotFoundError, SoftTemporaryDirectory, is_jinja_available
 
 from .testing_constants import (
     ENDPOINT_STAGING,
@@ -333,7 +333,7 @@ def test_update_existing_result_without_overwrite(self):
             metadata_update(self.repo_id, new_metadata, token=self.token, overwrite=False)
 
     def test_update_existing_field_without_overwrite(self):
-        new_datasets_data = {"datasets": "['test/test_dataset']"}
+        new_datasets_data = {"datasets": ["Open-Orca/OpenOrca"]}
         metadata_update(self.repo_id, new_datasets_data, token=self.token)
 
         with pytest.raises(
@@ -343,7 +343,7 @@ def test_update_existing_field_without_overwrite(self):
                 " Set `overwrite=True` to overwrite existing metadata."
             ),
         ):
-            new_datasets_data = {"datasets": "['test/test_dataset_2']"}
+            new_datasets_data = {"datasets": ["HuggingFaceH4/no_robots"]}
             metadata_update(self.repo_id, new_datasets_data, token=self.token, overwrite=False)
 
     def test_update_new_result_existing_dataset(self):
@@ -512,16 +512,11 @@ def test_metadata_update_missing_readme_on_space(self) -> None:
         self._api.delete_repo(self._repo_id, repo_type="space")
 
 
-class TestCaseWithCapLog(unittest.TestCase):
+class TestCaseWithHfApi(unittest.TestCase):
     _api = HfApi(endpoint=ENDPOINT_STAGING, token=TOKEN)
 
-    @pytest.fixture(autouse=True)
-    def inject_fixtures(self, caplog):
-        """Assign pytest caplog as attribute so we can use captured log messages in tests below."""
-        self.caplog = caplog
 
-
-class RepoCardTest(TestCaseWithCapLog):
+class RepoCardTest(TestCaseWithHfApi):
     def test_load_repocard_from_file(self):
         sample_path = SAMPLE_CARDS_DIR / "sample_simple.md"
         card = RepoCard.load(sample_path)
@@ -618,12 +613,8 @@ def test_repo_card_data_must_be_dict(self):
     def test_repo_card_without_metadata(self):
         sample_path = SAMPLE_CARDS_DIR / "sample_no_metadata.md"
 
-        with self.caplog.at_level(logging.WARNING):
+        with self.assertWarnsRegex(UserWarning, "Repo card metadata block was not found. Setting CardData to empty."):
             card = RepoCard(sample_path.read_text())
-        self.assertIn(
-            "Repo card metadata block was not found. Setting CardData to empty.",
-            self.caplog.text,
-        )
         self.assertEqual(card.data, CardData())
 
     def test_validate_repocard(self):
@@ -725,7 +716,7 @@ def test_does_not_match_with_leading_text(self):
         self.assertIsNone(REGEX_YAML_BLOCK.search("something\n---\nmetadata: 1\n---"))
 
 
-class ModelCardTest(TestCaseWithCapLog):
+class ModelCardTest(TestCaseWithHfApi):
     def test_model_card_with_invalid_model_index(self):
         """Test raise an error when loading a card that has invalid model-index."""
         sample_path = SAMPLE_CARDS_DIR / "sample_invalid_model_index.md"
@@ -738,12 +729,8 @@ def test_model_card_with_invalid_model_index_and_ignore_error(self):
         Some information is lost.
         """
         sample_path = SAMPLE_CARDS_DIR / "sample_invalid_model_index.md"
-        with self.caplog.at_level(logging.WARNING):
+        with self.assertWarnsRegex(UserWarning, "Invalid model-index. Not loading eval results into CardData."):
             card = ModelCard.load(sample_path, ignore_metadata_errors=True)
-        self.assertIn(
-            "Invalid model-index. Not loading eval results into CardData.",
-            self.caplog.text,
-        )
         self.assertIsNone(card.data.eval_results)
 
     def test_model_card_with_model_index(self):
@@ -828,7 +815,7 @@ def test_model_card_from_template_eval_results(self):
         self.assertEqual(str(card)[: len(DUMMY_MODELCARD_EVAL_RESULT)], DUMMY_MODELCARD_EVAL_RESULT)
 
 
-class DatasetCardTest(TestCaseWithCapLog):
+class DatasetCardTest(TestCaseWithHfApi):
     def test_load_datasetcard_from_file(self):
         sample_path = SAMPLE_CARDS_DIR / "sample_datasetcard_simple.md"
         card = DatasetCard.load(sample_path)
@@ -919,7 +906,7 @@ def test_dataset_card_from_custom_template(self):
 
 
 @with_production_testing
-class SpaceCardTest(TestCaseWithCapLog):
+class SpaceCardTest(TestCaseWithHfApi):
     def test_load_spacecard_from_hub(self) -> None:
         card = SpaceCard.load("multimodalart/dreambooth-training")
         self.assertIsInstance(card, SpaceCard)