Repository: huggingface/dataset-viewer

Issue:
FineWeb: Unexpected end of stream: Page was smaller (1862094) than expected (2055611)
The config-parquet-metadata job succeeds but the split-first-rows job fails when using `compute_first_rows_from_parquet_response`.

In the meantime I set the error code in the config-parquet-metadata response as `CachedResponseNotFound` to make the split-first-rows succeed

This workaround causes `ResponseNotFound` when opening page 2 in the viewer unfortunately (can't do random access in the parquet data without a valid config-parquet-metadata response)


Discussion:
The same issue occurs for https://huggingface.co/datasets/prometheus-eval/Feedback-Collection/discussions/4

```
OSError: Unexpected end of stream: Page was smaller (37332) than expected (542633)
```
The traceback (not necessarily in the right order!)

```
INFO: 2024-05-03 11:34:08,433 - root - /rows, dataset='prometheus-eval/Feedback-Collection', config='default', split='train', offset=0, length=100
INFO: 2024-05-03 11:34:08,436 - root - Create ParquetIndexWithMetadata for dataset=prometheus-eval/Feedback-Collection, config=default, split=train
INFO: 2024-05-03 11:34:08,436 - root - Query ParquetIndexWithMetadata for dataset=prometheus-eval/Feedback-Collection, config=default, split=train, offset=0, length=100
  File ""/src/services/rows/src/rows/routes/rows.py"", line 101, in rows_endpoint
  File ""pyarrow/_parquet.pyx"", line 1388, in pyarrow._parquet.ParquetReader.read_row_group
  File ""pyarrow/_parquet.pyx"", line 1418, in pyarrow._parquet.ParquetReader.read_row_groups
  File ""pyarrow/error.pxi"", line 91, in pyarrow.lib.check_status
ERROR: 2024-05-03 11:34:08,590 - root - Unexpected error.
  File ""/src/libs/libcommon/src/libcommon/parquet_utils.py"", line 577, in query
  File ""/src/libs/libcommon/src/libcommon/parquet_utils.py"", line 429, in query
    return self.parquet_file.read_row_group(i=self.group_id, columns=columns)
  File ""/src/services/rows/.venv/lib/python3.9/site-packages/pyarrow/parquet/core.py"", line 490, in read_row_group
Traceback (most recent call last):
    pa_table = rows_index.query(offset=offset, length=length)
    return self.parquet_index.query(offset=offset, length=length)
    [
        row_group_readers[i].read(self.supported_columns)
  File ""/src/libs/libcommon/src/libcommon/parquet_utils.py"", line 140, in read
OSError: Unexpected end of stream: Page was smaller (37332) than expected (542633)
  File ""/src/libs/libcommon/src/libcommon/parquet_utils.py"", line 430, in <listcomp>
    return self.reader.read_row_group(i, column_indices=column_indices,
INFO:     10.0.26.51:34806 - ""GET /rows?dataset=prometheus-eval%2FFeedback-Collection&config=default&split=train&offset=0&length=100 HTTP/1.1"" 500 Internal Server Error
```
I recreated the dataset viewer for https://huggingface.co/datasets/prometheus-eval/Feedback-Collection, and it works well now. So, I imagine the issue is in the created Parquet files (or in the metadata file).

Note that before recreating, all the cache entries were 200 - OK
For fineweb, the issue is only for the `default` config (the biggest one), as I was able to browse the pages of any other config.

I just tried to recompute `config-parquet-metadata` for this config, but it crashed. Trying again

<img width="1047" alt="Capture d’écran 2024-05-03 à 14 14 28" src="https://github.com/huggingface/dataset-viewer/assets/1676121/0c816aa7-2c2c-49cf-9583-9c9b6fd75ded">

I think the issue is that we create more than 10k parquet (metadata) files, and thus the parquet file number 10000 is named 0000.parquet, and overwrites the existing one.

<img width="1308" alt="Capture d’écran 2024-05-03 à 14 21 22" src="https://github.com/huggingface/dataset-viewer/assets/1676121/461dd0c6-9ba4-4264-947a-7062a7ed8679">

Possibly we have to apply the same technique as https://github.com/huggingface/dataset-viewer/pull/2503 for the parquet metadata files
Fineweb has more than 20k parquet files: https://huggingface.co/datasets/HuggingFaceFW/fineweb/tree/refs%2Fconvert%2Fparquet/default

<img width="680" alt="Capture d’écran 2024-05-03 à 14 29 23" src="https://github.com/huggingface/dataset-viewer/assets/1676121/17bca9f0-2b0b-41cc-9c61-7296f36254d9">

preparing a PR

Resolving patch:
diff --git a/services/worker/src/worker/job_runners/config/parquet_metadata.py b/services/worker/src/worker/job_runners/config/parquet_metadata.py
--- a/services/worker/src/worker/job_runners/config/parquet_metadata.py
+++ b/services/worker/src/worker/job_runners/config/parquet_metadata.py
@@ -34,10 +34,14 @@ def create_parquet_metadata_file_from_remote_parquet(
         parquet_file = get_parquet_file(url=parquet_file_item["url"], fs=fs, hf_token=hf_token)
     except Exception as e:
         raise FileSystemError(f"Could not read the parquet files: {e}") from e
+    split = parquet_file_item["url"].split("/")[-2]
+    # ^ https://github.com/huggingface/dataset-viewer/issues/2768
+    # to support more than 10k parquet files, in which case, instead of "train" for example,
+    # the subdirectories are "train-part0", "train-part1", "train-part2", etc.
     parquet_metadata_subpath = create_parquet_metadata_file(
         dataset=parquet_file_item["dataset"],
         config=parquet_file_item["config"],
-        split=parquet_file_item["split"],
+        split=split,
         parquet_file_metadata=parquet_file.metadata,
         filename=parquet_file_item["filename"],
         parquet_metadata_directory=parquet_metadata_directory,


Test patch:
diff --git a/services/worker/tests/job_runners/config/test_parquet_metadata.py b/services/worker/tests/job_runners/config/test_parquet_metadata.py
--- a/services/worker/tests/job_runners/config/test_parquet_metadata.py
+++ b/services/worker/tests/job_runners/config/test_parquet_metadata.py
@@ -95,10 +95,20 @@ def _get_job_runner(
             ConfigParquetResponse(
                 parquet_files=[
                     SplitHubFile(
-                        dataset="ok", config="config_1", split="train", url="url1", filename="filename1", size=0
+                        dataset="ok",
+                        config="config_1",
+                        split="train",
+                        url="https://url1/train/0000.parquet",
+                        filename="filename1",
+                        size=0,
                     ),
                     SplitHubFile(
-                        dataset="ok", config="config_1", split="train", url="url2", filename="filename2", size=0
+                        dataset="ok",
+                        config="config_1",
+                        split="train",
+                        url="https://url1/train/0001.parquet",
+                        filename="filename2",
+                        size=0,
                     ),
                 ],
                 partial=False,
@@ -111,7 +121,7 @@ def _get_job_runner(
                         dataset="ok",
                         config="config_1",
                         split="train",
-                        url="url1",
+                        url="https://url1/train/0000.parquet",
                         filename="filename1",
                         size=0,
                         num_rows=3,
@@ -121,7 +131,7 @@ def _get_job_runner(
                         dataset="ok",
                         config="config_1",
                         split="train",
-                        url="url2",
+                        url="https://url1/train/0001.parquet",
                         filename="filename2",
                         size=0,
                         num_rows=3,
@@ -161,7 +171,7 @@ def _get_job_runner(
                         dataset="with_features",
                         config="config_1",
                         split="train",
-                        url="url1",
+                        url="https://url1/train/0000.parquet",
                         filename="filename1",
                         size=0,
                     ),
@@ -169,7 +179,7 @@ def _get_job_runner(
                         dataset="with_features",
                         config="config_1",
                         split="train",
-                        url="url2",
+                        url="https://url1/train/0001.parquet",
                         filename="filename2",
                         size=0,
                     ),
@@ -184,7 +194,7 @@ def _get_job_runner(
                         dataset="with_features",
                         config="config_1",
                         split="train",
-                        url="url1",
+                        url="https://url1/train/0000.parquet",
                         filename="filename1",
                         size=0,
                         num_rows=3,
@@ -194,7 +204,7 @@ def _get_job_runner(
                         dataset="with_features",
                         config="config_1",
                         split="train",
-                        url="url2",
+                        url="https://url1/train/0001.parquet",
                         filename="filename2",
                         size=0,
                         num_rows=3,
@@ -206,6 +216,61 @@ def _get_job_runner(
             ),
             False,
         ),
+        (
+            "more_than_10k_files",
+            "config_1",
+            HTTPStatus.OK,
+            ConfigParquetResponse(
+                parquet_files=[
+                    SplitHubFile(
+                        dataset="with_features",
+                        config="config_1",
+                        split="train",
+                        url="https://url1/train-part0/0000.parquet",
+                        filename="filename1",
+                        size=0,
+                    ),
+                    SplitHubFile(
+                        dataset="with_features",
+                        config="config_1",
+                        split="train",
+                        url="https://url1/train-part1/0000.parquet",
+                        filename="filename2",
+                        size=0,
+                    ),
+                ],
+                partial=False,
+                features=Features({"a": Value("string")}).to_dict(),
+            ),
+            None,
+            ConfigParquetMetadataResponse(
+                parquet_files_metadata=[
+                    ParquetFileMetadataItem(
+                        dataset="with_features",
+                        config="config_1",
+                        split="train",
+                        url="https://url1/train-part0/0000.parquet",
+                        filename="filename1",
+                        size=0,
+                        num_rows=3,
+                        parquet_metadata_subpath="with_features/--/config_1/train-part0/filename1",
+                    ),
+                    ParquetFileMetadataItem(
+                        dataset="with_features",
+                        config="config_1",
+                        split="train",
+                        url="https://url1/train-part1/0000.parquet",
+                        filename="filename2",
+                        size=0,
+                        num_rows=3,
+                        parquet_metadata_subpath="with_features/--/config_1/train-part1/filename2",
+                    ),
+                ],
+                partial=False,
+                features=Features({"a": Value("string")}).to_dict(),
+            ),
+            False,
+        ),
     ],
 )
 def test_compute(