GitHub issue #6209: CI is broken with AssertionError: 3 failed, 12 errors
State: closed | Created: 2023-09-04T06:47:05Z

Our CI is broken: 3 failed, 12 errors

See: https://github.com/huggingface/datasets/actions/runs/6069947111/job/16465138041
```
=========================== short test summary info ============================
FAILED tests/test_load.py::ModuleFactoryTest::test_LocalDatasetModuleFactoryWithoutScript_with_data_dir - AssertionError: assert ({NamedSplit('train'): ['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/train.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/train.txt'], NamedSplit('test'): ['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/test.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/test.txt']} is not None and 2 == 1)
 +  where 2 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/train.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/train.txt'])
FAILED tests/test_load.py::test_load_dataset_arrow[False] - AssertionError: assert 20 == 10
 +  where 20 = Dataset({\n    features: ['col_1'],\n    num_rows: 20\n}).num_rows
FAILED tests/test_load.py::test_load_dataset_arrow[True] - assert 20 == 10
ERROR tests/packaged_modules/test_audiofolder.py::test_data_files_with_metadata_and_multiple_splits[jsonl-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_audiofolder.py::test_data_files_with_metadata_and_multiple_splits[jsonl-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_audiofolder.py::test_data_files_with_metadata_and_multiple_splits[csv-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/metadata.csv', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/metadata.csv'])
ERROR tests/packaged_modules/test_audiofolder.py::test_data_files_with_metadata_and_multiple_splits[csv-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/metadata.csv', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/metadata.csv'])
ERROR tests/packaged_modules/test_folder_based_builder.py::test_data_files_with_metadata_and_splits[1-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_folder_based_builder.py::test_data_files_with_metadata_and_splits[1-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_folder_based_builder.py::test_data_files_with_metadata_and_splits[2-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_imagefolder.py::test_data_files_with_metadata_and_multiple_splits[jsonl-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_imagefolder.py::test_data_files_with_metadata_and_multiple_splits[jsonl-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_folder_based_builder.py::test_data_files_with_metadata_and_splits[2-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_imagefolder.py::test_data_files_with_metadata_and_multiple_splits[csv-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/metadata.csv', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/metadata.csv'])
ERROR tests/packaged_modules/test_imagefolder.py::test_data_files_with_metadata_and_multiple_splits[csv-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/metadata.csv', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/metadata.csv'])
= 3 failed, 2383 passed, 26 skipped, 9 warnings, 12 errors in 280.79s (0:04:40) =
```