Encountered a Program Bug during Tokenization, Error Message: OverflowError: There was an overflow with type <class 'list'>. Try to reduce writer_batch_size to have batches smaller than 2GB.
Hello everyone,
During the process of tokenizing a dataset of approximately one million cells, I encountered the following error. I suspect that it might be due to an excessively large batch size or insufficient memory. The specific error message is as follows:
Traceback (most recent call last):
File "1.token.py", line 8, in
tk.tokenize_data(loom_data_directory, output_directory, output_prefix)
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/geneformer/tokenizer.py", line 102, in tokenize_data
tokenized_dataset = self.create_dataset(tokenized_cells, cell_metadata)
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/geneformer/tokenizer.py", line 204, in create_dataset
output_dataset = Dataset.from_dict(dataset_dict)
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 911, in from_dict
pa_table = InMemoryTable.from_pydict(mapping=mapping)
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/table.py", line 799, in from_pydict
return cls(pa.Table.from_pydict(*args, **kwargs))
File "pyarrow/table.pxi", line 3849, in pyarrow.lib.Table.from_pydict
File "pyarrow/table.pxi", line 5401, in pyarrow.lib._from_pydict
File "pyarrow/array.pxi", line 357, in pyarrow.lib.asarray
File "pyarrow/array.pxi", line 243, in pyarrow.lib.array
File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/arrow_writer.py", line 245, in arrow_array
raise OverflowError(
OverflowError: There was an overflow with type <class 'list'>. Try to reduce writer_batch_size to have batches smaller than 2GB.
(offset overflow while concatenating arrays)
Does anyone know what could be causing the error? I would greatly appreciate any assistance.
Thank you for your interest in Geneformer! This is likely due to a known issue with Huggingface Datasets that they are working on resolving. Please see PR 170 for more information as well as a possible solution using Dataset.from_generator. We are working on resolving a couple of errors and determining a faster solution with this PR before merging it but you could substitute the Dataset.from_generator instead of Dataset.from_dict for an intermediate solution (it is slower but seems to effectively resolve the overflow error encountered by other users with large datasets). Because we do not encounter this error, even when tokenizing datasets >1M, we cannot very effectively troubleshoot it. If you are able to email me a link to your dataset that caused the error, that would be very helpful so we can ensure any faster solution we put in place would truly resolve the error.
Hi,ctheodoris，
Even after modifying the Dataset.from_generator function, the following error still occurs：
HF google storage unreachable. Downloading and preparing it from source
Generating train split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/builder.py", line 1676, in _prepare_split_single
for key, record in generator:
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/packaged_modules/generator/generator.py", line 30, in _generate_examples
for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
TypeError: 'dict' object is not callable
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "1.token.py", line 8, in
tk.tokenize_data(loom_data_directory, output_directory, output_prefix)
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/geneformer/tokenizer.py", line 102, in tokenize_data
tokenized_dataset = self.create_dataset(tokenized_cells, cell_metadata)
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/geneformer/tokenizer.py", line 206, in create_dataset
output_dataset=Dataset.from_generator(dataset_dict)
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1064, in from_generator
return GeneratorDatasetInputStream(
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/io/generator.py", line 47, in read
self.builder.download_and_prepare(
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/builder.py", line 954, in download_and_prepare
self._download_and_prepare(
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/builder.py", line 1717, in _download_and_prepare
super()._download_and_prepare(
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/builder.py", line 1555, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File "/jdfssz1/ST_HEALTH/P20Z10200N0015/chenxuan/wzw/cancer/tools/anaconda3/envs/geneformer/lib/python3.8/site-packages/datasets/builder.py", line 1712, in _prepare_split_single
raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
We have decided to send you an email to resolve this issue. thank you for your patience！