Cleaner error tracebacks for dataset script errors
Opened by mariosasko (closed)

Make the traceback of the errors raised in `_generate_examples` cleaner for easier debugging. Additionally, initialize the `writer` in the for-loop to avoid the `ValueError` from `ArrowWriter.finalize` raised in the `finally` block when no examples are yielded before the `_generate_examples` error.

<details>
    <summary>
     The full traceback of the "SQLAlchemy ImportError" error that gets printed with these changes:
    </summary>

```bash
ImportError                               Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _prepare_split_single(self, arg)
   1759             _time = time.time()
-> 1760             for _, table in generator:
   1761                 # Only initialize the writer when we have the first record (to avoid having to do the clean-up if an error occurs before that)

9 frames
/usr/local/lib/python3.7/dist-packages/datasets/packaged_modules/sql/sql.py in _generate_tables(self)
    112         sql_reader = pd.read_sql(
--> 113             self.config.sql, self.config.con, chunksize=chunksize, **self.config.pd_read_sql_kwargs
    114         )

/usr/local/lib/python3.7/dist-packages/pandas/io/sql.py in read_sql(sql, con, index_col, coerce_float, params, parse_dates, columns, chunksize)
    598     """
--> 599     pandas_sql = pandasSQL_builder(con)
    600 

/usr/local/lib/python3.7/dist-packages/pandas/io/sql.py in pandasSQL_builder(con, schema, meta, is_cursor)
    789     elif isinstance(con, str):
--> 790         raise ImportError("Using URI string without sqlalchemy installed.")
    791     else:

ImportError: Using URI string without sqlalchemy installed.

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
<ipython-input-4-5af11af4737b> in <module>
----> 1 ds = Dataset.from_sql('''SELECT * from states WHERE state=="New York";''', "sqlite:///us_covid_data.db")

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in from_sql(sql, con, features, cache_dir, keep_in_memory, **kwargs)
   1152             cache_dir=cache_dir,
   1153             keep_in_memory=keep_in_memory,
-> 1154             **kwargs,
   1155         ).read()
   1156 

/usr/local/lib/python3.7/dist-packages/datasets/io/sql.py in read(self)
     47             # try_from_hf_gcs=try_from_hf_gcs,
     48             base_path=base_path,
---> 49             use_auth_token=use_auth_token,
     50         )
     51 

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    825                             verify_infos=verify_infos,
    826                             **prepare_split_kwargs,
--> 827                             **download_and_prepare_kwargs,
    828                         )
    829                     # Sync info

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    912             try:
    913                 # Prepare split will record examples associated to the split
--> 914                 self._prepare_split(split_generator, **prepare_split_kwargs)
    915             except OSError as e:
    916                 raise OSError(

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _prepare_split(self, split_generator, file_format, num_proc, max_shard_size)
   1652             job_id = 0
   1653             for job_id, done, content in self._prepare_split_single(
-> 1654                 {"gen_kwargs": gen_kwargs, "job_id": job_id, **_prepare_split_args}
   1655             ):
   1656                 if done:

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _prepare_split_single(self, arg)
   1789             raise DatasetGenerationError(
   1790                 f"An error occured while generating the dataset"
-> 1791             ) from e
   1792         finally:
   1793             yield job_id, False, num_examples_progress_update

DatasetGenerationError: An error occurred while generating the dataset
```
</details>


PS: I've also considered raising the error as follows:
```python
tb = sys.exc_info()[2]
raise DatasetGenerationError(f"An error occurred while generating the dataset: {type(e).__name__}: {e}").with_traceback(tb) from None # this raises the DatasetGenerationError with "e"'s traceback
```
But it seems like "from e" is now the [preferred](https://docs.python.org/3/library/exceptions.html#BaseException.with_traceback) way to chain exceptions.

Fix https://github.com/huggingface/datasets/issues/5186

cc @nateraw 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lhoestq Good catch! This currently leads to an AttributeError (due to `writer` being None) on this line:
https://github.com/huggingface/datasets/blob/fed1628d49a91f9ae259ddf6edbb252c7972d9a3/src/datasets/builder.py#L1552