Compression kwargs have no effect when saving datasets as csv
Opened by exs-avianello (open)

### Describe the bug

Attempting to save a dataset as a compressed csv file, the compression kwargs provided to `.to_csv()` that get piped to panda's `pandas.DataFrame.to_csv` do not have any effect - resulting in the dataset not getting compressed.

A warning is raised if explicitly providing a `compression` kwarg, but no warnings are raised if relying on the defaults. This can lead to datasets secretly not getting compressed for users expecting the behaviour to match panda's `.to_csv()`, where the compression format is automatically inferred from the destination path suffix.

### Steps to reproduce the bug

```python
# dataset is not compressed (but at least a warning is emitted)

import datasets

dataset = datasets.load_dataset("rotten_tomatoes", split="train")

dataset.to_csv("uncompressed.csv")
print(os.path.getsize("uncompressed.csv"))  # 1008607

dataset.to_csv("compressed.csv.gz",  compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1})
print(os.path.getsize("compressed.csv.gz"))  # 1008607
```

```shell
>>>
RuntimeWarning: compression has no effect when passing a non-binary object as input.
  csv_str = batch.to_pandas().to_csv(
```

```python
# dataset is not compressed and no warnings are emitted

dataset.to_csv("compressed.csv.gz")
print(os.path.getsize("compressed.csv.gz"))  # 1008607

# compare with
dataset.to_pandas().to_csv("pandas.csv.gz")
print(os.path.getsize("pandas.csv.gz"))  # 418561
```

---

I think that this is because behind the scenes `pandas.DataFrame.to_csv` is always called with a buf-like `path_or_buf`, but users that are providing a path-like to `datasets.Dataset.to_csv` are likely not to expect / know that - leading to a mismatch in their understanding of the expected behaviour of the `compression` kwarg.

### Expected behavior

The dataset to be saved as a compressed csv file when providing a `compression` kwarg, or when relying on the default `compression='infer'`

### Environment info

`datasets == 2.13.1`


Comments:
Hello @exs-avianello, I have reproduced the bug successfully and have understood the problem. But I am confused regarding this part of the statement, "`pandas.DataFrame.to_csv` is always called with a buf-like `path_or_buf`".

Can you please elaborate on it?

Thanks!

Hi @aryanxk02 ! Sure, what I actually meant is that when passing a path-like `path_or_buf` here

https://github.com/huggingface/datasets/blob/14f6edd9222e577dccb962ed5338b79b73502fa5/src/datasets/arrow_dataset.py#L4708-L4714 

it gets converted to a file object behind the scenes here

https://github.com/huggingface/datasets/blob/14f6edd9222e577dccb962ed5338b79b73502fa5/src/datasets/io/csv.py#L92-L94

and the eventual pandas `.to_csv()` calls that write to it always get `path_or_buf=None`, making pandas ignore the `compression` kwarg in the `to_csv_kwargs`

https://github.com/huggingface/datasets/blob/14f6edd9222e577dccb962ed5338b79b73502fa5/src/datasets/io/csv.py#L107-L109

@exs-avianello When `path_or_buf` is set to None, the `to_csv()` method will return the CSV data as a string instead of saving it to a file. Hence the compression doesn't take place. I think setting `path_or_buf=self.path_or_buf` should work. What you say?