Repository: dask/dask

Issue:
to_parquet default compression is `None` not `snappy`
The pandas to_parquet method defaults to using `snappy` for compression.
However, when you don't specify compression in dask, the data is saved uncompressed.


Discussion:
`None` was the original in fastparquet, since snappy is a dependency we didn't want to force onto people; I believe if you give None with the arrow backend, you get snappy. arrow persuaded pandas otherwise, since snappy is the default in, for example, spark (but not everywhere!). I am fine with having different defaults in different places.
I guess I was caught unawares. I wrote some pandas code, it all worked as expected, I switched it to dask and now my parquet files are huge. 

I wonder if in cases where dask intentionally, for good reasons, doesn't match the pandas API it could be highlighted in some way. 

I do see that the docstrings are different and that pandas uses snappy by default, but it's easy, as a newcomer to not fully understand the implications of the slightly different docstrings - or to not have checked them at all because the method just works....

This is pretty vague so I'm more than fine with closing this issue if there's not clear next step.
We usually conform to Pandas standards when there is no strong reason not
to do so.  Deviations from this seem to cause surprise in users.

@martindurant what are your thoughts on defaulting to snappy if snappy is
present?  Were there reasons to not do this or is this just historical?

On Fri, Mar 16, 2018 at 3:29 PM, Sarah Bird <notifications@github.com>
wrote:

> I guess I was caught unawares. I wrote some pandas code, it all worked as
> expected, I switched it to dask and now my parquet files are huge.
>
> I wonder if in cases where dask intentionally, for good reasons, doesn't
> match the pandas API it could be highlighted in some way.
>
> I do see that the docstrings are different and that pandas uses snappy by
> default, but it's easy, as a newcomer to not fully understand the
> implications of the slightly different docstrings - or to not have checked
> them at all because the method just works....
>
> This is pretty vague so I'm more than fine with closing this issue if
> there's not clear next step.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3288#issuecomment-373821080>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszA196JYj-hLMSSfDHRaiJKnOaLKcks5tfBKlgaJpZM4SrYjq>
> .
>

Indeed historical. The "if snappy is present" will obviously require a little logic similar to that which exists elsewhere.

Resolving patch:
diff --git a/dask/dataframe/io/parquet.py b/dask/dataframe/io/parquet.py
--- a/dask/dataframe/io/parquet.py
+++ b/dask/dataframe/io/parquet.py
@@ -12,6 +12,7 @@
 from ..core import DataFrame, Series
 from ..utils import (clear_known_categories, strip_unknown_categories,
                      UNKNOWN_CATEGORIES)
+from ...bytes.compression import compress
 from ...base import tokenize
 from ...compatibility import PY3, string_types
 from ...delayed import delayed
@@ -888,6 +889,8 @@ def to_parquet(df, path, engine='auto', compression='default', write_index=None,
 
     if compression != 'default':
         kwargs['compression'] = compression
+    elif 'snappy' in compress:
+        kwargs['compression'] = 'snappy'
 
     write = get_engine(engine)['write']
 


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -776,7 +776,7 @@ def test_columns_name(tmpdir, write_engine, read_engine):
 def test_writing_parquet_with_compression(tmpdir, compression, engine):
     fn = str(tmpdir)
 
-    if engine == 'fastparquet' and compression == 'snappy':
+    if engine == 'fastparquet' and compression in ['snappy', 'default']:
         pytest.importorskip('snappy')
 
     df = pd.DataFrame({'x': ['a', 'b', 'c'] * 10,
@@ -784,6 +784,12 @@ def test_writing_parquet_with_compression(tmpdir, compression, engine):
     ddf = dd.from_pandas(df, npartitions=3)
 
     ddf.to_parquet(fn, compression=compression, engine=engine)
+    if engine == 'fastparquet' and compression == 'default':
+        # ensure default compression for fastparquet is Snappy
+        import fastparquet
+        pf = fastparquet.ParquetFile(fn)
+        assert pf.row_groups[0].columns[0].meta_data.codec == 1
+
     out = dd.read_parquet(fn, engine=engine)
     assert_eq(out, df, check_index=(engine != 'fastparquet'))