Repository: dask/dask

Issue:
"New"/"Previous" swapped in Parquet writing error message?
I think the meaning of "New" and "Previous" are swapped in this error message:

https://github.com/dask/dask/blob/master/dask/dataframe/io/parquet.py#L412-L414

I would have expected the columns already on disk (`pf.columns`) to be labelled as "Previous", and the data about to be appended (`list(df.columns)`) to the Parquet file as "New".

Just wanted to sanity check my understanding before opening a PR to change the message.


Discussion:
cc @martindurant

On Sat, Jun 2, 2018 at 4:22 PM, Stan Seibert <notifications@github.com>
wrote:

> I think the meaning of "New" and "Previous" are swapped in this error
> message:
>
> https://github.com/dask/dask/blob/master/dask/dataframe/io/
> parquet.py#L412-L414
>
> I would have expected the columns already on disk (pf.columns) to be
> labelled as "Previous", and the data about to be appended (
> list(df.columns)) to the Parquet file as "New".
>
> Just wanted to sanity check my understanding before opening a PR to change
> the message.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3553>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszAKFKG_jRAVQSV459HBVbVrFfiOQks5t4vQTgaJpZM4UX1B0>
> .
>

Yes, you appear to be correct. 
@martindurant was this resolved?
I don't recall anything being done, but it's been a while. The relevant code does not seem to have changed since this issue.
If you get a chance can I ask you to make a small PR to solve this?  My hope is that this would take less than five minutes.
If I trust my statement above, I can just replace New and Previous, indeed quick. Should probably spend a few minutes making sure its true!

Resolving patch:
diff --git a/dask/dataframe/io/parquet.py b/dask/dataframe/io/parquet.py
--- a/dask/dataframe/io/parquet.py
+++ b/dask/dataframe/io/parquet.py
@@ -581,7 +581,7 @@ def _write_fastparquet(df, fs, fs_token, path, write_index=None, append=False,
                              'but existing file scheme is not.')
         elif ((set(pf.columns) != set(df.columns) - set(partition_on)) or (set(partition_on) != set(pf.cats))):
             raise ValueError('Appended columns not the same.\n'
-                             'New: {} | Previous: {}'
+                             'Previous: {} | New: {}'
                              .format(pf.columns, list(df.columns)))
         elif set(pf.dtypes[c] for c in pf.columns) != set(df[pf.columns].dtypes):
             raise ValueError('Appended dtypes differ.\n{}'
@@ -599,7 +599,7 @@ def _write_fastparquet(df, fs, fs_token, path, write_index=None, append=False,
             if divisions[0] < old_end:
                 raise ValueError(
                     'Appended divisions overlapping with the previous ones.\n'
-                    'New: {} | Previous: {}'.format(old_end, divisions[0]))
+                    'Previous: {} | New: {}'.format(old_end, divisions[0]))
     else:
         fmd = fastparquet.writer.make_metadata(df._meta,
                                                object_encoding=object_encoding,


Test patch: