Repository: NVIDIA-Merlin/NVTabular

Issue:
[BUG] Reading data from GCS creates issue
**Describe the bug**
Reading parquet file from Google Cloud Storage does not work.

**Steps/Code to reproduce bug**
```
dataset = nvt.Dataset("gs://bucket/file.parquet")
dataset.to_ddf().head()
```

Error:
```
cuDF failure at: ../src/table/table.cpp:42: Column size mismatch:
```

If the data is copied to the local disk, the code will work.
cuDF / dask_cudf can read from GCS.
This is with the latest NVTabular



Discussion:
@bschifferer - I am having trouble reproducing this.  However, I did run into another error related to the local buffer needing to be wrapped in a BytesIO object in order to interact with pyarrow correctly. Perhaps the errors are related?

When you say "cuDF / dask_cudf can read from GCS", are you referring to the latest version of RAPIDS, or 21.08?
@rjzamora 

As @bschifferer said, this fails 
```
dataset = nvt.Dataset("gs://bucket/file.parquet")
dataset.to_ddf().head()
```

but this seems to work fine (feel a lot slower)
```
ddf = dask_cudf.read_parquet("/path-to-data/*.parquet")
nvt.Dataset(ddf)
```

So this still doesn't work with the latest development branch (`main`)? Is the problematic file in a public bucket? If not, can you share a toy DataFrame example that cannot be read back like this after being written to gcs?
I haven't tried with the latest `main` branch. I'll check if #1158 fixed the issue and provide an update. 
>I haven't tried with the latest main branch. I'll check if #1158 fixed the issue and provide an update.

Thank @pchandar ! Note that I am not very confident that #1158 was related, but since I cannot reproduce with an arbitrary parquet file myself, this issue is a bit difficult to debug.
@rjzamora  sorry it took a while. It was a bit tricky to reproduce this on a test dataset. But if you copy the transformed parquet from [this](https://github.com/NVIDIA-Merlin/NVTabular/blob/main/examples/getting-started-movielens/02-ETL-with-NVTabular.ipynb) (Cell 18)  example to a GCS bucket and then 

```
ds = nvt.Dataset("gs://bucket/movielens.parquet")
ds.head()
```
will give the following error 
```
RuntimeError: cuDF failure at: ../src/table/table.cpp:42: Column size mismatch: 76 != 20000076
```
A couple of observations: (1) this seem to happen only when the list columns exists; and (2) for sufficiently large datasets (when I tried slicing the problematic dataset it seemed to work fine). Hope this help reproduce the error at your end. 
Thanks

I'm sorry for the delay here. This is indeed a bug in the optimized data-transfer logic for read_parquet from remote storage. It turns out that that the list column name is modified from "genres" to "genres.list.element" in the parquet metadata, and so we fail to transfer the data for that column.  In the near future, [all this logic will live directly in fsspec](https://github.com/intake/filesystem_spec/pull/806) (and will be removed from NVTabular), but I will submit a temporary fix asap for NVT.

Resolving patch:
diff --git a/nvtabular/io/fsspec_utils.py b/nvtabular/io/fsspec_utils.py
--- a/nvtabular/io/fsspec_utils.py
+++ b/nvtabular/io/fsspec_utils.py
@@ -179,13 +179,20 @@ def _get_parquet_byte_ranges(
             for c in range(row_group.num_columns):
                 column = row_group.column(c)
                 name = column.path_in_schema
-                # Skip this column if we are targeting a
-                # specific columns
-                if columns is None or name in columns:
+                # Skip this column if we are targeting
+                # specific columns, and this name is not
+                # in the list.
+                #
+                # Note that `column.path_in_schema` may
+                # modify the column name for list and struct
+                # columns. For example, a column named "a"
+                # may become "a.list.element"
+                split_name = name.split(".")[0]
+                if columns is None or name in columns or split_name in columns:
                     file_offset0 = column.dictionary_page_offset
                     if file_offset0 is None:
                         file_offset0 = column.data_page_offset
-                    num_bytes = column.total_uncompressed_size
+                    num_bytes = column.total_compressed_size
                     byte_ranges.append((file_offset0, num_bytes))
 
     return byte_ranges, footer_sample, file_size


Test patch: