Repository: NVIDIA-Merlin/NVTabular

Issue:
[BUG] Rossmann notebook not hitting expected RMSPE (again)
**Describe the bug**

Rossmann convergence and final RMSPEs are again considerably worse than they once were.

I'm separating this out from https://github.com/NVIDIA/NVTabular/issues/146 because that issue was resolved by @rjzamora  's PR (I confirmed this, as described below).

**Steps/Code to reproduce bug**

First, rewind to @rjzamora  's PR, which (for reasons that are right now unknown) fixed https://github.com/NVIDIA/NVTabular/issues/146:

```
# Align Dask and Single-GPU Writer Logic (#160)
git checkout 7407cfd
```

Run `examples/rossmann-store-sales-preproc.ipynb`

Run `examples/rossmann-store-sales-example.ipynb`

Here we see consistent convergence and good final RMSPEs, confirming the fix.

Note: NVTabular's `Workflow` outputs are now saved in `examples/data/jp_ross`

Now fast forward to master as of 2020 08 04:

```
# [REVIEW] Async torch Dataloaders (#127)
git checkout 7935f7e
```

Note: `examples/rossmann-store-sales-preproc.ipynb` is completely unchanged from `7407cfd ` to `7935f7e`.

If we run `examples/rossmann-store-sales-example.ipynb` 3 times, we now 1. see unstable convergence and 2. obtain final RMSPEs of

TensorFlow: 25.0%, 22.3%, 22.3%
fast.ai:    29.9%, 29.1%, 21.5%

**The problem seems to do with `Workflow` processing.**

Note that the newer version of `examples/rossmann-store-sales-example.ipynb` does not use `examples/data/jp_ross` for exporting `Workflow` data, but rather `examples/data/ross_pre`.

So, we can now run this notebook exactly as is but using `7407cfd `'s `Workflow` outputs instead. This was done by inserting

```
PREPROCESS_DIR = os.path.join(DATA_DIR, 'jp_ross')
PREPROCESS_DIR_TRAIN = os.path.join(PREPROCESS_DIR, 'train')
PREPROCESS_DIR_VALID = os.path.join(PREPROCESS_DIR, 'valid')
```

right before the Training a Network section.

Now, if we rerun the notebook 3 times, we once again get stable convergence, and the final RMSPEs are

```
TensorFlow: 18.9%, 17.4%, 17.9%
fast.ai:    19.7%, 19.5%, 21.4%
```

@benfred @rjzamora @jperez999 for visibility


Discussion:
I haven't thought through this issue yet, but it may be worth confirming that #196 (recently merged) doesn't fix things.  We discovered that there is a bug in cudf that is resulting in bad parquet data when slicing string columns (which we are doing when split the output data between multiple files)

[**EDIT**: This does **not** seem to fix the issue for me.]
Ok – I tried, too. Same here: problem not resolved as of `5a2d53d`.
Can you use ```git bisect``` to figure out where this started to break?

Resolving patch:
diff --git a/nvtabular/io.py b/nvtabular/io.py
--- a/nvtabular/io.py
+++ b/nvtabular/io.py
@@ -295,7 +295,7 @@ def add_data(self, gdf):
         # path is probably worth the (possible) minor overhead.
         nrows = gdf.shape[0]
         typ = np.min_scalar_type(nrows * 2)
-        if self.shuffle and self.shuffle != "full":
+        if self.shuffle:
             ind = cp.random.choice(cp.arange(self.num_out_files, dtype=typ), nrows)
         else:
             ind = cp.arange(nrows, dtype=typ)


Test patch:
diff --git a/tests/unit/test_torch_dataloader.py b/tests/unit/test_torch_dataloader.py
--- a/tests/unit/test_torch_dataloader.py
+++ b/tests/unit/test_torch_dataloader.py
@@ -74,7 +74,7 @@ def test_gpu_dl(tmpdir, df, dataset, batch_size, part_mem_fraction, engine):
     nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
 
     data_itr = nvt.torch_dataloader.AsyncTensorBatchDatasetItr(
-        nvt_data, batch_size=batch_size, cats=cat_names, conts=cont_names, labels=["label"],
+        nvt_data, batch_size=batch_size, cats=cat_names, conts=cont_names, labels=["label"]
     )
 
     columns = mycols_pq
@@ -124,11 +124,7 @@ def test_kill_dl(tmpdir, df, dataset, part_mem_fraction, engine):
     os.mkdir(output_train)
 
     processor.apply(
-        dataset,
-        apply_offline=True,
-        record_stats=True,
-        shuffle="partial",
-        output_path=output_train,
+        dataset, apply_offline=True, record_stats=True, shuffle="partial", output_path=output_train
     )
 
     tar_paths = [
@@ -138,7 +134,7 @@ def test_kill_dl(tmpdir, df, dataset, part_mem_fraction, engine):
     nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
 
     data_itr = nvt.torch_dataloader.AsyncTensorBatchDatasetItr(
-        nvt_data, cats=cat_names, conts=cont_names, labels=["label"],
+        nvt_data, cats=cat_names, conts=cont_names, labels=["label"]
     )
 
     results = {}