Repository: NVIDIA-Merlin/NVTabular

Issue:
[BUG] _file_list.txt is not written out in the validation output folder
**Describe the bug**
After we perform `workflow.transform()` both for train and validation set, we can see there is `_file_list.txt` is generated in the training output folder, but it is missing in the validation output folder.

`_file_list.txt` is required for HugeCTR training. There should be a file under train directory and another under valid directory and the file path is provided in the json file:

https://github.com/NVIDIA/NVTabular/blob/main/examples/hugectr/dlrm_fp32_64k.json#L35

**Expected behavior**
 `_file_list.txt` should be written out for validation also.

**Environment details (please complete the following information):**
 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)] :Conda
 - Method of NVTabular install: [conda, Docker, or from source]: Conda + pip install 
 


Discussion:
This seems to be because of the shuffling behaviour - we default to using the dask-cudf 'to_parquet' functionality when we are not shuffling:https://github.com/NVIDIA/NVTabular/blob/09f6afc05ab8872c96af8e5d91634b53ac1077b2/nvtabular/io/dataset.py#L406-L408 - which doesn't write out the hugectr filelist.txt file

@rjzamora should we just always use the NVT to _ddf_to_dataset code to write out parquet datasets? Is there any advantage to the dask_cudf.to_parquet code?
> @rjzamora should we just always use the NVT to out_files_per_proc=None code to write out parquet datasets? Is there any advantage to the dask_cudf.to_parquet code?

It's probably fine to use `out_files_per_proc=None`.  I don't recall if there are any real advantages to using `dask_cudf.to_parquet` - I'm not sure if it is still the case, but this code path was not originally the default, even with `shuffle=None`, because it required the user to explicitly specify `out_files_per_proc=None`.
@rnyak @rjzamora @benfred I get all the output files (file_list metadata, and parquet) with the updated Notebook: https://github.com/NVIDIA/NVTabular/blob/main/examples/hugectr/criteo-hugectr.ipynb
+1 I observed the same issue. Upon adding ` shuffle=nvt.io.Shuffle.PER_PARTITION` I finally get the meta data files 

```
proc.transform(train_dataset).to_parquet(output_path=output_train_dir, dtypes=dict_dtypes,
                                         shuffle=nvt.io.Shuffle.PER_PARTITION,
                                         cats=cat_feats.columns,
                                         conts=cont_feats.columns,
                                         labels=['target'])
```

Resolving patch:
diff --git a/nvtabular/io/dataset.py b/nvtabular/io/dataset.py
--- a/nvtabular/io/dataset.py
+++ b/nvtabular/io/dataset.py
@@ -448,32 +448,22 @@ def to_parquet(
 
         fs = get_fs_token_paths(output_path)[0]
         fs.mkdirs(output_path, exist_ok=True)
-        if shuffle or out_files_per_proc or cats or conts or labels:
-
-            # Output dask_cudf DataFrame to dataset
-            _ddf_to_dataset(
-                ddf,
-                fs,
-                output_path,
-                shuffle,
-                out_files_per_proc,
-                cats or [],
-                conts or [],
-                labels or [],
-                "parquet",
-                self.client,
-                num_threads,
-                self.cpu,
-            )
-            return
 
-        # Default (shuffle=None and out_files_per_proc=None)
-        # Just use `dask_cudf.to_parquet`
-        fut = ddf.to_parquet(output_path, compression=None, write_index=False, compute=False)
-        if self.client is None:
-            fut.compute(scheduler="synchronous")
-        else:
-            fut.compute()
+        # Output dask_cudf DataFrame to dataset
+        _ddf_to_dataset(
+            ddf,
+            fs,
+            output_path,
+            shuffle,
+            out_files_per_proc,
+            cats or [],
+            conts or [],
+            labels or [],
+            "parquet",
+            self.client,
+            num_threads,
+            self.cpu,
+        )
 
     def to_hugectr(
         self,


Test patch:
diff --git a/tests/unit/test_dask_nvt.py b/tests/unit/test_dask_nvt.py
--- a/tests/unit/test_dask_nvt.py
+++ b/tests/unit/test_dask_nvt.py
@@ -88,7 +88,7 @@ def test_dask_workflow_api_dlrm(
     output_path = os.path.join(tmpdir, "processed")
 
     transformed = workflow.fit_transform(dataset)
-    transformed.to_parquet(output_path=output_path, shuffle=shuffle)
+    transformed.to_parquet(output_path=output_path, shuffle=shuffle, out_files_per_proc=1)
 
     result = transformed.to_ddf().compute()
     assert len(df0) == len(result)
@@ -112,12 +112,12 @@ def test_dask_workflow_api_dlrm(
     # Read back from disk
     df_disk = dask_cudf.read_parquet(output_path, index=False).compute()
 
-    # Can directly compare the final ddf to the result if we didn't shuffle
-    if not shuffle:
-        for col in df_disk:
-            assert_eq(result[col], df_disk[col])
-    else:
-        assert len(df0) == len(df_disk)
+    # we don't have a deterministic ordering here, especially when using
+    # a dask client with multiple workers - so we need to sort the values here
+    columns = ["label", "x", "y", "id"] + cat_names
+    got = result.sort_values(columns).reset_index(drop=True)
+    expect = df_disk.sort_values(columns).reset_index(drop=True)
+    assert_eq(got, expect)
 
 
 @pytest.mark.parametrize("part_mem_fraction", [0.01])