Repository: rwth-i6/returnn

Issue:
`ConcatFilesDataset` needs a better name
#1521, #1519.

The name is misleading given all the other concatenation-related datasets we have, because people assume that the dataset will somehow modify the data itself when its true purpose is assisting in large-scale trainings. This hinders explainability.

A name focusing on the intent might make more sense to many people, like:
- `LazyLoadingDataset`
- `ShardBySubepochDataset` 
- maybe even `LargeScaleDataset`

Feel free to dump more ideas into this issue. Then we'll decide on a new name by June 11, 5pm CEST.


Discussion:
I like `LazyLoadingDataset` :)

Isn't it actually a wrapper of a dataset that allows for more efficient access to the internal dataset structure? Maybe then something like `LazyLoadingDatasetWrapper`?
> `LazyLoadingDatasetWrapper`

No, all datasets must end with `Dataset`. Also `MetaDataset`, `ConcatDataset`, `CombinedDataset`, `MultiProcDataset` etc do, and they also all work the same way, i.e. wrap other datasets.

I prefer that the name describes how it operates rather than the intent. Currently this is the case for most of our datasets, functions, layers, everything. Thus sth like `LargeScaleDataset` is really too generic.
`ShardBySubepochDataset` is misleading, as there is no sharding. (At least not now. Maybe we might add this as another feature, via #1531.)

`LazyLoadingDataset` is also a bit too generic. Lazy loading happens in basically all datasets. It can mean anything. Sequences are always lazily loaded once needed (via `load_seqs`). `MultiProcDataset` also does a lot of lazy init stuff. Many other datasets also have lazy init to various degrees.

But what the dataset is doing (mainly) isn't concatenating other datasets. It is mainly about partitioning the files over the subepochs in a reasonable way to reduce the data load. Concatenation of the resulting files is a very unimportant detail of that process, yet it determines the name. Even the `HDFDataset` is a concatenation dataset in that regard, given that it can load more than one HDF at a time.

The point is the existing name is way too close to other datasets that have a meaningful concatenation operation for people to be able to clearly differentiate between them upon first glance.

New suggestion: `PartitionDataBySubepochDataset`
Some suggestions:

* (For reference, `RandomShuffleConcatDataset` was the initial suggestion.)
* `ConcatFilesDataset`, i.e. keep current name. Note that the original motivation for this name was the strong relation to `ConcatDataset`, which basically does the same thing conceptually.
* `FileDistribDataset`. "distribution" can refer to distribution of the files over the sub epochs. But at the same time, it can also refer to distributed training.
* `FileListDataset`
* `FilesMetaDataset`

I am fine with FileDistribDataset
> The point is the existing name is way too close to other datasets that have a meaningful concatenation operation for people to be able to clearly differentiate between them upon first glance.

Which exactly do you mean? There are indeed a couple of datasets which can be confusing. However, they all do different things. E.g.:

* `MetaDataset`: combines multiple datasets, num seqs stays same as one single, just adds the data keys
* `ConcatDataset`: combines multiple datasets, num seqs is the sum over all datasets, data keys must be the same over all datasets
* `CombinedDataset`: combines multiple datasets, num seqs is the sum over all datasets, data keys can be different over the datasets
* `ConcatSeqsDataset`: operates on a single dataset, concatenates sequences, i.e. num seqs is less than the original dataset

Conceptually, `ConcatFilesDataset` is exactly the same as `ConcatDataset`. The others are all different.

I also like `FileDistribDataset`, or even more complete for clarity, `FileDistributedDataset`.

Resolving patch:
diff --git a/returnn/datasets/basic.py b/returnn/datasets/basic.py
--- a/returnn/datasets/basic.py
+++ b/returnn/datasets/basic.py
@@ -1367,7 +1367,7 @@ def get_dataset_class(name: Union[str, Type[Dataset]]) -> Optional[Type[Dataset]
         "raw_wav",
         "map",
         "multi_proc",
-        "concat_files",
+        "distrib_files",
     ]
     for mod_name in mod_names:
         mod = import_module("returnn.datasets.%s" % mod_name)
diff --git a/returnn/datasets/concat_files.py b/returnn/datasets/distrib_files.py
similarity index 97%
rename from returnn/datasets/concat_files.py
rename to returnn/datasets/distrib_files.py
--- a/returnn/datasets/concat_files.py
+++ b/returnn/datasets/distrib_files.py
@@ -1,5 +1,5 @@
 """
-:class:`ConcatFilesDataset `
+:class:`DistributeFilesDataset`
 
 https://github.com/rwth-i6/returnn/issues/1519
 """
@@ -23,19 +23,21 @@
 _mp = NonDaemonicSpawnContext(process_pre_init_func=SubProcCopyGlobalConfigPreInitFunc())
 
 
-__all__ = ["ConcatFilesDataset"]
+__all__ = ["DistributeFilesDataset"]
 
 Filename = str
 FileTree = Union[Filename, Tuple["FileTree", ...], Dict[Any, "FileTree"], List["FileTree"]]
 
 
-class ConcatFilesDataset(CachedDataset2):
+class DistributeFilesDataset(CachedDataset2):
     """
-    This is similar to :class:`ConcatDataset`, but instead of concatenating datasets,
-    we distribute files over subepochs,
-    and then create a sub dataset for every sub epoch
-    for a given subset of the files
-    via the given ``get_sub_epoch_dataset``.
+    Dataset that distributes files over subepochs and then creates a
+    sub dataset for every sub epoch for a given (random) subset of the files.
+    The sub dataset is user-defined via a function ``get_sub_epoch_dataset``.
+    Thus, this dataset wraps the sub datasets.
+
+    It is conceptually very similar to :class:`ConcatDataset` in the sense
+    that it concatenates all the sub datasets together to form one larger dataset.
 
     This scheme allows to shuffle over the files,
     which makes shuffling much more efficient over a large dataset
@@ -70,7 +72,7 @@ def get_sub_epoch_dataset(files_subepoch: List[str]) -> Dict[str, Any]:
           }
 
         train = {
-          "class": "ConcatFilesDataset",
+          "class": "DistributeFilesDataset",
           "files": [
             "/nfs/big_data_1.hdf",
             ...
@@ -105,7 +107,7 @@ def get_sub_epoch_dataset(files_subepoch: List[Tuple[str, str]]) -> Dict[str, An
           }
 
         train = {
-          "class": "ConcatFilesDataset",
+          "class": "DistributeFilesDataset",
           "files": [
             ("/nfs/alignment_1.hdf", "/nfs/features_1.hdf"),
             ...
@@ -321,7 +323,7 @@ def _get_files_per_sub_epochs(
         # [[1,1], [78], [120], []] or [[1,1,78], [120], [], []].
         # Or consider [5,5]+[10]*7, partition_epoch=5, which has avg size 16.
         # A simple algorithm could end up with [[5,5,10], [10,10], [10,10], [10,10], []].
-        # See test_ConcatFilesDataset_get_files_per_sub_epochs for some test cases.
+        # See test_DistributeFilesDataset_get_files_per_sub_epochs for some test cases.
         assert len(files_order) >= partition_epoch
         files_per_sub_epochs = [[] for _ in range(partition_epoch)]
         assert len(files_per_sub_epochs) == partition_epoch


Test patch:
diff --git a/tests/test_Dataset.py b/tests/test_Dataset.py
--- a/tests/test_Dataset.py
+++ b/tests/test_Dataset.py
@@ -623,13 +623,13 @@ def __getitem__(self, item):
     assert res.features["data"].shape == (5, 3)
 
 
-def test_ConcatFilesDataset_get_files_per_sub_epochs():
-    from returnn.datasets.concat_files import ConcatFilesDataset
+def test_DistributeFilesDataset_get_files_per_sub_epochs():
+    from returnn.datasets.distrib_files import DistributeFilesDataset
 
     def _test(sizes: List[int], partition_epoch: int, expected: List[List[int]]):
         files = [f"file-{i}" for i in range(len(sizes))]
         file_sizes = {f: s for f, s in zip(files, sizes)}
-        res = ConcatFilesDataset._get_files_per_sub_epochs(
+        res = DistributeFilesDataset._get_files_per_sub_epochs(
             partition_epoch=partition_epoch, file_sizes=file_sizes, files_order=files
         )
         assert all(res) and len(res) == partition_epoch
@@ -643,8 +643,8 @@ def _test(sizes: List[int], partition_epoch: int, expected: List[List[int]]):
     _test([5, 5] + [10] * 7, 5, [[5, 5, 10], [10, 10], [10, 10], [10], [10]])
 
 
-def test_ConcatFilesDataset():
-    from returnn.datasets.concat_files import ConcatFilesDataset
+def test_DistributeFilesDataset():
+    from returnn.datasets.distrib_files import DistributeFilesDataset
     from test_HDFDataset import generate_hdf_from_other
 
     # Create a few HDF files such that we can easily verify the data later.
@@ -686,7 +686,7 @@ def test_ConcatFilesDataset():
         total_num_seqs += 1
     assert total_num_seqs == num_hdf_files * num_seqs
 
-    # Test to load via ConcatFilesDataset.
+    # Test to load via DistributeFilesDataset.
 
     def _get_sub_epoch_dataset(files_subepoch: List[str]) -> Dict[str, Any]:
         return {"class": "HDFDataset", "files": files_subepoch, "seq_ordering": "default"}
@@ -695,13 +695,13 @@ def _get_sub_epoch_dataset(files_subepoch: List[str]) -> Dict[str, Any]:
     assert num_hdf_files % partition_epoch == 0  # just for easier testing here
     concat_dataset = init_dataset(
         {
-            "class": "ConcatFilesDataset",
+            "class": "DistributeFilesDataset",
             "files": hdf_files,
             "get_sub_epoch_dataset": _get_sub_epoch_dataset,
             "partition_epoch": partition_epoch,
         }
     )
-    assert isinstance(concat_dataset, ConcatFilesDataset)
+    assert isinstance(concat_dataset, DistributeFilesDataset)
     assert concat_dataset.get_data_keys() == ["classes"]
     num_hdfs_per_part = num_hdf_files // partition_epoch
     global_seq_idx = 0