Repository: yt-project/yt

Issue:
Memory and CPU usage with small Gadget data sets
After the change in #2575, we run into some real problems with some smaller datasets, where essentially the entire dataset gets expanded up to the maximum bitmap level.  This is problematic for datasets like `Gadget3-snap-format2`, which have multiple data files, very few particles, but big smoothing lengths that cover many low-order cells.

Default behavior for this script:

```python
import yt

with yt.memory_checker():
    ds = yt.load_sample("Gadget3-snap-format2")
    ds.index
```

produces a ridiculous amount of memory usage and takes a hubble time.

This can be essentially completely remedied by setting the `index_order` to have a *lower* order for the second component.  I'm tempted to consider a heuristic setting of the index order to be the appropriate fix.


Discussion:
@chummels you mentioned seeing something similar a while back with a different dataset; see if setting `index_order = (2, 4)` or something in the `load` call fixes it?
One possibility I'm exploring is to adjust the default `order2` to make sure it doesn't result in some cell size that's smaller than a reasonable fraction of the mean/median/min smoothing length.

Resolving patch:
diff --git a/yt/data_objects/static_output.py b/yt/data_objects/static_output.py
--- a/yt/data_objects/static_output.py
+++ b/yt/data_objects/static_output.py
@@ -1900,7 +1900,7 @@ def __init__(
 
 def validate_index_order(index_order):
     if index_order is None:
-        index_order = (7, 5)
+        index_order = (6, 2)
     elif not is_sequence(index_order):
         index_order = (int(index_order), 1)
     else:
diff --git a/yt/geometry/particle_geometry_handler.py b/yt/geometry/particle_geometry_handler.py
--- a/yt/geometry/particle_geometry_handler.py
+++ b/yt/geometry/particle_geometry_handler.py
@@ -163,12 +163,16 @@ def _initialize_index(self):
         )
 
         # Load Morton index from file if provided
-        if getattr(ds, "index_filename", None) is None:
-            fname = ds.parameter_filename + ".index{}_{}.ewah".format(
-                self.regions.index_order1, self.regions.index_order2
-            )
-        else:
-            fname = ds.index_filename
+        def _current_fname():
+            if getattr(ds, "index_filename", None) is None:
+                fname = ds.parameter_filename + ".index{}_{}.ewah".format(
+                    self.regions.index_order1, self.regions.index_order2
+                )
+            else:
+                fname = ds.index_filename
+            return fname
+
+        fname = _current_fname()
 
         dont_load = dont_cache and not hasattr(ds, "index_filename")
         try:
@@ -183,6 +187,8 @@ def _initialize_index(self):
             self.regions.reset_bitmasks()
             self._initialize_coarse_index()
             self._initialize_refined_index()
+            # We now update fname since index_order2 may have changed
+            fname = _current_fname()
             wdir = os.path.dirname(fname)
             if not dont_cache and os.access(wdir, os.W_OK):
                 # Sometimes os mis-reports whether a directory is writable,
@@ -195,6 +201,7 @@ def _initialize_index(self):
 
     def _initialize_coarse_index(self):
         pb = get_pbar("Initializing coarse index ", len(self.data_files))
+        max_hsml = 0.0
         for i, data_file in enumerate(self.data_files):
             pb.update(i)
             for ptype, pos in self.io._yield_coordinates(data_file):
@@ -203,12 +210,28 @@ def _initialize_coarse_index(self):
                     hsml = self.io._get_smoothing_length(
                         data_file, pos.dtype, pos.shape
                     )
+                    if hsml is not None and hsml.size > 0.0:
+                        max_hsml = max(max_hsml, hsml.max())
                 else:
                     hsml = None
                 self.regions._coarse_index_data_file(pos, hsml, data_file.file_id)
             self.regions._set_coarse_index_data_file(data_file.file_id)
         pb.finish()
         self.regions.find_collisions_coarse()
+        if max_hsml > 0.0 and len(self.data_files) > 1:
+            # By passing this in, we only allow index_order2 to be increased by
+            # two at most, never increased.  One place this becomes particularly
+            # useful is in the case of an extremely small section of gas
+            # particles embedded in a much much larger domain.  The max
+            # smoothing length will be quite small, so based on the larger
+            # domain, it will correspond to a very very high index order, which
+            # is a large amount of memory!  Having multiple indexes, one for
+            # each particle type, would fix this.
+            new_order2 = self.regions.update_mi2(max_hsml, ds.index_order[1] + 2)
+            mylog.info(
+                "Updating index_order2 from %s to %s", ds.index_order[1], new_order2
+            )
+            self.ds.index_order = (self.ds.index_order[0], new_order2)
 
     def _initialize_refined_index(self):
         mask = self.regions.masks.sum(axis=1).astype("uint8")


Test patch: