Repository: yt-project/yt

Issue:
yt-4.0 kdtree large memory usage
### Bug report

Jared Coughlin reports:

> Hello! I'm using the sph-viz branch of yt and I'm trying to make a density projection plot from a gadget-2 snapshot. This works like a charm when I use a snapshot with 428^3 particles in a 15 Mpc box. However, when I try to do it on a snapshot with 1024^3 particles in a 40 Mpc box, my job keeps getting killed because it is trying to exceed the available system memory. I'm submitting it to a node with 256 GB of RAM, and I'm requesting all of it, which means that more than this is trying to be used. The snapshot itself is ~ 40 GB. The memory usage appears to spike when allocating for the KDTree. Before that, it appears to generate the .ewah file without issue. I was just wondering if anyone had any thoughts on why this large spike in memory might be occurring and how I might go about fixing it? If not, no worries, I can try and investigate it myself, but it's a pain to do with pdb. My gadget simulation used a maximum of ~ 173 GB of RAM, so I feel like the kdtree shouldn't be using as much memory as it's trying to. 
> Thanks, and sorry for the bother! 
> Sincerely, -Jared

this should be reproducible with large datasets,

```python
ds = GadgetDataset(file_name) 
p = yt.ProjectionPlot(ds, axis = 'z', fields = ('Gas', 'Density')).save(file_name + '.png')
```

**Actual outcome**

The code is killed due to exceeding memory usage of 256 GB of RAM.

**Expected outcome**

The code *shouldn't* be making a KDTree unless a) gather smoothing is being used, or 2) the smoothing lengths aren't found.

**Version Information**
  * Operating System: -
  * Python Version: -
  * yt version: yt-4.0
  * Other Libraries (if applicable): -


Discussion:
I've managed to reproduce this. The code currently loads a `KDTreeProj` and then, for some reason, the KDTree seems to eat memory and crash for large datasets.
Argh, that's my fault. This is related to an experiment that I was doing in the spring that we should back out now that you've done your project in a way that scales better and has acceptable performance. We really need to come up with a nice way to regenerate projections quickly, but I think my ideas about doing that using the kdtree will have to be shelved until we can figure out ways to scale the kdtree better.

Ash, I can do the work here since I was the one who refactored things this Spring.

Resolving patch:
diff --git a/yt/data_objects/construction_data_containers.py b/yt/data_objects/construction_data_containers.py
--- a/yt/data_objects/construction_data_containers.py
+++ b/yt/data_objects/construction_data_containers.py
@@ -357,24 +357,18 @@ def _initialize_projected_units(self, fields, chunk):
             else:
                 self._projected_units[field] = field_unit
 
-class YTKDTreeProj(YTProj):
+class YTParticleProj(YTProj):
     """
-    TODO add docstring
+    A projection operation optimized for SPH particle data.
     """
     _type_name = "particle_proj"
     def __init__(self, field, axis, weight_field=None, center=None, ds=None,
                  data_source=None, style=None, method="integrate",
                  field_parameters=None, max_level=None):
-        super(YTKDTreeProj, self).__init__(
+        super(YTParticleProj, self).__init__(
             field, axis, weight_field, center, ds, data_source, style, method,
             field_parameters, max_level)
 
-        # ensure the dataset has a kdtree built
-        ds.index.kdtree
-
-    def _get_tree(self, nvals):
-        return self.ds.index.kdtree
-
     def _handle_chunk(self, chunk, fields, tree):
         raise NotImplementedError("Particle projections have not yet been "
                                   "implemented")
diff --git a/yt/geometry/coordinates/cartesian_coordinates.py b/yt/geometry/coordinates/cartesian_coordinates.py
--- a/yt/geometry/coordinates/cartesian_coordinates.py
+++ b/yt/geometry/coordinates/cartesian_coordinates.py
@@ -240,7 +240,7 @@ def _ortho_pixelize(self, data_source, field, bounds, size, antialias,
         from yt.data_objects.selection_data_containers import \
             YTSlice
         from yt.data_objects.construction_data_containers import \
-            YTKDTreeProj
+            YTParticleProj
         # We should be using fcoords
         field = data_source._determine_fields(field)[0]
         period = self.period[:2].copy() # dummy here
@@ -269,7 +269,7 @@ def _ortho_pixelize(self, data_source, field, bounds, size, antialias,
             ounits = data_source.ds.field_info[field].output_units
             px_name = 'particle_position_%s' % self.axis_name[self.x_axis[dim]]
             py_name = 'particle_position_%s' % self.axis_name[self.y_axis[dim]]
-            if isinstance(data_source, YTKDTreeProj):
+            if isinstance(data_source, YTParticleProj):
                 weight = data_source.weight_field
                 le = data_source.data_source.left_edge.in_units('code_length')
                 re = data_source.data_source.right_edge.in_units('code_length')


Test patch: