Repository: yt-project/yt

Issue:
bug: mesh sampling throws error with FLASH data
### Bug report
 
The script below attempts to create a mesh sampled particle field using a FLASH dataset with plt and part files. The expected outcome is a list of density values at the location of each particle, like those produced by the example script (https://yt-project.org/doc/reference/api/yt.data_objects.static_output.html#yt.data_objects.static_output.Dataset.add_mesh_sampling_particle_field). The script below, which is identical to the example script except with FLASH datafiles, produces the error "IndexError: index 16 is out of bounds for axis 0 with size 16". 

**Code for reproduction**
FLASH data found here:
plt file: http://use.yt/upload/a94cdffd 
part file: http://use.yt/upload/dec62ce5

test script, also copied below:
 http://paste.yt-project.org/show/266/

```python
import yt
ds = yt.load("./group_subh_hgas_iso_hdf5_plt_cnt_0000", particle_filename="./group_subh_hgas_iso_hdf5_part_0000")
ds.add_mesh_sampling_particle_field(('gas', 'density'), ptype='all')
print('The density at the location of the particle is:')
print(ds.r['all', 'cell_gas_density'])
```

**Actual outcome**

```
Traceback (most recent call last):
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 355, in _generate_particle_field
    finfo.check_available(gen_obj)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 204, in check_available
    validator(data)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 470, in __call__
    raise NeedsGridType(self.ghost_zones,self.fields)
yt.fields.field_exceptions.NeedsGridType: (0, None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mesh_script.py", line 5, in <module>
    print(ds.r['all', 'cell_gas_density'])
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/region_expression.py", line 42, in __getitem__
    return self.all_data[item]
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 256, in __getitem__
    self.get_data(f)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1583, in get_data
    self._generate_fields(fields_to_generate)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1603, in _generate_fields
    fd = self._generate_field(field)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 291, in _generate_field
    tr = self._generate_particle_field(field)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 363, in _generate_particle_field
    for i, chunk in enumerate(self.chunks(field, "spatial")):
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1477, in chunks
    self.get_data(fields)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1583, in get_data
    self._generate_fields(fields_to_generate)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1603, in _generate_fields
    fd = self._generate_field(field)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 291, in _generate_field
    tr = self._generate_particle_field(field)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 376, in _generate_particle_field
    rv = self.ds._get_field_info(*field)(gen_obj)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 256, in __call__
    dd = self._function(self, data)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/geometry/grid_geometry_handler.py", line 406, in _mesh_sampling_particle_field
    return field_values[i, j, k]
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/units/yt_array.py", line 1065, in __getitem__
    ret = super(YTArray, self).__getitem__(item)
IndexError: index 16 is out of bounds for axis 0 with size 16
```

**Expected Outcome**
The example script 
```python
import yt
ds = yt.load('output_00080/info_00080.txt')
... ds.add_mesh_sampling_particle_field(('gas', 'density'), ptype='all')
print('The density at the location of the particle is:')
print(ds.r['all', 'cell_gas_density'])
```
outputs:
The density at the location of the particle is:
[9.33886124e-30 1.22174333e-28 1.20402333e-28 ... 2.77410331e-30
 8.79467609e-31 3.50665136e-30] g/cm**3

**Version Information**
  * Operating System: Linux 7
  * Python Version: 3.7.6
  * yt version: 3.6

yt installed using installation script


bug: mesh sampling throws error with FLASH data
### Bug report
 
The script below attempts to create a mesh sampled particle field using a FLASH dataset with plt and part files. The expected outcome is a list of density values at the location of each particle, like those produced by the example script (https://yt-project.org/doc/reference/api/yt.data_objects.static_output.html#yt.data_objects.static_output.Dataset.add_mesh_sampling_particle_field). The script below, which is identical to the example script except with FLASH datafiles, produces the error "IndexError: index 16 is out of bounds for axis 0 with size 16". 

**Code for reproduction**
FLASH data found here:
plt file: http://use.yt/upload/a94cdffd 
part file: http://use.yt/upload/dec62ce5

test script, also copied below:
 http://paste.yt-project.org/show/266/

```python
import yt
ds = yt.load("./group_subh_hgas_iso_hdf5_plt_cnt_0000", particle_filename="./group_subh_hgas_iso_hdf5_part_0000")
ds.add_mesh_sampling_particle_field(('gas', 'density'), ptype='all')
print('The density at the location of the particle is:')
print(ds.r['all', 'cell_gas_density'])
```

**Actual outcome**

```
Traceback (most recent call last):
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 355, in _generate_particle_field
    finfo.check_available(gen_obj)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 204, in check_available
    validator(data)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 470, in __call__
    raise NeedsGridType(self.ghost_zones,self.fields)
yt.fields.field_exceptions.NeedsGridType: (0, None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mesh_script.py", line 5, in <module>
    print(ds.r['all', 'cell_gas_density'])
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/region_expression.py", line 42, in __getitem__
    return self.all_data[item]
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 256, in __getitem__
    self.get_data(f)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1583, in get_data
    self._generate_fields(fields_to_generate)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1603, in _generate_fields
    fd = self._generate_field(field)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 291, in _generate_field
    tr = self._generate_particle_field(field)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 363, in _generate_particle_field
    for i, chunk in enumerate(self.chunks(field, "spatial")):
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1477, in chunks
    self.get_data(fields)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1583, in get_data
    self._generate_fields(fields_to_generate)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1603, in _generate_fields
    fd = self._generate_field(field)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 291, in _generate_field
    tr = self._generate_particle_field(field)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 376, in _generate_particle_field
    rv = self.ds._get_field_info(*field)(gen_obj)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 256, in __call__
    dd = self._function(self, data)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/geometry/grid_geometry_handler.py", line 406, in _mesh_sampling_particle_field
    return field_values[i, j, k]
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/units/yt_array.py", line 1065, in __getitem__
    ret = super(YTArray, self).__getitem__(item)
IndexError: index 16 is out of bounds for axis 0 with size 16
```

**Expected Outcome**
The example script 
```python
import yt
ds = yt.load('output_00080/info_00080.txt')
... ds.add_mesh_sampling_particle_field(('gas', 'density'), ptype='all')
print('The density at the location of the particle is:')
print(ds.r['all', 'cell_gas_density'])
```
outputs:
The density at the location of the particle is:
[9.33886124e-30 1.22174333e-28 1.20402333e-28 ... 2.77410331e-30
 8.79467609e-31 3.50665136e-30] g/cm**3

**Version Information**
  * Operating System: Linux 7
  * Python Version: 3.7.6
  * yt version: 3.6

yt installed using installation script




Discussion:
Hi, and welcome to yt!  Thanks for opening your first issue.  We have an issue template that helps us to gather relevant information to help diagnosing and fixing the issue.

Hi! Thanks for the report. This is indeed a bug, but unfortunately I cannot reproduce it locally (I don't have access to a flash dataset). The bug will happen whenever a particle is slightly outside a grid.
If you feel like trying, you can use the master branch of yt (see https://yt-project.org/docs/dev/installing.html#installing-yt-from-source) and apply the following diff
```diff
diff --git a/yt/geometry/grid_geometry_handler.py b/yt/geometry/grid_geometry_handler.py
index 87db8ff15..3b64c333e 100644
--- a/yt/geometry/grid_geometry_handler.py
+++ b/yt/geometry/grid_geometry_handler.py
@@ -473,7 +473,7 @@ class GridIndex(Index, abc.ABC):
             if isinstance(data, FieldDetector):
                 return np.zeros(pos.shape[0])
 
-            i, j, k = np.floor((pos - data.LeftEdge) / data.dds).astype("int64").T
+            i, j, k = np.floor((pos - data.LeftEdge) / data.dds).astype("int64").T + 1
 
             return field_values[i, j, k]
 
@@ -483,7 +483,7 @@ class GridIndex(Index, abc.ABC):
             sampling_type="particle",
             units=units,
             take_log=take_log,
-            validators=[ValidateSpatial()],
+            validators=[ValidateSpatial(ghost_zones=1)],
         )
```
Tell me if that solves your issue!
Hi! Thanks for the response. I implemented the two changes above, and the new output is 

```
Traceback (most recent call last):
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/field_detector.py", line 108, in __missing__
    vv = finfo(self)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 249, in __call__
    self.check_available(data)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 204, in check_available
    validator(data)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 473, in __call__
    raise NeedsGridType(self.ghost_zones,self.fields)
yt.fields.field_exceptions.NeedsGridType: (1, None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mesh_script.py", line 3, in <module>
    ds.add_mesh_sampling_particle_field(('gas', 'density'), ptype='all')
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/static_output.py", line 1239, in add_mesh_sampling_particle_field
    return self.index._add_mesh_sampling_particle_field(sample_field, ftype, ptype)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/geometry/grid_geometry_handler.py", line 414, in _add_mesh_sampling_particle_field
    validators=[ValidateSpatial(ghost_zones=1)])
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/static_output.py", line 1194, in add_field
    deps, _ = self.field_info.check_derived_fields([name])
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/field_info_container.py", line 405, in check_derived_fields
    fd = fi.get_dependencies(ds = self.ds)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 216, in get_dependencies
    e[self.name]
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/field_detector.py", line 117, in __missing__
    if ngz > 0: vv = vv[ngz:-ngz, ngz:-ngz, ngz:-ngz]
IndexError: too many indices for array
```

where a print statement before this line shows that vv = [0.] in this iteration (vv is successfully printed many times before this, with a typical vv being a 3 layer YTArray of YTArrays with units of cm x code_time/(code_length x s))

Additionally, the error is being thrown on the line ds.add_mesh_sampling_particle_field(('gas', 'density'), ptype='all'), whereas before it was throwing an error on the line print(ds.r['all', 'cell_gas_density'])

If there's any other info I can provide to help, let me know! And if you're looking for FLASH data and don't mind downloading large files, I linked to the FLASH dataset I am using in the bug report. 

Thanks, 
Camryn

I cannot currently download any large dataset, but maybe you'll have a chance with the following patch (that replaces the previous one). You'll get some NaN, but at least it shouldn't crash!

Note that I did not test it, so you may have to adapt it to get it to work.
```diff
diff --git a/yt/geometry/grid_geometry_handler.py b/yt/geometry/grid_geometry_handler.py
index 87db8ff15..9687b78cd 100644
--- a/yt/geometry/grid_geometry_handler.py
+++ b/yt/geometry/grid_geometry_handler.py
@@ -475,7 +475,18 @@ class GridIndex(Index, abc.ABC):
 
             i, j, k = np.floor((pos - data.LeftEdge) / data.dds).astype("int64").T
 
-            return field_values[i, j, k]
+            maxi, maxj, maxk = field_values.shape
+
+            mask = (i < maxi) & (j < maxj) & (k < maxk)
+            mask &= (i >= 0) & (j >= 0) & (k >= 0)
+
+            result = np.full_like(i, np.nan, dtype="float64")
+            result[mask] = field_values[i[mask], j[mask], k[mask]]
+
+            return data.ds.arr(
+                result,
+                field_values.units
+            )
 
         self.ds.add_field(
             (ptype, field_name),
```

That works! The script now assigns a gas density to each particle and correctly prints the values. Thanks for the help!
Hi, and welcome to yt!  Thanks for opening your first issue.  We have an issue template that helps us to gather relevant information to help diagnosing and fixing the issue.

Hi! Thanks for the report. This is indeed a bug, but unfortunately I cannot reproduce it locally (I don't have access to a flash dataset). The bug will happen whenever a particle is slightly outside a grid.
If you feel like trying, you can use the master branch of yt (see https://yt-project.org/docs/dev/installing.html#installing-yt-from-source) and apply the following diff
```diff
diff --git a/yt/geometry/grid_geometry_handler.py b/yt/geometry/grid_geometry_handler.py
index 87db8ff15..3b64c333e 100644
--- a/yt/geometry/grid_geometry_handler.py
+++ b/yt/geometry/grid_geometry_handler.py
@@ -473,7 +473,7 @@ class GridIndex(Index, abc.ABC):
             if isinstance(data, FieldDetector):
                 return np.zeros(pos.shape[0])
 
-            i, j, k = np.floor((pos - data.LeftEdge) / data.dds).astype("int64").T
+            i, j, k = np.floor((pos - data.LeftEdge) / data.dds).astype("int64").T + 1
 
             return field_values[i, j, k]
 
@@ -483,7 +483,7 @@ class GridIndex(Index, abc.ABC):
             sampling_type="particle",
             units=units,
             take_log=take_log,
-            validators=[ValidateSpatial()],
+            validators=[ValidateSpatial(ghost_zones=1)],
         )
```
Tell me if that solves your issue!
Hi! Thanks for the response. I implemented the two changes above, and the new output is 

```
Traceback (most recent call last):
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/field_detector.py", line 108, in __missing__
    vv = finfo(self)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 249, in __call__
    self.check_available(data)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 204, in check_available
    validator(data)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 473, in __call__
    raise NeedsGridType(self.ghost_zones,self.fields)
yt.fields.field_exceptions.NeedsGridType: (1, None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mesh_script.py", line 3, in <module>
    ds.add_mesh_sampling_particle_field(('gas', 'density'), ptype='all')
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/static_output.py", line 1239, in add_mesh_sampling_particle_field
    return self.index._add_mesh_sampling_particle_field(sample_field, ftype, ptype)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/geometry/grid_geometry_handler.py", line 414, in _add_mesh_sampling_particle_field
    validators=[ValidateSpatial(ghost_zones=1)])
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/data_objects/static_output.py", line 1194, in add_field
    deps, _ = self.field_info.check_derived_fields([name])
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/field_info_container.py", line 405, in check_derived_fields
    fd = fi.get_dependencies(ds = self.ds)
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/derived_field.py", line 216, in get_dependencies
    e[self.name]
  File "/XMMShared/clp3ef/yt-conda/lib/python3.7/site-packages/yt/fields/field_detector.py", line 117, in __missing__
    if ngz > 0: vv = vv[ngz:-ngz, ngz:-ngz, ngz:-ngz]
IndexError: too many indices for array
```

where a print statement before this line shows that vv = [0.] in this iteration (vv is successfully printed many times before this, with a typical vv being a 3 layer YTArray of YTArrays with units of cm x code_time/(code_length x s))

Additionally, the error is being thrown on the line ds.add_mesh_sampling_particle_field(('gas', 'density'), ptype='all'), whereas before it was throwing an error on the line print(ds.r['all', 'cell_gas_density'])

If there's any other info I can provide to help, let me know! And if you're looking for FLASH data and don't mind downloading large files, I linked to the FLASH dataset I am using in the bug report. 

Thanks, 
Camryn

I cannot currently download any large dataset, but maybe you'll have a chance with the following patch (that replaces the previous one). You'll get some NaN, but at least it shouldn't crash!

Note that I did not test it, so you may have to adapt it to get it to work.
```diff
diff --git a/yt/geometry/grid_geometry_handler.py b/yt/geometry/grid_geometry_handler.py
index 87db8ff15..9687b78cd 100644
--- a/yt/geometry/grid_geometry_handler.py
+++ b/yt/geometry/grid_geometry_handler.py
@@ -475,7 +475,18 @@ class GridIndex(Index, abc.ABC):
 
             i, j, k = np.floor((pos - data.LeftEdge) / data.dds).astype("int64").T
 
-            return field_values[i, j, k]
+            maxi, maxj, maxk = field_values.shape
+
+            mask = (i < maxi) & (j < maxj) & (k < maxk)
+            mask &= (i >= 0) & (j >= 0) & (k >= 0)
+
+            result = np.full_like(i, np.nan, dtype="float64")
+            result[mask] = field_values[i[mask], j[mask], k[mask]]
+
+            return data.ds.arr(
+                result,
+                field_values.units
+            )
 
         self.ds.add_field(
             (ptype, field_name),
```

That works! The script now assigns a gas density to each particle and correctly prints the values. Thanks for the help!

Resolving patch:
diff --git a/yt/geometry/grid_geometry_handler.py b/yt/geometry/grid_geometry_handler.py
--- a/yt/geometry/grid_geometry_handler.py
+++ b/yt/geometry/grid_geometry_handler.py
@@ -462,7 +462,17 @@ def _mesh_sampling_particle_field(field, data):
 
             i, j, k = np.floor((pos - data.LeftEdge) / data.dds).astype("int64").T
 
-            return field_values[i, j, k]
+            # Make sure all particles are within the current grid, otherwise return nan
+            maxi, maxj, maxk = field_values.shape
+
+            mask = (i < maxi) & (j < maxj) & (k < maxk)
+            mask &= (i >= 0) & (j >= 0) & (k >= 0)
+
+            result = np.full(len(pos), np.nan, dtype="float64")
+            if result.shape[0] > 0:
+                result[mask] = field_values[i[mask], j[mask], k[mask]]
+
+            return data.ds.arr(result, field_values.units)
 
         self.ds.add_field(
             (ptype, field_name),


Test patch: