Repository: yt-project/yt

Issue:
ValueError when doing particle trajectories
### Bug report

**Bug summary**

When trying to get particle trajectories, I get the following error:
```python
ValueError                                Traceback (most recent call last)
~/Documents/These/ramses_tests/MC_tracers/test_feedback/plot.py in <module>()
     40     dss = yt.DatasetSeries(outputs[:5], **load_kwa)
     41
---> 42     trajs = dss.particle_trajectories(indices)

~/Documents/prog/yt/yt/data_objects/time_series.py in particle_trajectories(self, indices, fields, suppress_logging)
    435         >>>     print t["particle_velocity_x"].max(), t["particle_velocity_x"].min()
    436         """
--> 437         return ParticleTrajectories(self, indices, fields=fields, suppress_logging=suppress_logging)
    438
    439 class TimeSeriesQuantitiesContainer(object):

~/Documents/prog/yt/yt/data_objects/particle_trajectories.py in __init__(self, outputs, indices, fields, suppress_logging)
    121         for field in ("particle_position_%s" % ax for ax in "xyz"):
    122             for i, (fn, (time, indices, pfields)) in enumerate(sorted(my_storage.items())):
--> 123                 output_field[indices, i] = pfields[field]
    124             self.field_data[field] = array_like_field(
    125                 dd_first, output_field.copy(), fds[field])

ValueError: shape mismatch: value array of shape (10096,) could not be broadcast to indexing result of shape (96,)
```

**Code for reproduction**

```python
import yt
from glob import glob

def load_filters(ds):
    ptypes = {
        0: 'DM',
        1: 'star',
        100: 'MC_tracer',
        101: 'star_tracer',
        123: 'old_star_tracer'
    }

    def gen_filter(pt):
        def loc(pfilter, data):
            return data[(pfilter.filtered_type, "particle_type")] == pt
        return loc

    for pt in ptypes:
        yt.add_particle_filter(ptypes[pt], function=gen_filter(pt),
                               requires=['particle_type'], filtered_type='all')

    for pt in ptypes:
        ds.add_particle_filter(ptypes[pt])


if __name__ == '__main__':
    load_kwa = dict(extra_particle_fields=[('particle_type', 'I')])
    outputs = sorted(glob('output_*/info*.txt'))

    # Get the index of the particles released by stars
    last_ds = yt.load(outputs[-1], **load_kwa)
    load_filters(last_ds)

    indices = last_ds.r['old_star_tracer', 'particle_identifier'].astype(int)

    # Now let's take a look at their life
    dss = yt.DatasetSeries(outputs, **load_kwa)

    trajs = dss.particle_trajectories(indices)
```

**Version Information**
  * Linux x64
  * Python Version: 3.6
  * yt version: 3.5dev0

I have installed yt from source. The dataset is attached.
[dataset.zip](https://github.com/yt-project/yt/files/1228827/dataset.zip)




Discussion:
So I think what's happening here is that the third dataset in the time series has repeated `particle_index` values:

```
In [10]: np.unique(ad['io', 'particle_identifier']).shape
Out[10]: (65535,)

In [11]: ds.particle_type_counts
Out[11]: {'io': 75535}
```

(this is for `output_00003/info_00003.txt`)

For example, particle ID `1332` corresponds to two different particles:

```
In [1]: ad = ds.all_data()

In [2]: ids = ad['particle_identifier']

In [3]: ids[ids == 1332]
Out[3]: YTArray([ 1332.,  1332.]) (dimensionless)

In [4]: np.where(ids == 1332)
Out[4]: (array([ 3861, 38130]),)

In [5]: wh = np.where(ids == 1332)[0]

In [6]: ad['particle_mass'][wh]
Out[6]: YTArray([  0.00000000e+00,   4.91743232e+40]) g

In [7]: ad['particle_position'][wh]
Out[7]:
YTArray([[ 0.484375,  0.015625,  0.578125],
         [ 0.828125,  0.609375,  0.296875]]) code_length
```

The fact that there are exactly 2^16 unique particle IDs is a little suspicious. Perhaps integer overflow somewhere?
Hi,
you're right, my dataset is somehow buggy, if I change the ids to have uniqueness, the problem disappears.

Should yt try to detect this issue or is it always assumed that the dataset is correct?
We should probably raise a better error message if we find particles with identical IDs. 
Right; but isn't it quite costly to check uniqueness of IDs? (like `O(n log n)`)
It should only be done in places like this (particle trajectories) where we rely on such uniqueness. 
Something like https://github.com/cphyc/yt/commit/ecb1371c96637aea7db29fd7b36b367f77b39f4e#diff-5db40c3b9d593516f8cfbc0ec229e2cfR94 ?
I think so. Do you have a sense of how much this affects performance? 
For 2M particles, I get something around ~150ms per call, so it'd be too bad as long as the number of particles is small. For 20M particles, it takes ~1.8s and for 200M ~18s. Given that a (arg)sort is performed here two lines below, it means that the computation time will be doubled and can significantly increase for a large number of particles.

```python
In [1]: import numpy as np
In [2]: ints = np.random.randint(0, 1800000, 2000000)
In [3]: %timeit np.unique(ints)
10 loops, best of 3: 165 ms per loop
```

Another solution would be to add a try/except statement around https://github.com/cphyc/yt/commit/ecb1371c96637aea7db29fd7b36b367f77b39f4e#diff-5db40c3b9d593516f8cfbc0ec229e2cfL123 like:
```python
for field in ("particle_position_%s" % ax for ax in "xyz"):
    for i, (fn, (time, indices, pfields)) in enumerate(sorted(my_storage.items())):
        try:
            output_field[indices, i] = pfields[field]
        except ValueError:
            mylog.error('The dataset contains duplicate particle index.')
            raise IndexError # Or a more correct error here.
    self.field_data[field] = array_like_field(
        dd_first, output_field.copy(), fds[field])
    self.particle_fields.append(field)
```
The try/except seems like a good solution. Better to ask forgiveness and all that.

Resolving patch:
diff --git a/yt/data_objects/particle_trajectories.py b/yt/data_objects/particle_trajectories.py
--- a/yt/data_objects/particle_trajectories.py
+++ b/yt/data_objects/particle_trajectories.py
@@ -18,6 +18,7 @@
 from yt.funcs import mylog, get_pbar
 from yt.units.yt_array import array_like_field
 from yt.config import ytcfg
+from yt.utilities.exceptions import YTIllDefinedParticleData
 from collections import OrderedDict
 
 import numpy as np
@@ -120,7 +121,13 @@ def __init__(self, outputs, indices, fields=None, suppress_logging=False):
         output_field.fill(np.nan)
         for field in ("particle_position_%s" % ax for ax in "xyz"):
             for i, (fn, (time, indices, pfields)) in enumerate(sorted(my_storage.items())):
-                output_field[indices, i] = pfields[field]
+                try:
+                    # This will fail if particles ids are
+                    # duplicate. This is due to the fact that the rhs
+                    # would then have a different shape as the lhs
+                    output_field[indices, i] = pfields[field]
+                except ValueError:
+                    raise YTIllDefinedParticleData("This dataset contains duplicate particle indices!")
             self.field_data[field] = array_like_field(
                 dd_first, output_field.copy(), fds[field])
             self.particle_fields.append(field)
diff --git a/yt/utilities/exceptions.py b/yt/utilities/exceptions.py
--- a/yt/utilities/exceptions.py
+++ b/yt/utilities/exceptions.py
@@ -686,6 +686,9 @@ def __str__(self):
         ).format(self.level, self.axis)
         return msg
 
+class YTIllDefinedParticleData(YTException):
+    pass
+
 class YTIllDefinedAMRData(YTException):
     pass
 


Test patch:
diff --git a/yt/data_objects/tests/test_particle_trajectories.py b/yt/data_objects/tests/test_particle_trajectories.py
--- a/yt/data_objects/tests/test_particle_trajectories.py
+++ b/yt/data_objects/tests/test_particle_trajectories.py
@@ -1,11 +1,17 @@
 import glob
 import os
+import numpy as np
+from numpy.testing import \
+    assert_raises
 
 from yt.config import ytcfg
 from yt.data_objects.time_series import DatasetSeries
 from yt.utilities.answer_testing.framework import \
     requires_ds, \
     GenericArrayTest
+from yt.testing import fake_particle_ds
+from yt.utilities.exceptions import YTIllDefinedParticleData
+
 
 def setup():
     ytcfg["yt","__withintesting"] = "True"
@@ -43,3 +49,18 @@ def test_etc_traj():
         def field_func(name):
             return traj[field]
         yield GenericArrayTest(ds, field_func, args=[field])
+
+def test_uniqueness():
+    n_particles = 2
+    n_steps = 2
+    ids = np.arange(n_particles, dtype=int) % (n_particles//2)
+    data = {'particle_index': ids}
+    fields = ['particle_position_x', 'particle_position_y', 'particle_position_z', 'particle_index']
+    negative = [False, False, False, False]
+    units = ['cm', 'cm', 'cm', '1']
+
+    ts = DatasetSeries([fake_particle_ds(fields=fields, negative=negative, units=units,
+                                         npart=n_particles, data=data)
+                        for i in range(n_steps)])
+
+    assert_raises(YTIllDefinedParticleData, ts.particle_trajectories, [0])
diff --git a/yt/testing.py b/yt/testing.py
--- a/yt/testing.py
+++ b/yt/testing.py
@@ -272,7 +272,7 @@ def fake_particle_ds(
                   "particle_velocity_z"),
         units = ('cm', 'cm', 'cm', 'g', 'cm/s', 'cm/s', 'cm/s'),
         negative = (False, False, False, False, True, True, True),
-        npart = 16**3, length_unit=1.0):
+        npart = 16**3, length_unit=1.0, data=None):
     from yt.frontends.stream.api import load_particles
 
     prng = RandomState(0x4d3d3d3)
@@ -287,6 +287,9 @@ def fake_particle_ds(
             offsets.append(0.0)
     data = {}
     for field, offset, u in zip(fields, offsets, units):
+        if field in data:
+            v = data[field]
+            continue
         if "position" in field:
             v = prng.normal(loc=0.5, scale=0.25, size=npart)
             np.clip(v, 0.0, 1.0, v)