Check that we did not change values by downcasting

That check is probably not optimal, but at least it should not silently convert invalid indices into valid ones.