Repository: scipy/scipy

Issue:
scipy.spatial.SphericalVoronoi fails for large number of points
Trying to create a high density spherical grid. Starting with an icosphere mesh with ~276k points, SphericalVoronoi is successful (and slow). Moving up to ~500k points, SphericalVoronoi fails with:

    Traceback (most recent call last):

      File "<ipython-input-29-287623b904b6>", line 1, in <module>
        s = SphericalVoronoi(g.getPoints())

      File "c:\anaconda3\lib\site-packages\scipy\spatial\_spherical_voronoi.py", line 229, in __init__
        self._calc_vertices_regions()

      File "c:\anaconda3\lib\site-packages\scipy\spatial\_spherical_voronoi.py", line 268, in _calc_vertices_regions
        -1) == generator_indices).any(axis=1))

    AttributeError: 'bool' object has no attribute 'any'



Discussion:
@ktritz can you provide a self-contained example to reproduce the issue?

@tylerjereddy any ideas?
Sure, you can use this function to generated uniform points on the unit sphere:

```
def uniform(num):
    vec = np.random.randn(3, num)
    vec /= np.linalg.norm(vec, axis=0)
    return vec.T
```

SphericalVoronoi(uniform(1e5)) works, though it takes ~minute or longer to complete
SphericalVoronoi(uniform(2e5)) fails fairly quickly (few seconds) with the above error message

note, the points  I was testing on initially were an Icososphere grid rather than a random distribution, but it seems to have the same problem. With a little bit more testing, num = 1.44e5 works, 1.45e5 fails.
It's probably a memory allocation problem, because this part of line 268 in _spherical_voronoi.py:

(np.expand_dims(self._tri.simplices, -1) == generator_indices)

ends up creating an array with dimensions (2N-4, 3, N), which for N ~ 2e5 means an array with 240 billion points, and more memory than even my 32G can handle. For SphericalVoronoi to handle lots of points, this line will need to be more memory efficient.
Glad to hear people are actually using this code.

- The minute time scale performance is something I'm aware of. Nikolai and I knew of better algorithms when we wrote `SphericalVoronoi`, but as often happens in computational geometry, we implemented the one we could figure out! My plan for improving performance currently looks like this:
  - stage 1: (explore the possibility of) cythonizing the `SphericalVoronoi` algorithm to squeeze out as much performance as we can without having to reimplement the entire algorithm [you can see I've started simple with the point sorting component in #6768, but chipping away at other parts is on the to-do list]
  - stage 2: implement a cutting-edge algorithm with loglinear time complexity (there are a few of them, many in the last 10 years or so, so the math is understandably a bit tricky!)
- If I understand correctly, the more serious issue here is that `SphericalVoronoi` simply can't handle data sets on the order of 200,000 points. I'd propose:
  - the ideal solution: `SphericalVoronoi` would ideally not impose any additional constraints on the algorithm space complexity, beyond those imposed by `ConvexHull` (which is at the heart of `SphericalVoronoi`)-- which means we'd ideally be able to handle around 238 million points (see #6820 and links therein for details on `ConvexHull` point set limits)
  - more likely solution: I'm a bit skeptical I'll be able to find a solution that gives us a space complexity that gets up to the level of the highly-optimized `ConvexHull`, at least in the short term, but perhaps we can progressively raise the bar with improvements [I suspect performance is going to get pretty bad on the tens of millions of points scale because my benchmarks were consistent with quadratic time complexity for `SphericalVoronoi`, but would still be better if it worked rather than not at all] 

So, I'll start chipping away on a branch. The test code above should help, although I suspect a conventional unit test for the upper bound input tolerance would be a bit awkward, even with a slow decorator, because it would be slow / RAM stress a machine. What might work though is taking the part of the code that is limiting for space complexity (you appear to have done this already), and check the shape of the data structure that is produced for that code block (perhaps by abstracting it to another function that is then unit tested) using much smaller input.


Thanks for the detailed response. In the past, I've used a python interface to the STRIPACK algorithm, but I'm looking for a more native python solution, and I don't believe STRIPACK is in the public domain. Also, the FORTRAN is pretty awful, IMO, at least with regards to porting to a python solution. That said, I had thought it was an O(nlogn) algorithm, though I could be mistaken. If so, it may handle larger datasets better.

In the meantime, I've cheated a little bit. Because my icosphere grid algorithm already provides me with the generator points and their neighbors, plus all of the triangles are acute, I can directly construct the circumcircles and calculate the voronoi points without having to worry about any geometric testing. This has the advantage that the voronoi circumcenter calculation is linear in N, and takes about 60s for 100M points, using a python-ported numba-optimized routine from MeshKit. This will work for me for now, but obviously a more general solution would be great!

Resolving patch:
diff --git a/scipy/spatial/_spherical_voronoi.py b/scipy/spatial/_spherical_voronoi.py
--- a/scipy/spatial/_spherical_voronoi.py
+++ b/scipy/spatial/_spherical_voronoi.py
@@ -235,6 +235,7 @@ def _calc_vertices_regions(self):
 
         # add the center to each of the simplices in tri to get the same
         # tetrahedrons we'd have gotten from Delaunay tetrahedralization
+        # tetrahedrons will have shape: (2N-4, 4, 3)
         tetrahedrons = self._tri.points[self._tri.simplices]
         tetrahedrons = np.insert(
             tetrahedrons,
@@ -244,9 +245,11 @@ def _calc_vertices_regions(self):
         )
 
         # produce circumcenters of tetrahedrons from 3D Delaunay
+        # circumcenters will have shape: (2N-4, 3)
         circumcenters = calc_circumcenters(tetrahedrons)
 
         # project tetrahedron circumcenters to the surface of the sphere
+        # self.vertices will have shape: (2N-4, 3)
         self.vertices = project_to_sphere(
             circumcenters,
             self.center,
@@ -254,22 +257,26 @@ def _calc_vertices_regions(self):
         )
 
         # calculate regions from triangulation
-        generator_indices = np.arange(self.points.shape[0])
-        filter_tuple = np.where((np.expand_dims(self._tri.simplices,
-                                -1) == generator_indices).any(axis=1))
-
-        list_tuples_associations = zip(filter_tuple[1],
-                                       filter_tuple[0])
-
-        list_tuples_associations = sorted(list_tuples_associations,
-                                          key=lambda t: t[0])
+        # simplex_indices will have shape: (2N-4,)
+        simplex_indices = np.arange(self._tri.simplices.shape[0])
+        # tri_indices will have shape: (6N-12,)
+        tri_indices = np.column_stack([simplex_indices, simplex_indices,
+            simplex_indices]).ravel()
+        # point_indices will have shape: (6N-12,)
+        point_indices = self._tri.simplices.ravel()
+
+        # array_associations will have shape: (6N-12, 2)
+        array_associations = np.dstack((point_indices, tri_indices))[0]
+        array_associations = array_associations[np.lexsort((
+                                                array_associations[...,1],
+                                                array_associations[...,0]))]
 
         # group by generator indices to produce
         # unsorted regions in nested list
         groups = []
-        for k, g in itertools.groupby(list_tuples_associations,
+        for k, g in itertools.groupby(array_associations,
                                       lambda t: t[0]):
-            groups.append([element[1] for element in list(g)])
+            groups.append(list(list(zip(*list(g)))[1]))
 
         self.regions = groups
 


Test patch:
diff --git a/scipy/spatial/tests/test_spherical_voronoi.py b/scipy/spatial/tests/test_spherical_voronoi.py
--- a/scipy/spatial/tests/test_spherical_voronoi.py
+++ b/scipy/spatial/tests/test_spherical_voronoi.py
@@ -2,6 +2,7 @@
 import numpy as np
 import itertools
 from numpy.testing import (TestCase,
+                           assert_equal,
                            assert_almost_equal,
                            assert_array_equal,
                            assert_array_almost_equal)
@@ -121,6 +122,17 @@ def test_sort_vertices_of_regions_flattened(self):
         actual = list(itertools.chain(*sorted(sv.regions)))
         assert_array_equal(actual, expected)
 
+    def test_num_vertices(self):
+        # for any n >= 3, a spherical Voronoi diagram has 2n - 4
+        # vertices; this is a direct consequence of Euler's formula
+        # as explained by Dinis and Mamede (2010) Proceedings of the
+        # 2010 International Symposium on Voronoi Diagrams in Science
+        # and Engineering
+        sv = SphericalVoronoi(self.points)
+        expected = self.points.shape[0] * 2 - 4
+        actual = sv.vertices.shape[0]
+        assert_equal(actual, expected)
+
     def test_voronoi_circles(self):
         sv = spherical_voronoi.SphericalVoronoi(self.points)
         for vertex in sv.vertices: