Repository: pygfx/pygfx
Pull request #498 (2023-04-06 19:02:00)

Add frustum property to perspective camera
This PR adds a `camera.frustum` property to both the perspective and orthographic camera. It returns the camera's frustum as a set of corners/vertices of shape `(2, 4, 3)`. The first axis corresponds to the near/far plane, the second to the type of the corner (e.g., "bottom left"), and the third axis is that corner's 3D position in world space.

I've been using this while hunting bugs in the pylinalg integration PR, but realized that this might be useful on its own, so I am proposing it as a dedicated feature here. So far, I've used it to comprehend (and numerically access) the area of the scene that a camera has in view whenever the render itself just shows grey/background. It's surprisingly useful in figuring out if it is the camera or the WorldObject that is behaving funny. 

I also think it might come in handy in the future if we decide to do things like exclude objects from updating their GPU data if they are not in view by the main camera. The same idea might also work for skipping render passes of lights that are not in view.

Curious if you think this is useful, or if this already exists and I simply missed it.

FirefoxMetzger: The CI failures should resolve themself once we merge https://github.com/pygfx/pylinalg/pull/60
Korijn: I have no objections to merging this!

Typically if we were to implement a feature like camera frustum culling, you would implement an intersection test for bounding spheres/boxes against frustums (represented by six cutting planes), and use that to filter objects that don't intersect the camera frustum, to avoid rendering them.

I'm not sure if the corners would be useful for that operation. I don't think so.
FirefoxMetzger: > Do you think the points would help to generate the cutting planes? I'm curious to hear about your thoughts on this.

Absolutely. They are a full representation of the frustum, so we can convert it into any other representation we like. For example:

```python
import pygfx as gfx
import numpy as np
import pylinalg as la

# orthographic cameras are easier when checking the numerical values
# camera = gfx.PerspectiveCamera(70, 4 / 3, width=500)
camera = gfx.OrthographicCamera(500, 500)
camera.position.z = 500
camera.update_matrix_world()

# order:
# 0: (near, left, bottom)
# 1: (near, right, bottom)
# 2: (near, right, top)
# 3: (near, left, top)
# 4: (far, left, bottom)
# 5: (far, right, bottom)
# 6: (far, right, top)
# 7: (far, left, top)
corners_flat = camera.frustum.reshape((-1, 3))

# (left, right, top, bottom, near, far)
corners_by_plane = np.stack([
    corners_flat[[0, 3, 7, 4], :],
    corners_flat[[5, 6, 2, 1], :],
    corners_flat[[3, 2, 6, 7], :],
    corners_flat[[4, 5, 1, 0], :],
    corners_flat[[1, 2, 3, 0], :],
    corners_flat[[4, 7, 6, 5], :],
], axis=0)

# Various forms of cutting planes
# -------------------------------

# planes as sets of 3 points
planes_points = corners_by_plane[:, :3, :]

# planes as sets of (e_u, e_v, offset) where `point = u*e_u + v*e_v + offset`
# and both, u and v, are in [0, 1] respectively
planes_eq = np.stack([
    corners_by_plane[:, 0, :] - corners_by_plane[:, 3, :],
    corners_by_plane[:, 2, :] - corners_by_plane[:, 3, :],
    corners_by_plane[:, 3, :],
], axis=-1)

# planes in normal form (normals point away from the frustum area)
normals = np.cross(
    corners_by_plane[:, 0, :] - corners_by_plane[:, 3, :],
    corners_by_plane[:, 2, :] - corners_by_plane[:, 3, :]
)
normals /= np.linalg.norm(normals, axis=-1)[:, None] # normal normals ^_^
offset = np.sum(normals * corners_by_plane[:, 3, :], axis=-1)
planes_normal = np.hstack((normals, -offset[:, None]))


# Frustum-Sphere intersection
# ---------------------------

cube = gfx.Mesh(
    gfx.box_geometry(200, 200, 200),
    gfx.MeshPhongMaterial(color="#336699"),
)

sphere = cube.get_world_bounding_sphere()
radius = sphere[-1]
center_hom = la.vector_make_homogeneous(sphere[:-1])

signed_distance = np.sum(planes_normal * center_hom, axis=-1)
is_visible = np.all(signed_distance < radius)

if is_visible:
    print("Object visible")
else:
    print("Object NOT visible.")


```

> To support lights you would need to support more sophisticated shapes in the test. E.g. a spotlight needs a cone.

Hmm, can't we proxy this with a frustum-frustum intersection similar to how we would proxy a mesh-frustum intersection using a sphere-frustum intersection? I'm thinking this because we should be able to find a frustum that envelops the cone.
FirefoxMetzger: @almarklein Done. I've added a test for a camera with a 90-degree fov.

Patch:
diff --git a/pygfx/cameras/_perspective.py b/pygfx/cameras/_perspective.py
index 03925bb..7b8e665 100644
--- a/pygfx/cameras/_perspective.py
+++ b/pygfx/cameras/_perspective.py
@@ -456,6 +456,32 @@ class PerspectiveCamera(Camera):
         new_position = position + la.vector_apply_quaternion(offset, rotation)
         self.position.set(*new_position)
 
+    @property
+    def frustum(self):
+        """Corner positions of the viewing frustum in world space.
+
+        Returns
+        -------
+        frustum : ndarray, [2, 4, 3]
+            The coordinates of the frustum. The first axis corresponds to the
+            frustum's plane (near, far), the second to the corner within the
+            plane ((left, bottom), (right, bottom), (right, top), (left, top)),
+            and the third to the world position of that corner.
+
+        """
+
+        projection_matrix = self.projection_matrix.to_ndarray()
+
+        ndc_corners = np.array([(-1, -1), (1, -1), (1, 1), (-1, 1)])
+        depths = np.array((0, 1))[:, None]
+        local_corners = la.vector_unproject(
+            ndc_corners, projection_matrix, depth=depths
+        )
+        world_corners = la.vector_apply_matrix(
+            local_corners, self.matrix_world.to_ndarray()
+        )
+        return world_corners
+
 
 def fov_distance_factor(fov):
     # It's important that controller and camera use the same distance calculations


Test patch:
diff --git a/tests/cameras/test_cameras.py b/tests/cameras/test_cameras.py
index 606e251..c6601ce 100644
--- a/tests/cameras/test_cameras.py
+++ b/tests/cameras/test_cameras.py
@@ -124,3 +124,41 @@ def _run_for_camera(camera, near, far, check_halfway):
     assert np.allclose(pos_world3.to_array(), [0, 0, -far])
     if check_halfway:
         assert np.allclose(pos_world2.to_array(), [0, 0, -0.5 * (near + far)])
+
+
+def test_frustum():
+    unit_cube_corners = np.array(
+        [
+            [-1.0, -1.0, 1.0],
+            [1.0, -1.0, 1.0],
+            [1.0, 1.0, 1.0],
+            [-1.0, 1.0, 1.0],
+            [-1.0, -1.0, -1.0],
+            [1.0, -1.0, -1.0],
+            [1.0, 1.0, -1.0],
+            [-1.0, 1.0, -1.0],
+        ]
+    )
+
+    camera = gfx.OrthographicCamera(2, 2, depth_range=(-1, 1))
+    frustum_corners = camera.frustum
+
+    assert np.allclose(frustum_corners, unit_cube_corners.reshape(2, 4, 3))
+
+    expected_corners = np.array(
+        [
+            [-1.0, -1.0, -1.0],
+            [1.0, -1.0, -1.0],
+            [1.0, 1.0, -1.0],
+            [-1.0, 1.0, -1.0],
+            [-2.0, -2.0, -2.0],
+            [2.0, -2.0, -2.0],
+            [2.0, 2.0, -2.0],
+            [-2.0, 2.0, -2.0],
+        ]
+    )
+
+    camera = gfx.PerspectiveCamera(fov=90.0, aspect=1, depth_range=(1, 2))
+    frustum_corners = camera.frustum
+
+    assert np.allclose(frustum_corners, expected_corners.reshape(2, 4, 3))