Remove frustum allocation each frame.

For #1344