Fix some life-cycle issues for backgroup OpenCL kernel compilation

Now it no longer asserts or crashes in light testing, but it doesn't
properly recalculate either.

Change-Id: I7dc117341eff5ac1b21d7400122cc6e04ae2baf7