build: Fix building environment when running cached build