BuildQueue caches builds

Meaning changes to a build metadata would require a restart to take effect.