Don't send env vars from settings to the worker for PRs

travis-build should already reject them, but it's safer to not send
them to workers at all