As pointed out in my last review we should also update the config file regards a `by_application` section right below `by_branch`. That way we can directly specify on which hosts we want to run. But I think that gives us some extra work, which should be moved out to a new issue.