Setup foreman provisioning and some other minor improvements