Remove jenkins configuration. This should be done by hand.
Make sure we reload jenkins when we deploy.