Flush the Jenkins handlers.

Also, do an explicit Jenkins restart if the private key has changed,
before notifiers run. Otherwise the handlers fail because the private
key isn't recognized by Jenkins yet.