Ensure the app is really running before killing it

re-issuing #480 toward master