Ensure the app is really running before killing it