Account for app stopping asynchronously

Signed-off-by: Abhijit Hiremagalur <abhi@pivotallabs.com>