Revert "don't remove apps from app_runner until they're deleted"

This reverts commit 7609672926f6d18783ff25b7ab349394a85a41d9.