Fix DELETE action for an app. Closes #536