Remove app keys.