Remove app keys