Show a kill app dialog when changing preferences