Remove the symbols when the minor mode is being disabled