Properly remove the selection listener when necessary