Handle releasing set semantic pointers