Fix select related crash by removing handles from active select set when worker thread is set to not running