Removing self from to_visit list