No, typically they don’t return to the destroyed nest. However, you should remove the nest completely anyway.