When handling SIGINT, don't remove the signal source.

Prevents errors about not finding the source later, as we always
remove the source when the main loop is finished.