mainloop: Do a source check in source_remove also.