You have the upper hand here. After all, the kids will go back to where they came from and you'll have your man just the way you did before. A loving relationship with a good man is too hard to come by to blow it off over a couple of pre-schoolers. Don't complain about the kids. Instead just laugh it off.
Kids can be mean.
So learn to ignore them. If you let them think they can bother you, they'll do it more. In time, the kids will get used to you, if you hang around. Don't push yourself on them. But don't hide from them either.