I keep waiting for my Bess to start another nest. Have seen both adults around the box a couple of times. Have chased away wasps twice. It later flew to the top of the dead tree where Harry likes to perch and was chased away by a red-wing blackbird. Oh, boy, afraid things aren't going to go so smoothly this time.
Some people also try a VERY thin layer of vaseline.
Anything to keep them from being able to attach a nest. The birds won't use a box with a wasp or nest in it. I also once saw a red-tailed hawk on one of my bluebird boxes. It was amazing to see it so close. Like yours, it didn't stay.