Move arp fixer to just before runner

This way we don't run the risk of expiring our arp entry before the
tests have run.