Fix runner issues and delete binary before running (fixes #706)