Do the nice thing and remove arp entries after script execution