Add test for retry path and fix bugs it identified