Try harder to clean up the old dir

Because annoyingly we didn't have permission the other order.  Not
really sure how it ever worked for me.