Better handle error cloning branches

When cloning with `--with-branches`, we should clean the folder only when an error happens when cloning the main branch (and not when initializing and fetching the branches)