Fix remote add cache

Followup of #5917. /cc @bjaglin @vieux