Fix remote add cache

Followup of #5917. /cc @bjaglin @vieux 
Fixes #3847