Remove double dots from cache path without keys

This PR is a continuation of #5165 from @juriansluiman that was closed by the lack of tests.

I only have added tests cases to this feature.