Make the dockerfiles use cache better by not adding authorized_keys