Only chown the build cache directory