Reintroducing fontconfig caching change, but skipping build-time fc-cache.
Build-time fc-cache fails when using the 'with-cache-dir' argument, which is needed as per reverted commit 0aac9b0881029bff125e11ae4bc7e4c223f828da
This also has the added advantage of speeding up packaging, since the cache is not used and takes over a minute to build.