Remove fontconfig and freetype

Building fonts from the base pulls in fontconfig and freetype and we
don't want to do CI over them now.