Fix non-default harfbuzz configuration logic (#4848)