Revert last fix, add different fix for -without-x

See issue https://github.com/fontforge/fontforge/pull/461 plus the earlier
https://github.com/fontforge/fontforge/pull/442 (1) for more info.