Redo the font inheritance logic

commit 39e3851c caused lot of regressions. Redoing it