Use set-frame-font instead of set-default-font.