improved geometry calculation for hex/ascii line width