This fixes it, but probably in a stupid way.
Use font in whole buffer.
Maybe this fixes the weird re-positioning bug?