`autoscale(False)` works here (I'm not actually sure why plotting those points is rescaling since the points are no where near the boundary). `margins(0)` may be a little clearer.