Throwing points in geom_smooth set, when coordinate limits != c(0,1)