In part, we still use the primitive normals as rectangle normals. The extents are PCA'ed after the points are projected to the plane.

In box voter, we use only the most confident axis. We then PCA the projected points as initial guess of the other two axis and then search nearby axis for the smallest oriented bounding box.