Not at all. Nesting boxes do not have to be square-shaped. However, it should have room enough to house a laying hen, yet small enough to make them feel secure.  The shape of a nesting box shouldn't matter.