The BIG issue here is that conjugal rights are part of the polyamorous marriage contract.
There are some issues relating to children in situations where spouses opt out or divorce but, like business arrangements, there could be buy-outs or other agreed-to settlements.