It really depends on the insurer.
There are many insurers who will not provide any cover for the cost of childbirth or expenses related to the medical care of a newborn baby. If you select such a policy and your baby is born overseas, keep in mind that you will most likely face extensive out-of-pocket expenses and, depending on where you're travelling, may not be able to access the best possible level of care for your baby.
However, some insurers will provide cover if your baby arrives early, as long as your little bundle of joy is born inside the insurer's maximum weeks of pregnancy permitted time limit. As always, check the PDS closely to read the full terms and conditions and contact the insurer directly if there's anything you're unsure about.