In fact, <insertOlinkPdfFrag>1</insertOlinkPdfFrag> seems to be required.

<insertOlinkPdfFrag>1</insertOlinkPdfFrag> causes
external-destination="url(#dest=)" to be
external-destination="url(book.pdf#dest=chapter)".

I don't think I really know what I'm doing, but it seems to work.