The "custom enchanting" part refers to plugins that customize enchanting via the EnchantItemEvent by clearing the list of default enchants and performing some custom functionality.

This particular special case and the note about "[not spending] xp if adding no enchants to a book" was to preserve the intended functionality of ac1a2d0, I chose to leave the way books are handled alone.

If there is a better or more elegant solution for this, or if the new functionality is now considered "correct", then that is fine and I can close this request- as @drXor  mentioned there is a simple (if inelegant) work-around for the issue that we're likely stuck using anyway since this is the behavior since 1.7.2 R0-3.