Revert "manifest: track our libphonenumber"

This reverts commit 4f04c7deaa8f47ca4b065d1a548523ab6b921a93.