addressbook: Strengthen an assertion to avoid negative bit shifts

This should never happen in practice, but it is good to be completely
explicit in the assertion.

Follow up to commit 10d55d18.

Coverity issue: #1250458

https://bugzilla.gnome.org/show_bug.cgi?id=730378