Add missing infinity checks

This adds infinity checks when serializing or recovering public keys, and a test for it.

Either fix is enough to solve #109.