I think I had the wrong pem file.

Also, I was returning the wrong value with verify_signature.  Oops!