fix i2d_X509_NAME() error checking

use int instead of size_t to store the return of i2d_X509_NAME(), fixing
a < 0 check for error.