If BN_num_bytes() returns a signed type, comparing it to (size_t, unsigned) buffer_length will provoke compiler warnings (comparison between signed and unsigned).  It is good to avoid those.

Is BN_num_bytes() very cheap ?  If not (or you don't know - it can be hard to tell, in Eric Young's code), how about calling it once and caching its answer. <code>
const size_t bnlen = BN_num_bytes(rsa->e);
if (buffer_length < bnlen) { buffer_length = bnlen; }
</code>