libssh2_userauth_publickey authenticates with the public key bytes and a callback of the form:
int name(LIBSSH2_SESSION *session, unsigned char **sig, size_t *sig_len,   const unsigned char *data, size_t data_len, void **abstract);

You must then take the data that is passed in and sign that with something like RSA_sign for rsa keys and then assign that back to the sig parameter.

libssh2_userauth_publickey_fromfile works great if you just have the file path. But if you have a project that manages your ssh keys there is no need to go back to the filesystem in order to authenticate.