Also free this bufferevent if there's no context

Signed-off-by: Toon Schoenmakers <nighteyes1993@gmail.com>