Handle closing a PR

@sarina is this what you were hoping for?