Ensure that we close the connection if we got an error back

Otherwise we end up never re-opening the connection. oops.