Make sure that connection is closed when refused

Bug-Url: https://bugzilla.redhat.com/1131944
Change-Id: Ic093d1bf6553c8e387f5488f612b7bc1e509bef1
Signed-off-by: pkliczewski <piotr.kliczewski@gmail.com>