Making sure that we are cleaning up after ourselfes

Signed-off-by: Johannes Petzold <jpetzold@pivotallabs.com>