Delete only the key that we created (fix #35)