unref the elem after we are done with it