Only frees pointers we own