treemap: cut down on swap_unwrap in remove

Performance before:

    std::treemap::TreeMap
                   sequential_ints 0.083971 s
                       random_ints 0.095861 s
                       delete_ints 0.083931 s
                sequential_strings 0.278272 s
                    random_strings 0.240286 s
                    delete_strings 0.173581 s

Performance after:

    std::treemap::TreeMap
                   sequential_ints 0.083297 s
                       random_ints 0.097644 s
                       delete_ints 0.052602 s
                sequential_strings 0.287326 s
                    random_strings 0.242372 s
                    delete_strings 0.142269 s