BUG: Thoroughly dedup columns in read_csv
State: closed

When the user specifies `mangle_dupe_cols=False`, the columns should be thoroughly deduped.  However, `master` fails on examples like this:

~~~python
data = "a,a,a.1,a.1.1"
read_csv(StringIO(data))
   a  a.1  a.1.1  a.1.1
0  1    2      3      4
~~~
Note how there are two "a.1.1" columns.  Now, the output of those same commands is this:
~~~python
   a  a.1  a.1.1  a.1.1.1
0  1    2      3        4
~~~

Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=h1) Report
> Merging [#17060](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/9e6bb42fa50df808bffd60a665bf921e49b87032?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17060/graphs/tree.svg?height=150&src=pr&token=eZ4WkYLtcO&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17060      +/-   ##
==========================================
+ Coverage   91.03%   91.03%   +<.01%     
==========================================
  Files         161      161              
  Lines       49363    49366       +3     
==========================================
+ Hits        44936    44941       +5     
+ Misses       4427     4425       -2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.81% <100%> (+0.02%)` | :arrow_up: |
| #single | `40.25% <0%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.44% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.23% <0%> (-0.1%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.05% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=footer). Last update [9e6bb42...a8f13f5](https://codecov.io/gh/pandas-dev/pandas/pull/17060?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

lgtm.

no associated issue?
separately

- let's deprecate ``mangle_dupe_columns`` as a parameter (it should always be True as now).
- I think ``as_recarray`` can be removed now
- see about deprecating ``tupleize_cols`` 
> let's deprecate mangle_dupe_columns as a parameter (it should always be True as now).

What about #13262, which was opened directly after we disabled support for `mangle_dupe_cols=False` ?

> I think as_recarray can be removed now

Nope.  It's too early, as it was deprecated in `0.19.0` (see #6581)

> see about deprecating tupleize_cols

I keep on forgetting how many parameters we have in this function 😄 .  That's actually a good idea.  I don't know why people would want to NOT have a `MultiIndex` anymore.
> no associated issue?

Nope, accidentally stumbled upon this when working on reviving #6170.
@jreback : I was looking into deprecating `tupleize_cols` when I came across a curious contradiction: the parameter, when `True` means that we create `tuples` instead of `MultiIndex` (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)

However, when we create an `Index`, it has the opposite meaning (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Index.html) when passed in as `True` !

* Which one is correct?
* Do we eventually intend to deprecate this parameter from `Index` as well (if we are going to discourage it for `read_csv`) ?
Also, I see that this parameter as exists in `to_csv` ... it seems that for the sake of round-trip, we would also want to deprecate this parameter in this function too?
these should all be deprecated.
> these should all be deprecated.

Including the `Index` constructor one  (just to confirm)?
> Including the Index constructor one (just to confirm)?

I think so, with the idea the `Index([('a', 1)]) -> MultiIndex`, while `Index([('a', 1)], dtype=object)` returns a regular Index with tuples for values. xref https://github.com/pandas-dev/pandas/issues/17246