ENH: Support strings containing '%' in add_prefix/add_suffix (#17151)
State: closed

 - [X] closes #17151
 - [X] tests added / passed
 - [X] passes ``git diff upstream/master -u -- "*.py" | flake8 --diff``
 - [X] whatsnew entry

Updated the `DataFrame.add_prefix` and `DataFrame.add_suffix` methods to use the new style `.format` syntax to perform the string formatting.  These previously used the old style string formatting, which raises when the prefix/suffix to add contains the '%' character.

Labels:
name: Enhancement

Comments:
@jschendel : Changes LGTM ATM.  Ping when tests pass.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=h1) Report
> Merging [#17162](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/90913306595c36facde65b6858ff94b6e6d51668?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17162/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17162      +/-   ##
==========================================
- Coverage   91.01%      91%   -0.02%     
==========================================
  Files         162      162              
  Lines       49498    49499       +1     
==========================================
- Hits        45053    45046       -7     
- Misses       4445     4453       +8
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.78% <100%> (ø)` | :arrow_up: |
| #single | `40.26% <33.33%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.03% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.66% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.33% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=footer). Last update [9091330...d51aac7](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=h1) Report
> Merging [#17162](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/90913306595c36facde65b6858ff94b6e6d51668?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17162/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17162      +/-   ##
==========================================
- Coverage   91.01%      91%   -0.02%     
==========================================
  Files         162      162              
  Lines       49498    49499       +1     
==========================================
- Hits        45053    45046       -7     
- Misses       4445     4453       +8
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.78% <100%> (ø)` | :arrow_up: |
| #single | `40.26% <33.33%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.03% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.66% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.33% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=footer). Last update [9091330...d51aac7](https://codecov.io/gh/pandas-dev/pandas/pull/17162?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

cc @jreback @jorisvandenbossche : Thoughts?
check in merge suffixes as well
@jreback : To ensure that I'm not misunderstanding, do you mean the `suffixes` parameter of `pd.merge`?  If so, it looks like using the '%' character is technically supported:

```
import pandas as pd
df1 = pd.DataFrame({'A': [1, 2, 3], 'B': ['foo', 'bar', 'baz']})
df2 = pd.DataFrame({'A': [1, 2, 3], 'B': list('xyz')})

pd.merge(df1, df2, on='A', suffixes=('%', '_%')) 

   A   B% B_%
0  1  foo   x
1  2  bar   y
2  3  baz   z
```

The underlying code that does the formatting appears to be [`items_overlap_with_suffix`](https://github.com/pandas-dev/pandas/blob/v0.20.3/pandas/core/internals.py#L4703-L4711), which is using old style formatting, but in a way that supports the '%' character.  Looks like there are few other additional places where prefixes/suffixes are added in a similar manner, with the '%' character being supported, for example [`_maybe_rename_join`](https://github.com/pandas-dev/pandas/blob/v0.20.3/pandas/core/sparse/frame.py#L730-L738) and[ `_get_dummies_1d`](https://github.com/pandas-dev/pandas/blob/v0.20.3/pandas/core/reshape/reshape.py#L1251).

I'd be happy to modify the code to use new style formatting for adding prefixes/suffixes in all the locations I see it occurring.  It seems like it might be better suited for a separate issue/pull request though?  It could also be an opportunity to add generic functions for adding prefixes/suffixes, then calling those functions in each place where prefixes/suffixes are added, in order to keep the prefix/suffix logic confined to single location.  I don't know if that's an appropriate design decision though.

thanks @jschendel 

certainly take a follow-up on points . you raised in: https://github.com/pandas-dev/pandas/pull/17162#issuecomment-320046941
(e.g. consolidation of these routines is a good thing)
From a API standpoint, I'm wondering why these two methods exist, at least at the top namespace of DataFrame/Series. Why not move them to the ``.str`` namespace or just do ``my_prefix + df.columns.astype(str) + my_suffix``?

Am I missing something?
that's a good point - these could be deprecated and added to .str; if no adverse affects

pls make a new issue for this (and PR would be great)
These are for renaming row or column labels, not for modifying the data of a Series. 
I'd be a little hesitant to deprecate them, if only because they allow for seamless method chaining.  IMO it's cleaner to do something like:
```
df = df.operation1(...)
       .operation2(...)
       .operation3(...)
       .add_prefix('foo')
```
compared to:
```
df = df.operation1(...)
       .operation2(...)
       .operation3(...)
df.columns = df.columns.str.add_prefix('foo')  
```
I suppose you could method chain with `pipe` if they were deprecated, but it still seems like it'd be messier.

Regardless of deprecation, it seems like it might be beneficial to add these methods to the `.str` namespace?  If so, I'd be happy to submit a PR implementing them, if @topper-123 doesn't want to.  Seems to line up well with the consolidation of prefix/suffix routines discussed earlier, though I'd want the `.str` methods implemented prior to doing the consolidation.
I'm ok with @jschendel doing this. Maybe I don't understand @jschendel correctly, but I'm +1 on deprecating these at the top namespace methods at the same time.

Thinking further and to add generality, I'm thinking why not add a ``.format`` method instead/also, which would format a arbitrary datatype into a given string. ``.format`` method would then take one argument (a string), and that string should have a ``{}`` component as per the standard Python format specification. 

This means you could do ``df.a_series = df.a_series.format("{:.2f} $")`` or similar string formatting. This is an large improvement IMO in readability over ``df.a_series = df.a_series.apply(lambda x: "{:.2f} $".format(x))``. 

If this could be vectorized we could also have some speedup, compared to doing this though ``.apply``, though I'm out of my depth on this.
I'm -1 on deprecating them.

The `.str.add_prefix` and `.str.add_suffix` are already covered by `df.col + other`

```python
In [9]: df.A + '_foo'
Out[9]:
0    aa_foo
1    bb_foo
2    cc_foo
Name: A, dtype: object
```

The `.format` method would be generally useful I think. @topper-123 would you mind opening an issue (or searching to see if it's been suggested before?)
BTW, if you want something similar as `add_prefix`/`add_suffix` (so renaming the column labels), but with the general functionality, you can pass a function to `rename`:

```
In [64]: df = pd.DataFrame([[1, 2], [3, 4]], columns=['A', 'B'])

In [65]: df.add_suffix('_1')
Out[65]: 
   A_1  B_1
0    1    2
1    3    4

In [66]: df.rename(columns=lambda x: x + '_1')
Out[66]: 
   A_1  B_1
0    1    2
1    3    4

In [67]: df.rename(columns='{}_1'.format)
Out[67]: 
   A_1  B_1
0    1    2
1    3    4
```

I never use  `add_prefix`/`add_suffix`, and if they didn't exist, I would not be in favor to add them given the above, but not sure it is worth to deprecate them.