API: Add DataFrame.assign method
State: closed

Closes https://github.com/pydata/pandas/issues/9229

signature: `DataFrame.transform(**kwargs)`
- the keyword is the name of the new column (existing columns are overwritten if there's a name conflict, as in dplyr)
- the value is either
  - called on self if it's callable. The callable should be a function of 1 argument, the DataFrame being called on.
  - inserted otherwise

``` python
In [7]: df.head()
Out[7]: 
   sepal_length  sepal_width  petal_length  petal_width species
0           5.1          3.5           1.4          0.2  setosa
1           4.9          3.0           1.4          0.2  setosa
2           4.7          3.2           1.3          0.2  setosa
3           4.6          3.1           1.5          0.2  setosa
4           5.0          3.6           1.4          0.2  setosa

In [8]: (df.query('species == "virginica"')
           .transform(sepal_ratio=lambda x: x.sepal_length / x.sepal_width)
           .head())
Out[8]: 
     sepal_length  sepal_width  petal_length  petal_width    species  \
100           6.3          3.3           6.0          2.5  virginica   
101           5.8          2.7           5.1          1.9  virginica   
102           7.1          3.0           5.9          2.1  virginica   
103           6.3          2.9           5.6          1.8  virginica   
104           6.5          3.0           5.8          2.2  virginica   

     sepal_ratio  
100     1.909091  
101     2.148148  
102     2.366667  
103     2.172414  
104     2.166667  
```

My question now is
- How strict should we be on the shape of the `transformed` DataFrame? Should we do any kind of checking on the index or columns?


Labels:
name: API Design

Comments:
I like `transform` as the name; no-one in favor of `mutate`? any other possibilities?

only minus against transform is the use in groupby is _somewhat_ different.

I like `mutate` better because:
1. we already have `transform` on groupby
2. we might want `mutate` as grouped operation -- dplyr has it

If you intend to keep the function pure (as it is currently) then the term `mutate` might be misleading.

Neither `transform` nor `mutate` are very descriptive.  It may make sense to seek out a better term, even if it means breaking from tradition.

FWIW, dlpyr's mutate is also pure, despite the misleading name.

_le sigh_

Augment? Enhance? I'll keep thinking. 

> On Jan 13, 2015, at 18:32, Matthew Rocklin notifications@github.com wrote:
> 
> le sigh
> 
> —
> Reply to this email directly or view it on GitHub.

I think we should go a slightly different route here.

I would change the signature of `df.update(*args, **kwargs)`. Then you can detect if its the 'new' mode or the 'original' mode (you can simply look at the `args/kwargs` and figure this out unambiguously).

I would vote to effectively deprecate the original `.update` mode (of course ATM this would just show a `FutureWarning`. This function is really not necessary except in very rare circumstances, and is not well implemented internally.

`df.update(A=df.B/df.C)` looks like a winner to me. (same idea in that it IS pure, but has a more suggestive name than `transform/mutate`. Further we can add the same method to `groupby`).

That sounds pretty clean.

I wanted to say that `df.add_column()` is descriptive, but ugly name .. but I like `update` more! 
Only, the combination with the existing use seems a bit confusing to users I think (but have to look in more detail)

+1 for update.

I still vote for allowing multiple variables at once :).

OK, another idea: df.assign?

Update is not very useful in pandas, but it is part of the standard mapping API, where it's known as a method that does an in-place operation.

Thanks for the input. My favorite is `df.assign`, it conveys the meaning well. I could live with update though. I only worry about the cognitive clash with `dict.update()`, which is inplace.

I'll also allow multiple variables I think. I'm going to do all of the calculations before the assignment so that we don't run into issues with one calculation depending on another, and having the success or failure of the call dependent upon the dict ordering.

More tests and docs coming soon.

Updated with some docs and handling multiple assigns. I wasn't sure about the best place for the docs so I threw it in `basics.rst`. I still need to build them to make sure everything looks good.

I am worried about people hitting subtle bugs with assigning multiple columns in one `assign` since the order won't be preserved. I've tried to document that.

I've got a few more things to clean up and then I'll ping for review.

Ok, ready for feedback.

Just a summary,
- I went with `assign`, but update could work
- keyword arguments only (potentially multiple)
- if the value is callable, it's called on `self`
- if the value is not callable, it's inserted

Nice feature, and some points to be considered:
- `inplace` option like other functions (but it results we cannot create column named `inplace` though)
- Should `Series` also have `assign` to make a `DataFrame` for consistency?
- Better to care partial string slicing if `DataFrame` has `DatetimeIndex` and `PeriodIndex`, which outputs unexpected results.

- I'm -1 on an inplace option. Overall, I think we're discouraging users from using inplace these days. It also kills what I think is the main use of assign: inside a chain of operations.
- Series could have an assign. I didn't include it yet since that would _necessarily_ involve transforming a Series to a DataFrame, which we have the `to_frame` method for.
- For now I'm taking the approach that people need to be _very_ careful when using assign. I'm not doing any checking of the results to ensure that you're computation hasn't caused a reindexing that creates a bunch of NaNs.

I really think this should be called `.update`. Adding _another_ function is just confusing. 

@jreback I'm all for deprecating .update, but I think it is clearer to give them a distinct name, given that the behavior is different and also different from the update method on dicts.

Another name possibly worth considering is `.set`.

@shoyer we already have way too many update/set/filter etc methods. I think some consolidation is in order. I don't think adding another new method is useful at all. 

And you dont' actually need to deprecate the current functionaility of `.update`. (e.g. `other` is a dict-like).

@jreback I agree on the problem but not the solution (in this specific case). Taking the deprecation and eventual removal of the current meaning of update as a given, I would rather have an assign method than an update method, because the later will always have the confusing association with duct.update (the issue is similar to the name mutate but worse). But is mostly bike shedding. 

@jreback To follow up on your edit, I am pretty opposed to functions that do very different things depending on how you call them, except as a transitional step. That is poor API design :).

@shoyer fair enough, and I DO like `.set` :)

@shoyer update to my update (pun intended). I am pushing this because `.update` functionaility is currently completely encompassed by `.loc` (maybe with a couple of edge cases), and is not handled correctly internally (e.g.its a giant hack right now). 

So I would view this as a net positive to remove it and simplify the API.

>  `.update` functionaility is currently completely encompassed by .loc (maybe with a couple of edge cases), and is not handled correctly internally (e.g.its a giant hack right now). 

How would you exactly do such an update with `loc`? Say an `df1.update(df2)` is equivalent to `df1.loc[df2.index, df2.columns] = df2`. I don't know if that is so obvious for users to do?

This could be an argument from authority, but Wes does breifly use `DataFrame.update` in The Book (page 338), as an alternative to `.combine_first`. I'm think we're trying to avoid breaking examples from that.

Picking this up again today.

I'm even mored down on `update` than before, since the couple releases where there's overlap will be very muddled. With `df.update(other_df)` being inplace since `other_df` is positional, but `df.update(a=other_thing`) being the new behavior since it's a keyword. Just in my testing and putting together I've accidentally forgotten the keyword argument multiple times. Having that unexpectedly update my data, potentially without me realizing, would suck.

I was thinking maybe it'd be OK since DataFrames aren't dict-like, so `DataFrame.update` shouldn't have to much cognitive dissonance with dict.update, but we also have a `Seris.update`, and Series _are_ dict-like.

So I guess what I'm saying is I want to go with `.assign`. (Or maybe [enhance](http://giphy.com/gifs/enhance-yekZxRr4nM1q0) (NSFWish I guess, it's the Super Troopers reference) for the comedic value of chaining a bunch of enhances together, :laughing: ).

Thanks. I'm building the docs now.

@jreback thoughts on keeping it at `.assign` vs. changing to `.update`?

I like the overall idea and your tests and docs looks nice. can you put up a list of the names to consider one more time? (eg assign,mutate,update...)

Sure thing
- `mutate`
  - pros: same as dplyr
  - cons: confusing since it [doesn't actually mutate](https://twitter.com/hadleywickham/status/559026527262617600)
- `assign`
  - pros: expresses idea of `assign`ing new columns well
  - cons: not as clear that the whole frame is returned. New method
- `update`
  - pros: again, pretty clear what we're doing
  - cons: break in API (eventually). Awkward period where we support old behavior on `*args` and new behavior with `**kwargs`. Clashes with `dict.update` which is inplace, while this is not.
- others: `augment`, `enhance`, `transform`, none of which really gained traction. Also `.set`, (thanks @shoyer )

My vote is for assign.

@TomAugspurger one more is `set`, which I mentioned above. (Though I agree, probably not an improvement over `assign`.)