Repository: pandas-dev/pandas

Issue:
Suggestion: Make assign accepts list of dictionaries
#### Problem description
I really like the assign function and it's ability to be applied in pipelines.
However, if you pass a dictionary via prefixed by **, the dictionary must only contain columns that already exist in the preceeding dataframe. So in a dataframe, that contains column 'A', and I want to construct column B as f(A) and column C = g(A, B), im forced to do
```python
# Your code here
pd.DataFrame({'A':[1, 2]}).assign(**{'B': lambda x:f(x['A'])}).assign(**{'C': lambda x:g(x['A'], x['B])})

       A    B     C
0      1    f(1)  g(1, f(1))
1      2    f(2)  g(2, f(2))
```
for some f and g and obtain a result like seen above. In extreme cases, this can lead to a lot of chained assign statements.

For convenience we could change the signature slightly to accept also *args, but every element in args should be such that the original assign function could be applied. In particular args could be a list of dictionaries.

With this, we could write the previous code as
```python
# Your code here
pd.DataFrame({'A':[1, 2]}).assign(*[{'B': lambda x:f(x['A']),{'C': lambda x:g(x['A'], x['B])}])
```
Of course (also as it is right now) the user is responsible to construct a correct "computational graph" here. Additionally, the implementation I currently think of would use len(args) (-1 intermediate) copies of the original dataframe. However, using the stacked procedure above, this also happens.

Thus, we obtain a simpler syntactic way of using assign and we don't break the original implementation.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.20.3
pytest: 3.2.1
pip: None
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
In my opinion, a cleaner solution to this would be #14207 (taking advantage of dict ordered-ness in 3.6+)
Agreed with @chris-b1.

Actually, one of the first things I checked was if the dict-orderedness can be used for this task. Unfortunately, this is not the case. 

However, if I understand the current source code correct, the naive implementation to use the dict-orderedness must perform a full data frame copy after each added keyword. In fact, in the worst case (e.g. as described above), this will inevitably be necessary. So for this case, we keep the notation simple and there is no speed tradeoff. 
But what if we have lots of assign statements that only depend on the already existing columns? Then, it seems, that we must perform a copy after each computed column.This could degrade performance in many (major) use cases. 
> However, if I understand the current source code correct, the naive implementation to use the dict-orderedness must perform a full data frame copy after each added keyword.

Not sure what you mean here, but `df.assign(a=1)` and `df.assign(a=1, b=2)`  each perform one copy.

The current implementation does two steps:

1. all the computation into an intermediate dictionary
2. all the assignments from that intermediate dictionary

If we allowed dependent assignment, we would interleave those two, compute and assign for each key - value pair. There would still be a single copy per `.assign`, but you won't have to break up assignments into multiple as often (thus saving copies).
Sorry for not being precise enough:

my original idea was to wrap a for loop around
https://github.com/pandas-dev/pandas/blob/c28b6249471ffddfed51d62437a65ca97766fb7f/pandas/core/frame.py#L2700-L2715
this would result in only a copy for each "list entry" of my suggestion.

Anyway, the "do all calculations first" section, will not be "valid" anymore for python 3.6, right?
for python 3.6, this could look like
```{python}
for k, v in kwargs.items():
    data[k] = = com._apply_if_callable(v, data)
```

for older versions of python, everything should remain the same. So the version distinction should be made directly after the copy statement, right?
Mind, if I make a pull request for that in the next couple of days?
>  this would result in only a copy for each "list entry" of my suggestion.

Thanks for the clarification, sorry I didn't get that sooner.

>  So the version distinction should be made directly after the copy statement, right?  Mind, if I make a pull request for that in the next couple of days?

Yeah, that sounds about right. Please do!
I'm also sorry, for not explaining my thought process better. You couldn't know what I know/knew about this issues, specifically since my contribution track record is almost non existing ;)

So, I just rearranged some of the tests and also some of the code.
It works fine if the arguments are callables. However, the shortened notation we used (df.assign(a=1, b=2)) can turn out to be df.assign(b=df.a, c=df.b), then of course df.b is not defined at the moment of the definition of the kwargs dictionary. Using callables has simply the advantage of evaluating the column names lazily.

Would it be okay, to only work well with callables?

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -2687,12 +2687,17 @@ def assign(self, **kwargs):
 
         Notes
         -----
-        For python 3.6 and above, the columns are inserted in the order of
-        \*\*kwargs. For python 3.5 and earlier, since \*\*kwargs is unordered,
-        the columns are inserted in alphabetical order at the end of your
-        DataFrame.  Assigning multiple columns within the same ``assign``
-        is possible, but you cannot reference other columns created within
-        the same ``assign`` call.
+        Assigning multiple columns within the same ``assign`` is possible.
+        For Python 3.6 and above, later items in '\*\*kwargs' may refer to
+        newly created or modified columns in 'df'; items are computed and
+        assigned into 'df' in order.  For Python 3.5 and below, the order of
+        keyword arguments is not specified, you cannot refer to newly created
+        or modified columns. All items are computed first, and then assigned
+        in alphabetical order.
+
+        .. versionmodified :: 0.23.0
+
+            Keyword argument order is maintained for Python 3.6 and later.
 
         Examples
         --------
@@ -2728,22 +2733,34 @@ def assign(self, **kwargs):
         7   8 -1.495604  2.079442
         8   9  0.549296  2.197225
         9  10 -0.758542  2.302585
+
+        Where the keyword arguments depend on each other
+
+        >>> df = pd.DataFrame({'A': [1, 2, 3]})
+
+        >>> df.assign(B=df.A, C=lambda x:x['A']+ x['B'])
+            A  B  C
+         0  1  1  2
+         1  2  2  4
+         2  3  3  6
         """
         data = self.copy()
 
-        # do all calculations first...
-        results = OrderedDict()
-        for k, v in kwargs.items():
-            results[k] = com._apply_if_callable(v, data)
-
-        # preserve order for 3.6 and later, but sort by key for 3.5 and earlier
+        # >= 3.6 preserve order of kwargs
         if PY36:
-            results = results.items()
+            for k, v in kwargs.items():
+                data[k] = com._apply_if_callable(v, data)
         else:
+            # <= 3.5: do all calculations first...
+            results = OrderedDict()
+            for k, v in kwargs.items():
+                results[k] = com._apply_if_callable(v, data)
+
+            # <= 3.5 and earlier
             results = sorted(results.items())
-        # ... and then assign
-        for k, v in results:
-            data[k] = v
+            # ... and then assign
+            for k, v in results:
+                data[k] = v
         return data
 
     def _sanitize_column(self, key, value, broadcast=True):


Test patch:
diff --git a/pandas/tests/frame/test_mutate_columns.py b/pandas/tests/frame/test_mutate_columns.py
--- a/pandas/tests/frame/test_mutate_columns.py
+++ b/pandas/tests/frame/test_mutate_columns.py
@@ -89,11 +89,35 @@ def test_assign_bad(self):
             df.assign(lambda x: x.A)
         with pytest.raises(AttributeError):
             df.assign(C=df.A, D=df.A + df.C)
+
+    @pytest.mark.skipif(PY36, reason="""Issue #14207: valid for python
+                        3.6 and above""")
+    def test_assign_dependent_old_python(self):
+        df = DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
+
+        # Key C does not exist at defition time of df
         with pytest.raises(KeyError):
-            df.assign(C=lambda df: df.A, D=lambda df: df['A'] + df['C'])
+            df.assign(C=lambda df: df.A,
+                      D=lambda df: df['A'] + df['C'])
         with pytest.raises(KeyError):
             df.assign(C=df.A, D=lambda x: x['A'] + x['C'])
 
+    @pytest.mark.skipif(not PY36, reason="""Issue #14207: not valid for
+                        python 3.5 and below""")
+    def test_assign_dependent(self):
+        df = DataFrame({'A': [1, 2], 'B': [3, 4]})
+
+        result = df.assign(C=df.A, D=lambda x: x['A'] + x['C'])
+        expected = DataFrame([[1, 3, 1, 2], [2, 4, 2, 4]],
+                             columns=list('ABCD'))
+        assert_frame_equal(result, expected)
+
+        result = df.assign(C=lambda df: df.A,
+                           D=lambda df: df['A'] + df['C'])
+        expected = DataFrame([[1, 3, 1, 2], [2, 4, 2, 4]],
+                             columns=list('ABCD'))
+        assert_frame_equal(result, expected)
+
     def test_insert_error_msmgs(self):
 
         # GH 7432