Repository: pandas-dev/pandas
Pull request: ASV: DataFrame.__setitem__ performance with object dtype
- [ ] closes #19299 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x]  Added new benchmark SetitemObjectDtype

Problem statement:
DataFrame.__setitem__ performance regression with object dtype
This is between 0.20.3 and 0.21.1. Haven't narrowed further yet. Note that

- It's specific to setitem, getitem is fine
- It's specific to object dtype


```python
In [1]: import pandas as pd
   ...: pd.__version__
   ...:
Out[1]: '0.20.3'

In [2]: df = pd.DataFrame(index=range(1000), columns=range(100), dtype=object)

In [3]: %timeit df.loc[0, 1] = 1.0
132 µs ± 2.56 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```

```python
In [1]: import pandas as pd
   ...: pd.__version__
   ...:
Out[1]: '0.21.1'

In [2]: df = pd.DataFrame(index=range(1000), columns=range(100), dtype=object)

In [3]: %timeit df.loc[0, 1] = 1.0
1.64 ms ± 70.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```

This is present on master. I haven't tried profiling yet.

Hints from issue discussion:
Performance looks better now. Could use an asv

```
In [46]: In [2]: df = pd.DataFrame(index=range(1000), columns=range(100), dtype=object)
    ...:
    ...: In [3]: %timeit df.loc[0, 1] = 1.0
45.3 µs ± 313 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```
that's a cached result though but looks fine

Commit message: add github issue context

Patch to review:
diff --git a/asv_bench/benchmarks/indexing.py b/asv_bench/benchmarks/indexing.py
index d8b1bf327294a..fa0830a6143bc 100644
--- a/asv_bench/benchmarks/indexing.py
+++ b/asv_bench/benchmarks/indexing.py
@@ -515,6 +515,19 @@ def time_setitem_list(self):
         self.df[[100, 200, 300]] = 100
 
 
+class SetitemObjectDtype:
+    # GH#19299
+
+    def setup(self):
+        N = 1000
+        cols = 500
+        self.df = DataFrame(index=range(N), columns=range(cols), dtype=object)
+        self.df = DataFrame(np.random.rand(N, cols))
+
+    def time_setitem_object_dtype(self):
+        self.df.loc[0, 1] = 1.0
+
+
 class ChainIndexing:
     params = [None, "warn"]
     param_names = ["mode"]


Code review comments:
diff_hunk: @@ -515,6 +515,19 @@ def time_setitem_list(self):
         self.df[[100, 200, 300]] = 100
 
 
+class SetitemObjectDtype:
+    # GH#19299
+
+    def setup(self):
+        N = 1000
+        cols = 500
+        self.df = DataFrame(index=range(N), columns=range(cols), dtype=object)
+        self.df = DataFrame(np.random.rand(N, cols))
path: asv_bench/benchmarks/indexing.py
text: @user1:
Why is this assignment here? It's overriding the prior `self.df`