Repository: pandas-dev/pandas
Pull request: Add asv benchmarks for Block.setitem()
- [x] closes #25756 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I believe that this solves the issue , however let me know if i need to change anything as i am not familliar with asv benchmarks

Problem statement:
Extremely Slow `Block.setitem()` when `value` is a builtin python type.
#### Code Sample

This code example is simply calling repeated `setitem` via a `LocIndexer` on a reasonably sized DataFrame of all bools. The key is that it takes exponentially longer to call this when you're setting a python `bool` type value versus a `np.array[bool]` value.

```python
from datetime import datetime
import pandas as pd
import numpy as np

# Create dataframe of 500 columns, 1 year of dates, all False values
df = pd.DataFrame(
    False,
    columns=np.arange(500).astype(str),
    index=pd.date_range('2010-01-01', '2011-01-01')
)

def test(true_value):
    """ Test time for assigning a slice `True` and `np.array(True)` """
    tmp_df = df.copy()
    
    start = datetime(2010, 5, 1)
    end = datetime(2010, 9, 1)
    tmp_df.loc[start:end, :] = true_value

print("True")
%timeit test(True)
print("\nnp.array(True)")
%timeit test(np.array(True))
```
```
Output:

True
3.4 ms ± 39.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

np.array(True)
512 µs ± 4.91 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```
#### Problem description
A code profiling reveals that almost all the computation time in the longer run is due to `ensure_object` being called in `cast.py:78` from a `Block.setitem`. 

In the above example, a bool `True` is passed to the `blocks.py:Block.setitem(...)` method via `LocIndexer`. Since a builtin bool has no `dtype` attribute, the dtype is set to `infer`. This causes the `cast.py:maybe_downcast_to_dtype(...)` to be called with `dtype == 'infer'`, and if the dataset is large the `ensure_object(...)` method takes a _long_ time to run. Wrapping the value in a numpy array `np.array(True)` exponentially reduces the computation since it then has a `dtype`. 

This behavior did not exist in older versions of pandas (pre-20ish). Could we do some pre-defined type checking in `Block.setitem(...)` to infer builtin python dtypes instead of requiring `dtype` attributed objects?

I would be happy to code a fix for this, but I am unsure of the reasoning behind only supporting objects with a `dtype` attribute as a value.


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Darwin
OS-release: 18.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.25.0.dev0+266.g707c7201a
pytest: None
pip: 19.0.3
setuptools: 40.8.0
Cython: 0.29.6
numpy: 1.16.2
scipy: None
pyarrow: None
xarray: None
IPython: 7.3.0
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>

Hints from issue discussion:
Updated code snippet, had a typo.
Could you simplify your performance test to just highlight the assignment portion (if thats the main issue)? I don't see how the loop is really relevant (and not really idiomatic to assign values in a DataFrame with looping).
Apologies for the poor example. I've simplified the code to illustrate the point more directly. 
Thanks for simplifying the example.

When assigning arrays with only value, I could see an optimization where we don't need to cast to object and treat the assignment the same as a scalar. Definitely feel free to put a PR!
These looks comparable on master now. Could use a ASV benchmark.

```
In [16]: from datetime import datetime
    ...: import pandas as pd
    ...: import numpy as np
    ...:
    ...: # Create dataframe of 500 columns, 1 year of dates, all False values
    ...: df = pd.DataFrame(
    ...:     False,
    ...:     columns=np.arange(500).astype(str),
    ...:     index=pd.date_range('2010-01-01', '2011-01-01')
    ...: )
    ...:
    ...: def test(true_value):
    ...:     """ Test time for assigning a slice `True` and `np.array(True)` """
    ...:     tmp_df = df.copy()
    ...:
    ...:     start = datetime(2010, 5, 1)
    ...:     end = datetime(2010, 9, 1)
    ...:     tmp_df.loc[start:end, :] = true_value
    ...:
    ...: print("True")
    ...: %timeit test(True)
    ...: print("\nnp.array(True)")
    ...: %timeit test(np.array(True))
True
218 µs ± 3.52 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

np.array(True)
233 µs ± 18.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```
Hi @kykosic @mroeschke ,

I just starting out on my open source journey. I'm interested in the issue. Is there any way you guys can assign the issue to me?
take
@mroeschke
The issue seems to be fixed. Is there anything that I could help with, since the issue is still open?

Commit message: change name to mode

Patch to review:
diff --git a/asv_bench/benchmarks/indexing.py b/asv_bench/benchmarks/indexing.py
index 53827cfcf64fb..fea45298adaba 100644
--- a/asv_bench/benchmarks/indexing.py
+++ b/asv_bench/benchmarks/indexing.py
@@ -531,4 +531,28 @@ def time_chained_indexing(self, mode):
                 df2["C"] = 1.0
 
 
+from datetime import datetime
+
+
+class Block:
+    params = [
+        (True, "True"),
+        (np.array(True), "np.array(True)"),
+    ]
+
+    def setup(self, true_value, mode):
+        self.df = DataFrame(
+            False,
+            columns=np.arange(500).astype(str),
+            index=date_range("2010-01-01", "2011-01-01"),
+        )
+
+        self.true_value = true_value
+
+    def time_test(self, true_value, mode):
+        start = datetime(2010, 5, 1)
+        end = datetime(2010, 9, 1)
+        self.df.loc[start:end, :] = true_value
+
+
 from .pandas_vb_common import setup  # noqa: F401 isort:skip


Code review comments:
diff_hunk: @@ -531,4 +531,28 @@ def time_chained_indexing(self, mode):
                 df2["C"] = 1.0
 
 
+from datetime import datetime
path: asv_bench/benchmarks/indexing.py
text: @user1:
Can you move this to the top of the file