Repository: pandas-dev/pandas

Issue:
fillna not throwing error when limit is not positive
#### Code Sample

```python
import pandas as pd
import numpy as np
# Your code here
NCOLS = 3
NROWS = 50
data = {"col{}".format(int((i - NCOLS / 2) % NCOLS + 1)): np.random.randint(
            -100, 100, size=(NROWS)
        )
        for i in range(NCOLS)
}
df = pd.dataframe(data)
df.fillna(0, limit=-5)
```
#### Problem description

I am currently on pandas 0.24.2 and running the above code does not throw an error even though the limit is less than or equal to zero. This error does not arise when the `data` is made up of floats instead of ints (replace `np.random.randint` with `np.random.uniform`).

#### Expected Output

`ValueError: Limit must be greater than 0`

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Darwin
OS-release: 18.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: 4.3.0
pip: 19.0.3
setuptools: 40.8.0
Cython: 0.29.7
numpy: 1.15.0
scipy: 1.2.1
pyarrow: 0.14.0.RAY
xarray: 0.12.0
IPython: 7.3.0
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: 0.2.0
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
</details>


Discussion:
Thanks for the report. The validation should probably occur somewhere around https://github.com/pandas-dev/pandas/blob/c9182df84736ce060c30d386c9f3a97614ca7778/pandas/core/generic.py#L6036.


I've had a bit of a look and it appears that when randint is used, the fillna method in pandas/core/internals/blocks.py has self._can_hold_na set to false and returns before running the checks on limit, hence not raising the errors. Not entirely sure how _can_hold_na is set but it may have something to do with whether the DataFrame contains any NAs or not, as when I set one element in the above example to np.nan the error is thrown correctly.
We'll want to raise on invalid arguments, even if there aren't / can't be
any missing values.

On Wed, Jun 26, 2019 at 10:01 AM Inevitable-Marzipan <
notifications@github.com> wrote:

> I've had a bit of a look and it appears that when randint is used, the
> fillna method in pandas/core/internals/blocks.py has self._can_hold_na set
> to false and returns before running the checks on limit, hence not raising
> the errors. Not entirely sure how _can_hold_na is set but it may have
> something to do with whether the DataFrame contains any NAs or not, as when
> I set one element in the above example to np.nan the error is thrown
> correctly.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/27042?email_source=notifications&email_token=AAKAOIRAACK6CE2PDO4FIYDP4OADTA5CNFSM4H3LWVK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYT2KFQ#issuecomment-505914646>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOISKTIWHHI3T3QJLAELP4OADTANCNFSM4H3LWVKQ>
> .
>

I've drilled into it a little bit more and it appears that _can_hold_na is set at the class level and is set to True for FloatBlock and False for IntBlock which is causing the difference in behaviour. I propose a change to the function: 
https://github.com/pandas-dev/pandas/blob/6119d0253e013b31de7d050a52af2226e864d304/pandas/util/_validators.py#L325
such that it takes an additional arugment limit, and performs the necessary checks in there. These checks would be those that are already implemented at 
https://github.com/pandas-dev/pandas/blob/a7f1d69b135bbbf649cf1af9a62d79acb963e47c/pandas/core/internals/blocks.py#L355
Happy to try and implement myself, just want to make sure this make sense?
That seems reasonable.

On Wed, Jun 26, 2019 at 11:17 AM Inevitable-Marzipan <
notifications@github.com> wrote:

> I've drilled into it a little bit more and it appears that _can_hold_na is
> set at the class level and is set to True for FloatBlock and False for
> IntBlock which is causing the difference in behaviour. I propose a change
> to the function:
>
> https://github.com/pandas-dev/pandas/blob/6119d0253e013b31de7d050a52af2226e864d304/pandas/util/_validators.py#L325
> such that it takes an additional arugment limit, and performs the
> necessary checks in there. These checks would be those that are already
> implemented at
>
> https://github.com/pandas-dev/pandas/blob/a7f1d69b135bbbf649cf1af9a62d79acb963e47c/pandas/core/internals/blocks.py#L355
> Would this make sene?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/27042?email_source=notifications&email_token=AAKAOIQQB4GPLWMZ3ASXRM3P4OJB7A5CNFSM4H3LWVK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYUBXZQ#issuecomment-505945062>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIWYWI7IK3TTJ2J5HZ3P4OJB7ANCNFSM4H3LWVKQ>
> .
>


Resolving patch:
diff --git a/pandas/core/internals/blocks.py b/pandas/core/internals/blocks.py
--- a/pandas/core/internals/blocks.py
+++ b/pandas/core/internals/blocks.py
@@ -344,12 +344,6 @@ def fillna(self, value, limit=None, inplace=False, downcast=None):
         """
         inplace = validate_bool_kwarg(inplace, 'inplace')
 
-        if not self._can_hold_na:
-            if inplace:
-                return self
-            else:
-                return self.copy()
-
         mask = isna(self.values)
         if limit is not None:
             if not is_integer(limit):
@@ -361,6 +355,12 @@ def fillna(self, value, limit=None, inplace=False, downcast=None):
                                           "is currently limited to 2")
             mask[mask.cumsum(self.ndim - 1) > limit] = False
 
+        if not self._can_hold_na:
+            if inplace:
+                return self
+            else:
+                return self.copy()
+
         # fillna, but if we cannot coerce, then try again as an ObjectBlock
         try:
             values, _ = self._try_coerce_args(self.values, value)


Test patch:
diff --git a/pandas/tests/frame/test_missing.py b/pandas/tests/frame/test_missing.py
--- a/pandas/tests/frame/test_missing.py
+++ b/pandas/tests/frame/test_missing.py
@@ -516,6 +516,22 @@ def test_fillna_skip_certain_blocks(self):
         # it works!
         df.fillna(np.nan)
 
+    @pytest.mark.parametrize("type", [int, float])
+    def test_fillna_positive_limit(self, type):
+        df = DataFrame(np.random.randn(10, 4)).astype(type)
+
+        msg = "Limit must be greater than 0"
+        with pytest.raises(ValueError, match=msg):
+            df.fillna(0, limit=-5)
+
+    @pytest.mark.parametrize("type", [int, float])
+    def test_fillna_integer_limit(self, type):
+        df = DataFrame(np.random.randn(10, 4)).astype(type)
+
+        msg = "Limit must be an integer"
+        with pytest.raises(ValueError, match=msg):
+            df.fillna(0, limit=0.5)
+
     def test_fillna_inplace(self):
         df = DataFrame(np.random.randn(10, 4))
         df[1][:4] = np.nan