BUG: Fixed TypeError for Series.isin() when large series and values contains NA (#60678)
State: closed

- [x] closes #60678 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

## Issue
[`Series.isin()`](https://pandas.pydata.org/docs/reference/api/pandas.Series.isin.html) raises `TypeError: boolean value of NA is ambiguous` when Series is large enough (>[1_000_000](https://github.com/pandas-dev/pandas/blob/main/pandas/core/algorithms.py#L477)) and `values` contains `NA`

## Reason
- `Series.isin()` internally uses `np.isin()` for large series with smaller values to increase performance but it does not handles the case when values is of `dtype=object` and contains `NA` and passes it to `np.isin`

https://github.com/pandas-dev/pandas/blob/a4e814954b6f1c41528c071b028df62def7765c0/pandas/core/algorithms.py#L543-L568

- [`np.isin()`](https://numpy.org/doc/stable/reference/generated/numpy.isin.html) internally calls [`in1d()`](https://numpy.org/doc/stable/reference/generated/numpy.in1d.html#numpy.in1d) which uses `==` sign with `NA` . 

```python
            mask = np.zeros(len(ar1), dtype=bool)
            for a in ar2:
                mask |= (ar1 == a)
 ```
- Using ar1 == NA raises a `TypeError` because the boolean value of `pd.NA` is ambiguous.  [refer docs](https://pandas.pydata.org/docs/user_guide/missing_data.html#na-in-a-boolean-context).

## Fix Implemented
Explicitly checking if `values` contains `NA` when large series and small number of values (<= 26) to avoid using `np.isin` in `algorithms.py`.

```python
from pandas._libs.missing import NA

def isin(comps: ListLike, values: ListLike) -> npt.NDArray[np.bool_]:

   
    # GH60678
    # Ensure values don't contain <NA>, otherwise it throws exception with np.in1d
    values_contains_NA = False
    
    if comps_array.dtype != object and len(values) <= 26:  
        values_contains_NA = any(v is NA for v in values)

    if (
        len(comps_array) > _MINIMUM_COMP_ARR_LEN
        and len(values) <= 26
        and comps_array.dtype != object
        and values_contains_NA == False
    ):

```

## Testing
Successfully pass all existing test cases in `test_isin.py` with tests added for large series with `dtype` as `boolean`, `Int64` and `Float64` as follow: 

1. Series `dtype==boolean` and `values` contain `pd.NA` 
2. Series `dtype==boolean` and `values` contains mixed data with `pd.NA`
3. Series `dtype==boolean` and `values` empty
4. Series `dtype==Int64` and `values` contains `pd.NA`
5. Series `dtype==Float64` and `values` contains `pd.NA`


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Thanks @akj2018