Repository: pandas-dev/pandas

Issue:
DEPR: filter & select
do we need label selectors? we should for sure just have a single method for this. maybe call it `query_labels`? to be consistent with `.query` as the workhorse for data selection.

- [x] ``.select`` (#17633)
- [ ] ``.filter``

xref #6599 



Discussion:
I personally find `filter` a useful function (at least I have used it to good purpose in my own work) to *select* certain columns. See also the examples added in https://github.com/pandas-dev/pandas/pull/12399. Although it should rather be called `select` ...

Less sure about `select`. That seems less useful, certainly now `loc` accepts a function. 
I think I have revised my thoughts here.

we should promote (in the doc / the-one-way-to-do-it), ``.select`` as the main *label* filtering function, and deprecate ``.filter`` (which ATM serve the same purpose). Maybe needs some API tweaks. 

``.filter`` is traditionally a *data* selection / filtering function.
They are quite different at the moment:

- `filter`:
  - acts on columns by default (for dataframe)
  - can select based on list, or simple 'like'/more advanced regex
- `select`:
  - acts on index by default
  - selects based on function applied to index labels
further: ``.filter`` uses ``.select`` for regex matching in its implementation.
further: we use ``.filter()`` in ``.groupby()`` to allow a filter for group inclusion (boolean return)
I have found `DataFrame.filter` to be useful, especially with `like` or `regex`. I have never used `DataFrame.select`, which feels very non-idiomatic to me.

So I would be happy to deprecate `select`. It's also highly confusing how `GroupBy.filter` works like `DataFrame.select`, not `.filter`.
>  It's also highly confusing how `GroupBy.filter` works like `DataFrame.select`, not `.filter`.

I agree this is highly confusing. Is renaming one of those out of the question? `filter` is a common name for a higher-order function which filters elements based on the result of a Boolean-valued function that was passed in, exactly like `GroupBy.filter`, so that seems like an appropriate name for what is currently `DataFrame.select`. There's also Python's builtin `filter` function.

Another option might be merging the functionality of `select` and `filter` under one name, so it supports both list-like and function arguments.
so the problem as highlited by @jorisvandenbossche is that ``.select`` acts on the index (which is what ``groubpy.filter`` and boolean selection does). so it is a highly confusing name.

``.filter`` is also a confusing name as it acts on the labels of columns.

We need a combined functionaility of the current ``DataFrame.select/filter`` (IOW to select *labels* from an axis and should accept a list-like, scalar and callable, like most other functions)

signature should be something like this (default for most functions is axis=0)

```
def select_labels(arraylike or scalar or callable, axis=0, regex=False)
```

now as to what to do:

- ``select_labels`` I think is a nice name (open to suggestions), though other systems (spark & sql), use ``.select`` to mean label/column selection.
- deprecate ``.select`` in favor of ``.select_labels``
- deprecate ``.filter`` in favor of ``select_labels``

@dkasak interested in taking this on?
I would suggest simply deprecating/removing `select` without making a replacement. Indexing is a fine alternative.

`DataFrame.filter()` is useful. I wish it were called `select` instead, both because that matches SQL and `filter` suggests filtering rows with a boolean expression (like `filter` in dplyr or Ibis), but I don't think changing the name is worth the hassle. 

In general, I think we should avoid making small changes in the API for the basic grammar of data manipulation in pandas, unless we rethink things more broadly for a larger, breaking change (e.g., in pandas2).
>  but I don't think changing the name is worth the hassle

sure it is - pandas is going to exist for 1.x for quite some time 

better to make changes to the right spelling sooner rather than later 

I am all for deprecating filter and calling it select (or select_labels)

I don't have time to handle this at the moment, but I may be interested in doing it when time permits if it hasn't been done already by then.

FWIW, upon some thought, I still think changing the name of `.filter` to `.select*` would be best. I don't feel strongly about `.select` vs `.select_labels`. I generally prefer shorter names, but the added verbosity here might make things clearer. Calling it `.select` has the benefit that only one name is deprecated, not two.

I'm not so sure about dropping the current behaviour of `.select` entirely because I have a use case which I'm not sure how to implement without it (and without resorting to things like `.reset_index()` to regain the ability to select by using a function).

In particular, I have a `MultiIndex` with 2 levels, each of which has elements of type `str`. In other words, each index value is conceptually a pair of strings. Currently I'm doing something like

    df.select(lambda x: condition1(x[0]) and condition2(x[1]))

and similar to select particular rows. How could this be implemented without current `.select` functionality?


can u show a complete example of how using select 

Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -445,9 +445,17 @@ def _apply_if_callable(maybe_callable, obj, **kwargs):
     """
     Evaluate possibly callable input using obj and kwargs if it is callable,
     otherwise return as it is
+
+    Parameters
+    ----------
+    maybe_callable : possibly a callable
+    obj : NDFrame
+    **kwargs
     """
+
     if callable(maybe_callable):
         return maybe_callable(obj, **kwargs)
+
     return maybe_callable
 
 
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -2339,6 +2339,8 @@ def select(self, crit, axis=0):
         """
         Return data corresponding to axis labels matching criteria
 
+        DEPRECATED: use df.loc[df.index.map(crit)] to select via labels
+
         Parameters
         ----------
         crit : function
@@ -2349,6 +2351,11 @@ def select(self, crit, axis=0):
         -------
         selection : type of caller
         """
+        warnings.warn("'select' is deprecated and will be removed in a "
+                      "future release. You can use "
+                      ".loc[labels.map(crit)] as a replacement",
+                      FutureWarning, stacklevel=2)
+
         axis = self._get_axis_number(axis)
         axis_name = self._get_axis_name(axis)
         axis_values = self._get_axis(axis)
@@ -3101,7 +3108,7 @@ def filter(self, items=None, like=None, regex=None, axis=None):
 
         See Also
         --------
-        pandas.DataFrame.select
+        pandas.DataFrame.loc
 
         Notes
         -----
@@ -3120,20 +3127,23 @@ def filter(self, items=None, like=None, regex=None, axis=None):
 
         if axis is None:
             axis = self._info_axis_name
-        axis_name = self._get_axis_name(axis)
-        axis_values = self._get_axis(axis_name)
+        labels = self._get_axis(axis)
 
         if items is not None:
-            return self.reindex(**{axis_name:
-                                   [r for r in items if r in axis_values]})
+            name = self._get_axis_name(axis)
+            return self.reindex(
+                **{name: [r for r in items if r in labels]})
         elif like:
-            matchf = lambda x: (like in x if isinstance(x, string_types) else
-                                like in str(x))
-            return self.select(matchf, axis=axis_name)
+            def f(x):
+                if not isinstance(x, string_types):
+                    x = str(x)
+                return like in x
+            values = labels.map(f)
+            return self.loc(axis=axis)[values]
         elif regex:
             matcher = re.compile(regex)
-            return self.select(lambda x: matcher.search(str(x)) is not None,
-                               axis=axis_name)
+            values = labels.map(lambda x: matcher.search(str(x)) is not None)
+            return self.loc(axis=axis)[values]
         else:
             raise TypeError('Must pass either `items`, `like`, or `regex`')
 
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -99,6 +99,8 @@ def __call__(self, axis=None):
         # we need to return a copy of ourselves
         new_self = self.__class__(self.obj, self.name)
 
+        if axis is not None:
+            axis = self.obj._get_axis_number(axis)
         new_self.axis = axis
         return new_self
 
@@ -107,7 +109,8 @@ def __iter__(self):
 
     def __getitem__(self, key):
         if type(key) is tuple:
-            key = tuple(com._apply_if_callable(x, self.obj) for x in key)
+            key = tuple(com._apply_if_callable(x, self.obj)
+                        for x in key)
             try:
                 values = self.obj.get_value(*key)
                 if is_scalar(values):
@@ -117,10 +120,16 @@ def __getitem__(self, key):
 
             return self._getitem_tuple(key)
         else:
+            # we by definition only have the 0th axis
+            axis = self.axis or 0
+
             key = com._apply_if_callable(key, self.obj)
-            return self._getitem_axis(key, axis=0)
+            return self._getitem_axis(key, axis=axis)
+
+    def _get_label(self, label, axis=None):
+        if axis is None:
+            axis = self.axis or 0
 
-    def _get_label(self, label, axis=0):
         if self.ndim == 1:
             # for perf reasons we want to try _xs first
             # as its basically direct indexing
@@ -135,10 +144,14 @@ def _get_label(self, label, axis=0):
 
         return self.obj._xs(label, axis=axis)
 
-    def _get_loc(self, key, axis=0):
+    def _get_loc(self, key, axis=None):
+        if axis is None:
+            axis = self.axis
         return self.obj._ixs(key, axis=axis)
 
-    def _slice(self, obj, axis=0, kind=None):
+    def _slice(self, obj, axis=None, kind=None):
+        if axis is None:
+            axis = self.axis
         return self.obj._slice(obj, axis=axis, kind=kind)
 
     def _get_setitem_indexer(self, key):
@@ -173,7 +186,8 @@ def _get_setitem_indexer(self, key):
 
     def __setitem__(self, key, value):
         if isinstance(key, tuple):
-            key = tuple(com._apply_if_callable(x, self.obj) for x in key)
+            key = tuple(com._apply_if_callable(x, self.obj)
+                        for x in key)
         else:
             key = com._apply_if_callable(key, self.obj)
         indexer = self._get_setitem_indexer(key)
@@ -192,10 +206,12 @@ def _has_valid_tuple(self, key):
                                  "[{types}] types"
                                  .format(types=self._valid_types))
 
-    def _should_validate_iterable(self, axis=0):
+    def _should_validate_iterable(self, axis=None):
         """ return a boolean whether this axes needs validation for a passed
         iterable
         """
+        if axis is None:
+            axis = self.axis or 0
         ax = self.obj._get_axis(axis)
         if isinstance(ax, MultiIndex):
             return False
@@ -233,6 +249,8 @@ def _convert_range(self, key, is_setter=False):
 
     def _convert_scalar_indexer(self, key, axis):
         # if we are accessing via lowered dim, use the last dim
+        if axis is None:
+            axis = 0
         ax = self.obj._get_axis(min(axis, self.ndim - 1))
         # a scalar
         return ax._convert_scalar_indexer(key, kind=self.name)
@@ -895,7 +913,9 @@ def _multi_take(self, tup):
         except(KeyError, IndexingError):
             raise self._exception
 
-    def _convert_for_reindex(self, key, axis=0):
+    def _convert_for_reindex(self, key, axis=None):
+        if axis is None:
+            axis = self.axis or 0
         labels = self.obj._get_axis(axis)
 
         if is_bool_indexer(key):
@@ -925,7 +945,7 @@ def _handle_lowerdim_multi_index_axis0(self, tup):
 
         try:
             # fast path for series or for tup devoid of slices
-            return self._get_label(tup, axis=0)
+            return self._get_label(tup, axis=self.axis)
         except TypeError:
             # slices are unhashable
             pass
@@ -1015,7 +1035,7 @@ def _getitem_nested_tuple(self, tup):
 
             # this is a series with a multi-index specified a tuple of
             # selectors
-            return self._getitem_axis(tup, axis=0)
+            return self._getitem_axis(tup, axis=self.axis)
 
         # handle the multi-axis by taking sections and reducing
         # this is iterative
@@ -1049,7 +1069,10 @@ def _getitem_nested_tuple(self, tup):
 
         return obj
 
-    def _getitem_axis(self, key, axis=0):
+    def _getitem_axis(self, key, axis=None):
+
+        if axis is None:
+            axis = self.axis or 0
 
         if self._should_validate_iterable(axis):
             self._has_valid_type(key, axis)
@@ -1084,7 +1107,10 @@ def _getitem_axis(self, key, axis=0):
 
             return self._get_label(key, axis=axis)
 
-    def _getitem_iterable(self, key, axis=0):
+    def _getitem_iterable(self, key, axis=None):
+        if axis is None:
+            axis = self.axis or 0
+
         if self._should_validate_iterable(axis):
             self._has_valid_type(key, axis)
 
@@ -1138,7 +1164,7 @@ def _getitem_iterable(self, key, axis=0):
 
                 return result
 
-    def _convert_to_indexer(self, obj, axis=0, is_setter=False):
+    def _convert_to_indexer(self, obj, axis=None, is_setter=False):
         """
         Convert indexing key into something we can use to do actual fancy
         indexing on an ndarray
@@ -1153,6 +1179,9 @@ def _convert_to_indexer(self, obj, axis=0, is_setter=False):
         raise AmbiguousIndexError with integer labels?
         - No, prefer label-based indexing
         """
+        if axis is None:
+            axis = self.axis or 0
+
         labels = self.obj._get_axis(axis)
 
         if isinstance(obj, slice):
@@ -1255,9 +1284,12 @@ def _tuplify(self, loc):
         tup[0] = loc
         return tuple(tup)
 
-    def _get_slice_axis(self, slice_obj, axis=0):
+    def _get_slice_axis(self, slice_obj, axis=None):
         obj = self.obj
 
+        if axis is None:
+            axis = self.axis or 0
+
         if not need_slice(slice_obj):
             return obj.copy(deep=False)
         indexer = self._convert_slice_indexer(slice_obj, axis)
@@ -1325,7 +1357,8 @@ class _LocationIndexer(_NDFrameIndexer):
 
     def __getitem__(self, key):
         if type(key) is tuple:
-            key = tuple(com._apply_if_callable(x, self.obj) for x in key)
+            key = tuple(com._apply_if_callable(x, self.obj)
+                        for x in key)
             try:
                 if self._is_scalar_access(key):
                     return self._getitem_scalar(key)
@@ -1333,8 +1366,11 @@ def __getitem__(self, key):
                 pass
             return self._getitem_tuple(key)
         else:
-            key = com._apply_if_callable(key, self.obj)
-            return self._getitem_axis(key, axis=0)
+            # we by definition only have the 0th axis
+            axis = self.axis or 0
+
+            maybe_callable = com._apply_if_callable(key, self.obj)
+            return self._getitem_axis(maybe_callable, axis=axis)
 
     def _is_scalar_access(self, key):
         raise NotImplementedError()
@@ -1342,10 +1378,12 @@ def _is_scalar_access(self, key):
     def _getitem_scalar(self, key):
         raise NotImplementedError()
 
-    def _getitem_axis(self, key, axis=0):
+    def _getitem_axis(self, key, axis=None):
         raise NotImplementedError()
 
-    def _getbool_axis(self, key, axis=0):
+    def _getbool_axis(self, key, axis=None):
+        if axis is None:
+            axis = self.axis or 0
         labels = self.obj._get_axis(axis)
         key = check_bool_indexer(labels, key)
         inds, = key.nonzero()
@@ -1354,8 +1392,11 @@ def _getbool_axis(self, key, axis=0):
         except Exception as detail:
             raise self._exception(detail)
 
-    def _get_slice_axis(self, slice_obj, axis=0):
+    def _get_slice_axis(self, slice_obj, axis=None):
         """ this is pretty simple as we just have to deal with labels """
+        if axis is None:
+            axis = self.axis or 0
+
         obj = self.obj
         if not need_slice(slice_obj):
             return obj.copy(deep=False)
@@ -1528,7 +1569,10 @@ def _get_partial_string_timestamp_match_key(self, key, labels):
 
         return key
 
-    def _getitem_axis(self, key, axis=0):
+    def _getitem_axis(self, key, axis=None):
+        if axis is None:
+            axis = self.axis or 0
+
         labels = self.obj._get_axis(axis)
         key = self._get_partial_string_timestamp_match_key(key, labels)
 
@@ -1717,7 +1761,9 @@ def _getitem_tuple(self, tup):
 
         return retval
 
-    def _get_slice_axis(self, slice_obj, axis=0):
+    def _get_slice_axis(self, slice_obj, axis=None):
+        if axis is None:
+            axis = self.axis or 0
         obj = self.obj
 
         if not need_slice(slice_obj):
@@ -1729,7 +1775,7 @@ def _get_slice_axis(self, slice_obj, axis=0):
         else:
             return self.obj._take(slice_obj, axis=axis, convert=False)
 
-    def _get_list_axis(self, key, axis=0):
+    def _get_list_axis(self, key, axis=None):
         """
         Return Series values by list or array of integers
 
@@ -1742,13 +1788,17 @@ def _get_list_axis(self, key, axis=0):
         -------
         Series object
         """
+        if axis is None:
+            axis = self.axis or 0
         try:
             return self.obj._take(key, axis=axis, convert=False)
         except IndexError:
             # re-raise with different error message
             raise IndexError("positional indexers are out-of-bounds")
 
-    def _getitem_axis(self, key, axis=0):
+    def _getitem_axis(self, key, axis=None):
+        if axis is None:
+            axis = self.axis or 0
 
         if isinstance(key, slice):
             self._has_valid_type(key, axis)
@@ -1781,8 +1831,10 @@ def _getitem_axis(self, key, axis=0):
 
             return self._get_loc(key, axis=axis)
 
-    def _convert_to_indexer(self, obj, axis=0, is_setter=False):
+    def _convert_to_indexer(self, obj, axis=None, is_setter=False):
         """ much simpler as we only have to deal with our valid types """
+        if axis is None:
+            axis = self.axis or 0
 
         # make need to convert a float key
         if isinstance(obj, slice):
@@ -1818,7 +1870,8 @@ def __getitem__(self, key):
 
     def __setitem__(self, key, value):
         if isinstance(key, tuple):
-            key = tuple(com._apply_if_callable(x, self.obj) for x in key)
+            key = tuple(com._apply_if_callable(x, self.obj)
+                        for x in key)
         else:
             # scalar callable may return tuple
             key = com._apply_if_callable(key, self.obj)


Test patch:
diff --git a/pandas/tests/frame/test_alter_axes.py b/pandas/tests/frame/test_alter_axes.py
--- a/pandas/tests/frame/test_alter_axes.py
+++ b/pandas/tests/frame/test_alter_axes.py
@@ -143,10 +143,11 @@ def test_set_index_nonuniq(self):
     def test_set_index_bug(self):
         # GH1590
         df = DataFrame({'val': [0, 1, 2], 'key': ['a', 'b', 'c']})
-        df2 = df.select(lambda indx: indx >= 1)
-        rs = df2.set_index('key')
         xp = DataFrame({'val': [1, 2]},
                        Index(['b', 'c'], name='key'))
+
+        df2 = df.loc[df.index.map(lambda indx: indx >= 1)]
+        rs = df2.set_index('key')
         assert_frame_equal(rs, xp)
 
     def test_set_index_pass_arrays(self):
diff --git a/pandas/tests/frame/test_axis_select_reindex.py b/pandas/tests/frame/test_axis_select_reindex.py
--- a/pandas/tests/frame/test_axis_select_reindex.py
+++ b/pandas/tests/frame/test_axis_select_reindex.py
@@ -796,16 +796,38 @@ def test_filter_corner(self):
         assert_frame_equal(result, empty)
 
     def test_select(self):
+
+        # deprecated: gh-12410
         f = lambda x: x.weekday() == 2
-        result = self.tsframe.select(f, axis=0)
-        expected = self.tsframe.reindex(
-            index=self.tsframe.index[[f(x) for x in self.tsframe.index]])
-        assert_frame_equal(result, expected)
+        index = self.tsframe.index[[f(x) for x in self.tsframe.index]]
+        expected_weekdays = self.tsframe.reindex(index=index)
+
+        with tm.assert_produces_warning(FutureWarning,
+                                        check_stacklevel=False):
+            result = self.tsframe.select(f, axis=0)
+            assert_frame_equal(result, expected_weekdays)
+
+            result = self.frame.select(lambda x: x in ('B', 'D'), axis=1)
+            expected = self.frame.reindex(columns=['B', 'D'])
+            assert_frame_equal(result, expected, check_names=False)
+
+        # replacement
+        f = lambda x: x.weekday == 2
+        result = self.tsframe.loc(axis=0)[f(self.tsframe.index)]
+        assert_frame_equal(result, expected_weekdays)
 
-        result = self.frame.select(lambda x: x in ('B', 'D'), axis=1)
+        crit = lambda x: x in ['B', 'D']
+        result = self.frame.loc(axis=1)[(self.frame.columns.map(crit))]
         expected = self.frame.reindex(columns=['B', 'D'])
+        assert_frame_equal(result, expected, check_names=False)
+
+        # doc example
+        df = DataFrame({'A': [1, 2, 3]}, index=['foo', 'bar', 'baz'])
 
-        # TODO should reindex check_names?
+        crit = lambda x: x in ['bar', 'baz']
+        with tm.assert_produces_warning(FutureWarning):
+            expected = df.select(crit)
+        result = df.loc[df.index.map(crit)]
         assert_frame_equal(result, expected, check_names=False)
 
     def test_take(self):
diff --git a/pandas/tests/frame/test_mutate_columns.py b/pandas/tests/frame/test_mutate_columns.py
--- a/pandas/tests/frame/test_mutate_columns.py
+++ b/pandas/tests/frame/test_mutate_columns.py
@@ -83,6 +83,7 @@ def test_assign_order(self):
 
     def test_assign_bad(self):
         df = DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
+
         # non-keyword argument
         with pytest.raises(TypeError):
             df.assign(lambda x: x.A)
diff --git a/pandas/tests/groupby/test_groupby.py b/pandas/tests/groupby/test_groupby.py
--- a/pandas/tests/groupby/test_groupby.py
+++ b/pandas/tests/groupby/test_groupby.py
@@ -3103,7 +3103,8 @@ def agg_before(hour, func, fix=False):
             """
 
             def _func(data):
-                d = data.select(lambda x: x.hour < 11).dropna()
+                d = data.loc[data.index.map(
+                    lambda x: x.hour < 11)].dropna()
                 if fix:
                     data[data.index[0]]
                 if len(d) == 0:
diff --git a/pandas/tests/series/test_indexing.py b/pandas/tests/series/test_indexing.py
--- a/pandas/tests/series/test_indexing.py
+++ b/pandas/tests/series/test_indexing.py
@@ -2225,14 +2225,18 @@ def test_rename(self):
         assert result.name == expected.name
 
     def test_select(self):
-        n = len(self.ts)
-        result = self.ts.select(lambda x: x >= self.ts.index[n // 2])
-        expected = self.ts.reindex(self.ts.index[n // 2:])
-        assert_series_equal(result, expected)
 
-        result = self.ts.select(lambda x: x.weekday() == 2)
-        expected = self.ts[self.ts.index.weekday == 2]
-        assert_series_equal(result, expected)
+        # deprecated: gh-12410
+        with tm.assert_produces_warning(FutureWarning,
+                                        check_stacklevel=False):
+            n = len(self.ts)
+            result = self.ts.select(lambda x: x >= self.ts.index[n // 2])
+            expected = self.ts.reindex(self.ts.index[n // 2:])
+            assert_series_equal(result, expected)
+
+            result = self.ts.select(lambda x: x.weekday() == 2)
+            expected = self.ts[self.ts.index.weekday == 2]
+            assert_series_equal(result, expected)
 
     def test_cast_on_putmask(self):
 
diff --git a/pandas/tests/test_multilevel.py b/pandas/tests/test_multilevel.py
--- a/pandas/tests/test_multilevel.py
+++ b/pandas/tests/test_multilevel.py
@@ -1239,7 +1239,8 @@ def test_groupby_level_no_obs(self):
             'f2', 's1'), ('f2', 's2'), ('f3', 's1'), ('f3', 's2')])
         df = DataFrame(
             [[1, 2, 3, 4, 5, 6], [7, 8, 9, 10, 11, 12]], columns=midx)
-        df1 = df.select(lambda u: u[0] in ['f2', 'f3'], axis=1)
+        df1 = df.loc(axis=1)[df.columns.map(
+            lambda u: u[0] in ['f2', 'f3'])]
 
         grouped = df1.groupby(axis=1, level=0)
         result = grouped.sum()