Repository: pandas-dev/pandas
Pull request #61195: DOC: User Guide Page on user-defined functions
Author: login: arthurlw
Labels: name: Docs
name: Apply
description: Apply, Aggregate, Transform, Map
Created: 2025-03-28T19:15:48Z | Merged: 2025-05-18T19:31:46Z

- [x] closes #61126 
- [ ] ~[Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature~
- [ ] ~All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).~
- [ ] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [ ] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~


Linked issue #61126: DOC: Write user guide page on apply/map/transform methods
There is some information in our documentation regarding how to use user defined functions in pandas. The API pages of the used methods, and these sections:

- https://pandas.pydata.org/docs/user_guide/groupby.html#aggregation-with-user-defined-functions
- https://pandas.pydata.org/docs/user_guide/gotchas.html#gotchas-udf-mutation

My understanding is that we've been mostly discouraging the use of functions like apply, or at least the community has with many posts and comments regarding `apply` is slow, which seem fair. With the work going on supporting JIT compilers on these functions (see https://github.com/pandas-dev/pandas/pull/54666 and https://github.com/pandas-dev/pandas/pull/61032) this can hopefully change, and allow in some cases for clearer code while not compromising speed.

I think it may be difficult to communicate all the information related to udf in the existing sections on group by and FAQ pages and in the API docs. A dedicated page in the users guide that guides users on when to use udf, a general idea of the API, the differences between the different methods, the options available... seems a better idea.

Also, the APIs of the different methods are quite inconsistent, and in some cases cumbersome. I think writing this page will be a good exercise to identify cases when explaining the functionality to the users is complex and not intuitive, and see if we can address them.
Linked issue comments: user: login: arthurlw
created_at: 2025-03-23T00:18:58Z
body: I'm interested in working on this user guide. Just to clarify, this user guide should provide guidance on everything about udfs (when to use them, their differences, etc), rather than just document what they do, correct?
author_association: MEMBER
user: login: arthurlw
created_at: 2025-03-24T18:33:08Z
body: take
author_association: MEMBER
user: login: rhshadrach
created_at: 2025-03-29T13:02:25Z
body: > Also, the APIs of the different methods are quite inconsistent, and in some cases cumbersome.

Related: https://github.com/pandas-dev/pandas/issues/40112
author_association: MEMBER

Diff:
diff --git a/doc/source/user_guide/index.rst b/doc/source/user_guide/index.rst
index f0d6a76f0de5b..230b2b86b2ffd 100644
--- a/doc/source/user_guide/index.rst
+++ b/doc/source/user_guide/index.rst
@@ -78,6 +78,7 @@ Guides
     boolean
     visualization
     style
+    user_defined_functions
     groupby
     window
     timeseries
diff --git a/doc/source/user_guide/user_defined_functions.rst b/doc/source/user_guide/user_defined_functions.rst
new file mode 100644
index 0000000000000..c2472b3c229db
--- /dev/null
+++ b/doc/source/user_guide/user_defined_functions.rst
@@ -0,0 +1,305 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                                                                                                  |
++============================+========================+==========================+==============================================================================================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Apply a function to each element                                                                                                             |
++----------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                                                                                              |
++----------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                                                                                                 |
++----------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer                                                                                 |
++----------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+
+| :meth:`transform` (axis=0) | Column (Series)        | Column(Series)           | Same as :meth:`apply` with (axis=0), but it raises an exception if the function changes the shape of the data                                |
++----------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+
+| :meth:`transform` (axis=1) | Row (Series)           | Row (Series)             | Same as :meth:`apply` with (axis=1), but it raises an exception if the function changes the shape of the data                                |
++----------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+
+| :meth:`filter`             | Series or DataFrame    | Boolean                  | Only accepts UDFs in group by. Function is called for each group, and the group is removed from the result if the function returns ``False`` |
++----------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+
+| :meth:`pipe`               | Series/DataFrame       | Series/DataFrame         | Chain functions together to apply to Series or Dataframe                                                                                     |
++----------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+
+
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`apply` method allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`agg` for performing custom aggregations, where the operation returns
+a scalar value on each input.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`transform` method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    }).set_index("x")
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        x = group.index.to_frame()
+        y = group
+        model = LinearRegression()
+        model.fit(x, y)
+        pred = model.predict(x)
+        return pred
+
+    result = df.groupby('group').transform(fit_model)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'AA': [1, 2, 3],
+        'BB': [4, 5, 6],
+        'C': [7, 8, 9],
+        'D': [10, 11, 12]
+    })
+
+    # Function that filters out columns where the name is longer than 1 character
+    def is_long_name(column_name):
+        return len(column_name) > 1
+
+    df_filtered = df.filter(items=[col for col in df.columns if is_long_name(col)])
+    print(df_filtered)
+
+Since filter does not directly accept a UDF, you have to apply the UDF indirectly,
+for example, by using list comprehensions.
+
+:meth:`DataFrame.map`
+~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`map` method is used specifically to apply element-wise UDFs.
+
+When to use: Use :meth:`map` for applying element-wise UDFs to DataFrames or Series.
+
+:meth:`DataFrame.pipe`
+~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`pipe` method is useful for chaining operations together into a clean and readable pipeline.
+It is a helpful tool for organizing complex data processing workflows.
+
+When to use: Use :meth:`pipe` when you need to create a pipeline of operations and want to keep the code readable and maintainable.
+
+
+Performance
+-----------
+
+While UDFs provide flexibility, their use is generally discouraged as they can introduce
+performance issues, especially when written in pure Python. To improve efficiency,
+consider using built-in ``NumPy`` or ``pandas`` functions instead of UDFs
+for common operations.
+
+.. note::
+    If performance is critical, explore **vectorized operations** before resorting
+    to UDFs.
+
+Vectorized Operations
+~~~~~~~~~~~~~~~~~~~~~
+
+Below is a comparison of using UDFs versus using Vectorized Operations:
+
+.. code-block:: python
+
+    # User-defined function
+    def calc_ratio(row):
+        return 100 * (row["one"] / row["two"])
+
+    df["new_col"] = df.apply(calc_ratio, axis=1)
+
+    # Vectorized Operation
+    df["new_col2"] = 100 * (df["one"] / df["two"])
+
+Measuring how long each operation takes:
+
+.. code-block:: text
+
+    User-defined function:  5.6435 secs
+    Vectorized:             0.0043 secs
+
+Vectorized operations in pandas are significantly faster than using :meth:`DataFrame.apply`
+with UDFs because they leverage highly optimized C functions
+via ``NumPy`` to process entire arrays at once. This approach avoids the overhead of looping
+through rows in Python and making separate function calls for each row, which is slow and
+inefficient. Additionally, ``NumPy`` arrays benefit from memory efficiency and CPU-level
+optimizations, making vectorized operations the preferred choice whenever possible.
+
+
+Improving Performance with UDFs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In scenarios where UDFs are necessary, there are still ways to mitigate their performance drawbacks.
+One approach is to use **Numba**, a Just-In-Time (JIT) compiler that can significantly speed up numerical
+Python code by compiling Python functions to optimized machine code at runtime.
+
+By annotating your UDFs with ``@numba.jit``, you can achieve performance closer to vectorized operations,
+especially for computationally heavy tasks.
+
+.. note::
+    You may also refer to the user guide on `Enhancing performance <https://pandas.pydata.org/pandas-docs/dev/user_guide/enhancingperf.html#numba-jit-compilation>`_
+    for a more detailed guide to using **Numba**.
+
+Using :meth:`DataFrame.pipe` for Composable Logic
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Another useful pattern for improving readability and composability, especially when mixing
+vectorized logic with UDFs, is to use the :meth:`DataFrame.pipe` method.
+
+:meth:`DataFrame.pipe` doesn't improve performance directly, but it enables cleaner
+method chaining by passing the entire object into a function. This is especially helpful
+when chaining custom transformations:
+
+.. code-block:: python
+
+    def add_ratio_column(df):
+        df["ratio"] = 100 * (df["one"] / df["two"])
+        return df
+
+    df = (
+        df
+        .query("one > 0")
+        .pipe(add_ratio_column)
+        .dropna()
+    )
+
+This is functionally equivalent to calling ``add_ratio_column(df)``, but keeps your code
+clean and composable. The function you pass to :meth:`DataFrame.pipe` can use vectorized operations,
+row-wise UDFs, or any other logic; :meth:`DataFrame.pipe` is agnostic.
+
+.. note::
+    While :meth:`DataFrame.pipe` does not improve performance on its own,
+    it promotes clean, modular design and allows both vectorized and UDF-based logic
+    to be composed in method chains.


Reviews:
user: login: rhshadrach
body: Thanks for the PR! I'm not opposed to a dedicated page on UDFs, but I am opposed to duplicating documentation that exists elsewhere in the user guide, as I think much of this does. Instead of e.g. examples of `apply`, I recommend linking to the appropriate section. This page can then focus on recommendations of when to use apply vs other methods.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-29T13:02:36Z
user: login: rhshadrach
body: I think this is looking a lot better. Can we also link to https://pandas.pydata.org/pandas-docs/dev/user_guide/enhancingperf.html#numba-jit-compilation at the very bottom in a section titled something like "Improving Performance with UDFs".
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-06T13:19:26Z
user: login: rhshadrach
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-12T12:50:06Z
user: login: arthurlw
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-12T19:31:19Z
user: login: datapythonista
body: Excellent job here @arthurlw, thanks for taking care of this. I added a general comment about using examples to incrementally illustrate what it's explain here, and changing a bit the order of the sections.

Please let me know if it doesn't make sense or you have any comment. I'll review more in depth after the proposed changes are implemented or discussed. But in a first look, this is really nice.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-12T21:27:44Z
user: login: rhshadrach
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T12:29:16Z
user: login: rhshadrach
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T12:30:30Z
user: login: datapythonista
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-19T12:02:32Z
user: login: datapythonista
body: Very nice, just a couple of small comment. And we need to decide about duplication, but in general looks great. Thanks for the work here @arthurlw 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-19T12:36:33Z
user: login: rhshadrach
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-19T12:50:51Z
user: login: rhshadrach
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-19T12:57:10Z
user: login: arthurlw
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T00:38:15Z
user: login: datapythonista
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T10:16:34Z
user: login: datapythonista
body: Looks great. Some minor comments, and as you've seen, we need to make a decision on the structure to use. But I think this guide is a great addition.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T12:26:10Z
user: login: rhshadrach
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T12:53:58Z
user: login: datapythonista
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T13:22:33Z
user: login: datapythonista
body: I added few comment. Happy to get this merged once those are addressed, and any further improvement to this doc can be done as a follow up.

Thanks for all the work on this @arthurlw, really good job. 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-03T14:39:50Z
user: login: arthurlw
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-04T18:42:17Z
user: login: datapythonista
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-04T18:48:10Z
user: login: rhshadrach
body: lgtm
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-18T19:31:25Z

Review comments:
diff_hunk: @@ -0,0 +1,305 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Use User-Defined Functions?
+-------------------------------
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: I think we should lead with `Why _not_ User-Defined Functions`. While performance is called out down below, I think the poor behavior of UDFs should be mentioned as well. Namely that pandas has no information on what a UDF is doing, and so has to infer (guess) at how to handle the result.

In particular, I think it should be mentioned that none of the examples on this page should be UDFs in practice.
created_at: 2025-03-29T12:54:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,234 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: In my opinion the primary drawback is behavior and not performance, but others may disagree. I'd suggest not being opinionated here between the two, but rather saying `primarily related to performance and behavior`.

In any case, can you include that pandas must perform inference on the result, and that inference can be incorrect.
created_at: 2025-04-06T12:55:20Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,234 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
+insight into what they are computing, making it difficult to apply efficient handling or optimization
+techniques. As a result, pandas resorts to less efficient processing methods that significantly
+slow down computations. Additionally, relying on UDFs often sacrifices the benefits
+of pandas’ built-in, optimized methods, limiting compatibility and overall performance.
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: It seems to me these three sentences are all staying the same thing - and that one sentence here would do.
created_at: 2025-04-06T12:57:01Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,234 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
+insight into what they are computing, making it difficult to apply efficient handling or optimization
+techniques. As a result, pandas resorts to less efficient processing methods that significantly
+slow down computations. Additionally, relying on UDFs often sacrifices the benefits
+of pandas’ built-in, optimized methods, limiting compatibility and overall performance.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: For the last line here, would love to see a real-world example of this that couldn't be broken down into supported operations. But I'm okay with this staying regardless.
created_at: 2025-04-06T12:58:35Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,234 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
+insight into what they are computing, making it difficult to apply efficient handling or optimization
+techniques. As a result, pandas resorts to less efficient processing methods that significantly
+slow down computations. Additionally, relying on UDFs often sacrifices the benefits
+of pandas’ built-in, optimized methods, limiting compatibility and overall performance.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series,
+  DataFrames, or groups of data.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting multiple
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to groups while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters groups based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process entire DataFrames or
+  Series in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: Can you also make a mention of resample, rolling, expanding, and ewm. Perhaps link to each section in the User Guide.


created_at: 2025-04-06T13:02:47Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,234 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
+insight into what they are computing, making it difficult to apply efficient handling or optimization
+techniques. As a result, pandas resorts to less efficient processing methods that significantly
+slow down computations. Additionally, relying on UDFs often sacrifices the benefits
+of pandas’ built-in, optimized methods, limiting compatibility and overall performance.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series,
+  DataFrames, or groups of data.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting multiple
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to groups while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters groups based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process entire DataFrames or
+  Series in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Performance               | Recommended Use Case                     |
++==================+======================================+===========================+====================+===========================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Slow                      | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Fast (if using built-ins) | Custom aggregation logic                 |
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: `Fast (if using built-ins)`

We should decide if this page is about using UDFs, in which case I think e.g. `.agg("sum")` is not within the scope, or if it's about using methods that take UDFs.

I'd suggest the former, and remove any mention of not using UDFs - and with that the performance column.
created_at: 2025-04-06T13:07:56Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,234 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
+insight into what they are computing, making it difficult to apply efficient handling or optimization
+techniques. As a result, pandas resorts to less efficient processing methods that significantly
+slow down computations. Additionally, relying on UDFs often sacrifices the benefits
+of pandas’ built-in, optimized methods, limiting compatibility and overall performance.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series,
+  DataFrames, or groups of data.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting multiple
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to groups while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters groups based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process entire DataFrames or
+  Series in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Performance               | Recommended Use Case                     |
++==================+======================================+===========================+====================+===========================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Slow                      | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Fast (if using built-ins) | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Fast (if vectorized)      | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Moderate                  | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Depends on function       | Building clean pipelines                 |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Fast                      | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method is available, but consider
+optimizing performance with vectorized operations wherever possible.
+
+Examples of usage can be found :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing aggregations like sum, mean, or custom aggregation
+functions across groups.
+
+Examples of usage can be found :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It’s generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found :meth:`~DataFrame.transform`.
+
+Attempting to use common aggregation functions such as ``mean`` or ``sum`` will result in
+values being broadcasted to the original dimensions:
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'Category': ['A', 'A', 'B', 'B', 'B'],
+        'Values': [10, 20, 30, 40, 50]
+    })
+
+    # Using transform with mean
+    df['Mean_Transformed'] = df.groupby('Category')['Values'].transform('mean')
+
+    # Using transform with sum
+    df['Sum_Transformed'] = df.groupby('Category')['Values'].transform('sum')
+
+    # Result broadcasted to DataFrame
+    print(df)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'AA': [1, 2, 3],
+        'BB': [4, 5, 6],
+        'C': [7, 8, 9],
+        'D': [10, 11, 12]
+    })
+
+    # Define a function that filters out columns where the name is longer than 1 character
+    def is_long_name(column_name):
+        return len(column_name) > 1
+
+    df_filtered = df[[col for col in df.columns if is_long_name(col)]]
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: This example doesn't actually use `.filter`. Shouldn't it?
created_at: 2025-04-06T13:10:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,234 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
+insight into what they are computing, making it difficult to apply efficient handling or optimization
+techniques. As a result, pandas resorts to less efficient processing methods that significantly
+slow down computations. Additionally, relying on UDFs often sacrifices the benefits
+of pandas’ built-in, optimized methods, limiting compatibility and overall performance.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series,
+  DataFrames, or groups of data.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting multiple
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to groups while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters groups based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process entire DataFrames or
+  Series in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Performance               | Recommended Use Case                     |
++==================+======================================+===========================+====================+===========================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Slow                      | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Fast (if using built-ins) | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Fast (if vectorized)      | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Moderate                  | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Depends on function       | Building clean pipelines                 |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Fast                      | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method is available, but consider
+optimizing performance with vectorized operations wherever possible.
+
+Examples of usage can be found :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing aggregations like sum, mean, or custom aggregation
+functions across groups.
+
+Examples of usage can be found :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It’s generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found :meth:`~DataFrame.transform`.
+
+Attempting to use common aggregation functions such as ``mean`` or ``sum`` will result in
+values being broadcasted to the original dimensions:
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'Category': ['A', 'A', 'B', 'B', 'B'],
+        'Values': [10, 20, 30, 40, 50]
+    })
+
+    # Using transform with mean
+    df['Mean_Transformed'] = df.groupby('Category')['Values'].transform('mean')
+
+    # Using transform with sum
+    df['Sum_Transformed'] = df.groupby('Category')['Values'].transform('sum')
+
+    # Result broadcasted to DataFrame
+    print(df)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'AA': [1, 2, 3],
+        'BB': [4, 5, 6],
+        'C': [7, 8, 9],
+        'D': [10, 11, 12]
+    })
+
+    # Define a function that filters out columns where the name is longer than 1 character
+    def is_long_name(column_name):
+        return len(column_name) > 1
+
+    df_filtered = df[[col for col in df.columns if is_long_name(col)]]
+    print(df_filtered)
+
+Since filter does not directly accept a UDF, you have to apply the UDF indirectly,
+such as by using list comprehensions.
+
+Documentation can be found :meth:`~DataFrame.filter`.
+
+:meth:`DataFrame.map`
+~~~~~~~~~~~~~~~~~~~~~
+
+:meth:`DataFrame.map` is used specifically to apply element-wise UDFs and is better
+for this purpose compared to :meth:`DataFrame.apply` because of its better performance.
+
+When to use: Use map for applying element-wise UDFs to DataFrames or Series.
+
+Documentation can be found :meth:`~DataFrame.map`.
+
+:meth:`DataFrame.pipe`
+~~~~~~~~~~~~~~~~~~~~~~
+
+The pipe method is useful for chaining operations together into a clean and readable pipeline.
+It is a helpful tool for organizing complex data processing workflows.
+
+When to use: Use pipe when you need to create a pipeline of transformations and want to keep the code readable and maintainable.
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: Nit: We'll probably want to stay away from `transformations` here to avoid confusion with `.transform`. I'd suggest `operations`.
created_at: 2025-04-06T13:12:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,234 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
+insight into what they are computing, making it difficult to apply efficient handling or optimization
+techniques. As a result, pandas resorts to less efficient processing methods that significantly
+slow down computations. Additionally, relying on UDFs often sacrifices the benefits
+of pandas’ built-in, optimized methods, limiting compatibility and overall performance.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series,
+  DataFrames, or groups of data.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting multiple
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to groups while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters groups based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process entire DataFrames or
+  Series in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Performance               | Recommended Use Case                     |
++==================+======================================+===========================+====================+===========================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Slow                      | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Fast (if using built-ins) | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Fast (if vectorized)      | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Moderate                  | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Depends on function       | Building clean pipelines                 |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Fast                      | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method is available, but consider
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: I think we should recommend when no other UDF method is suitable as well.
created_at: 2025-04-06T13:16:11Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: nit: "change the data differently" sounds very close to mutating in a UDF, which we explicitly do not support. What do you think of "behave differently".
created_at: 2025-04-12T12:22:54Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series,
+  DataFrames, or groups of data.
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: I'm thinking we should remove `groups of data` here. `DataFrame.apply` that you're referencing doesn't operate on groups, and you mention groupby below.
created_at: 2025-04-12T12:28:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,234 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
+insight into what they are computing, making it difficult to apply efficient handling or optimization
+techniques. As a result, pandas resorts to less efficient processing methods that significantly
+slow down computations. Additionally, relying on UDFs often sacrifices the benefits
+of pandas’ built-in, optimized methods, limiting compatibility and overall performance.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series,
+  DataFrames, or groups of data.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting multiple
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to groups while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters groups based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process entire DataFrames or
+  Series in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: Can we add the other objects to this note, it seems to me they all belong together.

```suggestion
    Some of these methods are can also be applied to groupby, resample, and various window objects. See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`, and :ref:`ewm()<window>` for details.
```
created_at: 2025-04-12T12:30:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series,
+  DataFrames, or groups of data.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting multiple
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to groups while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters groups based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process entire DataFrames or
+  Series in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
+
+Additionally, operations such as :ref:`resample()<timeseries>`, :ref:`rolling()<window>`,
+:ref:`expanding()<window>`, and :ref:`ewm()<window>` also support UDFs for performing custom
+computations over temporal or statistical windows.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing aggregations like sum, mean, or custom aggregation
+functions across groups.
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: Things like `.agg(["sum", "mean"])` aren't UDFs, so I don't think they should be mentioned here, and it could be make users think these types of usages are slow (they are not).

```suggestion
When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns a scalar value on each input.
```
created_at: 2025-04-12T12:35:56Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series,
+  DataFrames, or groups of data.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting multiple
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to groups while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters groups based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process entire DataFrames or
+  Series in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
+
+Additionally, operations such as :ref:`resample()<timeseries>`, :ref:`rolling()<window>`,
+:ref:`expanding()<window>`, and :ref:`ewm()<window>` also support UDFs for performing custom
+computations over temporal or statistical windows.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing aggregations like sum, mean, or custom aggregation
+functions across groups.
+
+Documentation can be found at :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found at :meth:`~DataFrame.transform`.
+
+Attempting to use common aggregation functions such as ``mean`` or ``sum`` will result in
+values being broadcasted to the original dimensions:
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'Category': ['A', 'A', 'B', 'B', 'B'],
+        'Values': [10, 20, 30, 40, 50]
+    })
+
+    # Using transform with mean
+    df['Mean_Transformed'] = df.groupby('Category')['Values'].transform('mean')
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: This isn't an example of a UDF. I really like your example of using linear regression - can we do that here? It's a bit unfortunate that groupby.transform does not allow operating on the entire group (only works column-by-column) here.

```python
from sklearn.linear_model import LinearRegression

df = pd.DataFrame({
    'group': ['A', 'A', 'A', 'B', 'B', 'B'],
    'x': [1, 2, 3, 1, 2, 3],
    'y': [2, 4, 6, 1, 2, 1.5]
}).set_index("x")

# Function to fit a model to each group
def fit_model(group):
    x = group.index.to_frame()
    y = group
    model = LinearRegression()
    model.fit(x, y)
    pred = model.predict(x)
    return pred

result = df.groupby('group').transform(fit_model)
```
created_at: 2025-04-12T12:46:18Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
path: doc/source/user_guide/user_defined_functions.rst
user: login: arthurlw
body: “Behave differently” sounds clearer and avoids implying mutation. I'll update it!
created_at: 2025-04-12T19:31:19Z
author_association: MEMBER
diff_hunk: @@ -88,3 +88,4 @@ Guides
     sparse
     gotchas
     cookbook
+    user_defined_functions
path: doc/source/user_guide/index.rst
user: login: datapythonista
body: I would move this before the `groupby` section. It feels more natural to me to explain first `Series.apply` and later explain `groupby("col").apply`.
created_at: 2025-04-12T21:11:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: Not sure if Sphinx is more flexible now, but this had to be the same exact length as the title before.
created_at: 2025-04-12T21:14:41Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: ```suggestion
User-Defined Functions (UDFs)
```

This will be what will be shown in the index too, so better to be concise. Also, better for consistency to remove the `Introduction to`, which we could have in every other user guide too.
created_at: 2025-04-12T21:17:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: Maybe just personal opinion, but to me it makes more sense to explain what UDFs are in pandas before explaining when not to use them. This order seems reasonable assuming users already know what pandas udfs are in practice, but I'd personally prefer not to assume it in the user guide for UDFs.

In my opinion, after the previous introduction which is great, I'd show a very simple example so we make sure users reading this understand the very basics.

Something like:

```python
def add_one(x):
    return x + 1

my_series = pd.Series([1, 2, 3])

my_series.map(add_one)
```

Building on top of this, like then showing the same with a `DataFrame`, at some point showing UDFs that receive the whole column with `.apply`... should help make sure users are following and understanding all the information provided here.
created_at: 2025-04-12T21:19:17Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
+-----------------------------------------
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: Title marker needs to be at least as long as the text, but can be longer.
created_at: 2025-04-16T12:29:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: I am a bit negative here. This is duplicating a lot of other documentation that we already have. I think we should instead link to that documentation.
created_at: 2025-04-16T12:30:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: Do you mind pointing out to an specific example @rhshadrach? I found documentation for the aggregate functions, but not much for the `map`, `apply`... on `Series` and `DataFrame` other than in the API docs. I agree with not having much duplication. Personally, if there is few here and there like in the FAQs, Performance page... I'd rather have the docs related to these methods in this page, as it feels like the natural place, and link to the sections here in the FAQs, performance hints, groupby user guide... Of course there can be cases where it makes more sense the opposite, but maybe we can discuss the specific cases where there is duplication.
created_at: 2025-04-19T12:02:31Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,301 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series and
+  DataFrames.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting custom
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to Series and Dataframes while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters Series and Dataframes based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series or Dataframe, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process Series or
+  Dataframes in a clean, readable manner.
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: What do you think about having this as a table? Personally I think it should make it easier to understand the differences about the methods. As a general idea:

| method | function input | function output | description |
|-------------|----------|----------------|-------|
| map       | scalar | scalar | map each element to the element returned by the function elementwise |
| apply(axis=0 | column | column | map each column to the column returned by the function |
| apply(axis=1) | row | row |  map each row to the row returned by the function |
| pipe | series or dataframe | series or dataframe | map the series or dataframe to a new series or dataframe returned by the function |

Not sure if it makes sense to combine with the table below.
created_at: 2025-04-19T12:19:26Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,301 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series and
+  DataFrames.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting custom
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to Series and Dataframes while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters Series and Dataframes based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series or Dataframe, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process Series or
+  Dataframes in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns
+a scalar value on each input.
+
+Documentation can be found at :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found at :meth:`~DataFrame.transform`.
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    }).set_index("x")
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        x = group.index.to_frame()
+        y = group
+        model = LinearRegression()
+        model.fit(x, y)
+        pred = model.predict(x)
+        return pred
+
+    result = df.groupby('group').transform(fit_model)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: I'm unsure on having `filter` here for now. I think it's very good that you added it, as it doesn't support udfs, but it probably should. So, it opens a discussion we probably want to have about adding them. @rhshadrach thoughts?
created_at: 2025-04-19T12:23:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,301 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series and
+  DataFrames.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting custom
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to Series and Dataframes while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters Series and Dataframes based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series or Dataframe, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process Series or
+  Dataframes in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns
+a scalar value on each input.
+
+Documentation can be found at :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found at :meth:`~DataFrame.transform`.
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    }).set_index("x")
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        x = group.index.to_frame()
+        y = group
+        model = LinearRegression()
+        model.fit(x, y)
+        pred = model.predict(x)
+        return pred
+
+    result = df.groupby('group').transform(fit_model)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'AA': [1, 2, 3],
+        'BB': [4, 5, 6],
+        'C': [7, 8, 9],
+        'D': [10, 11, 12]
+    })
+
+    # Function that filters out columns where the name is longer than 1 character
+    def is_long_name(column_name):
+        return len(column_name) > 1
+
+    df_filtered = df.filter(items=[col for col in df.columns if is_long_name(col)])
+    print(df_filtered)
+
+Since filter does not directly accept a UDF, you have to apply the UDF indirectly,
+for example, by using list comprehensions.
+
+Documentation can be found at :meth:`~DataFrame.filter`.
+
+:meth:`DataFrame.map`
+~~~~~~~~~~~~~~~~~~~~~
+
+:meth:`DataFrame.map` is used specifically to apply element-wise UDFs and is better
+for this purpose compared to :meth:`DataFrame.apply` because of its better performance.
+
+When to use: Use map for applying element-wise UDFs to DataFrames or Series.
+
+Documentation can be found at :meth:`~DataFrame.map`.
+
+:meth:`DataFrame.pipe`
+~~~~~~~~~~~~~~~~~~~~~~
+
+The pipe method is useful for chaining operations together into a clean and readable pipeline.
+It is a helpful tool for organizing complex data processing workflows.
+
+When to use: Use pipe when you need to create a pipeline of operations and want to keep the code readable and maintainable.
+
+Documentation can be found at :meth:`~DataFrame.pipe`.
+
+
+Best Practices
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: Maybe just personal preference, but these last 3 sections seem to be talking about the same (performance), I'd have just a section about performance.

I'd keep it short for now, and we can iterate over it later. The reason is that each time we review this before merging it we need to re-read the whole document. So, if we can finish the main part above first, and have this as a placeholder, then in a second PR we can focus more on performance without having to keep reviewing the first part again.
created_at: 2025-04-19T12:35:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,301 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series and
+  DataFrames.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting custom
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to Series and Dataframes while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters Series and Dataframes based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series or Dataframe, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process Series or
+  Dataframes in a clean, readable manner.
+
+All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
+ways to apply UDFs across different pandas data structures.
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns
+a scalar value on each input.
+
+Documentation can be found at :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found at :meth:`~DataFrame.transform`.
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    }).set_index("x")
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        x = group.index.to_frame()
+        y = group
+        model = LinearRegression()
+        model.fit(x, y)
+        pred = model.predict(x)
+        return pred
+
+    result = df.groupby('group').transform(fit_model)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: I suspect the reason this was added is that `DataFrameGroupBy.filter` does accept UDFs. Perhaps that should be mentioned instead?

I actually think `DataFrame.filter` should accept Boolean masks, similar to PySpark and Polars. But agreed that discussion is not for here!
created_at: 2025-04-19T12:50:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: apply: https://pandas.pydata.org/docs/user_guide/basics.html#row-or-column-wise-function-application
map: https://pandas.pydata.org/docs/user_guide/basics.html#applying-elementwise-functions

> I'd rather have the docs related to these methods in this page, as it feels like the natural place

If we are going to move the docs on e.g. `DataFrame.agg` here, then this no longer is a page just about UDFs as `DataFrame.agg` does more than just use UDFs. In addition, that seems like a large reworking of the docs for little (in my opinion, actually negative) benefit.

created_at: 2025-04-19T12:57:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,301 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
+* :meth:`~DataFrame.apply` - A flexible method that allows applying a function to Series and
+  DataFrames.
+* :meth:`~DataFrame.agg` (Aggregate) - Used for summarizing data, supporting custom
+  aggregation functions.
+* :meth:`~DataFrame.transform` - Applies a function to Series and Dataframes while preserving the shape of
+  the original data.
+* :meth:`~DataFrame.filter` - Filters Series and Dataframes based on a list of Boolean conditions.
+* :meth:`~DataFrame.map` - Applies an element-wise function to a Series or Dataframe, useful for
+  transforming individual values.
+* :meth:`~DataFrame.pipe` - Allows chaining custom functions to process Series or
+  Dataframes in a clean, readable manner.
path: doc/source/user_guide/user_defined_functions.rst
user: login: arthurlw
body: Yeah I agree with you, thanks for the suggestion! I will keep the two tables separate for now
created_at: 2025-04-20T00:38:15Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: I totally missed the Essential basic functionality page, thanks for pointing that out. Fully agree with you that what I proposed here is repeating again the whole https://pandas.pydata.org/docs/user_guide/basics.html#function-application section . And I  agree that's not a good idea.

Personally, I'd rather not have that section, and have that content here. At least in my experience, map and apply are common, but not essential as other parts described in that page. And also, I think the structure of the user guide will be clearer and easier to find things with the changes.

For the `DataFrame.agg`, there is already a groupby page, and I think just having the methods in the lists of methods that support udfs would be good, and then just a mention that points out to the group by page where all the detail explanation regarding groupping is presented with examples.

There may be other structures, but what I'd like is that we can give users structure to the related methods. I think `Series` has around 200 methods and attributes. Users having to navigate that whole API to find out themselves that map, apply and pipe are kind of the same just changing the input of the udf, doesn't seem ideal. I think this page here can really help in that.

What do you think?
created_at: 2025-04-20T10:16:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,304 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Maps each element to the element returned by the function element-wise    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: I think this comes from my suggestion, but checking the descriptions now, feels like it'd be helpful to use the use the same terminology, to show how similar both functions are. Meaning that if we use `Apply a function to each column`, I think it'd be helpful to use `Apply a function to each element`. Feel free to disagree, and I see the point on using the name of the method in the description. But personally, I think it'd be good to highlight how similar these methods are, and let users understand the difference easily and quickly, and I think what I'm proposing should help with that.
created_at: 2025-04-20T10:40:29Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,304 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Maps each element to the element returned by the function element-wise    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Transform values while preserving shape                                   |
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: Transform is actually very similar to apply. The function input is a column or a row depending on axis. And the output is also a column or a row (exactly same as apply). The only difference is that `apply` allows to return a shorter or longer dataframe, while transform will raise. See this example:

```python
# apply works fine when the output still have 3 samples:
>>> pandas.DataFrame({"points": [100, 30, 50]}).apply(lambda x: pandas.Series([1, 2, 3]))
   points
0       1
1       2
2       3

# transform also works fine
>>> pandas.DataFrame({"points": [100, 30, 50]}).transform(lambda x: pandas.Series([1, 2, 3]))
   points
0       1
1       2
2       3

# apply is still happy now that we removed one of the samples:
>>> pandas.DataFrame({"points": [100, 30, 50]}).apply(lambda x: pandas.Series([1, 2]))
   points
0       1
1       2

# transform is not happy:
>>> pandas.DataFrame({"points": [100, 30, 50]}).transform(lambda x: pandas.Series([1, 2]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mgarcia/src/pandas/pandas/core/frame.py", line 10269, in transform
    result = op.transform()
             ^^^^^^^^^^^^^^
  File "/home/mgarcia/src/pandas/pandas/core/apply.py", line 356, in transform
    raise ValueError("Function did not transform")
ValueError: Function did not transform
```

This is more useful with groupby, and I'm not even sure if `DataFrame.transform` is that useful, it probably mostly exist for consistency.
created_at: 2025-04-20T10:50:39Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,304 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Maps each element to the element returned by the function element-wise    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Transform values while preserving shape                                   |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`filter`             | Series/DataFrame       | Series/DataFrame         | Filter data using a boolean array                                         |
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: For now I'd remove the function input/output fields, as there is no function.
created_at: 2025-04-20T10:52:44Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,304 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Maps each element to the element returned by the function element-wise    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Transform values while preserving shape                                   |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`filter`             | Series/DataFrame       | Series/DataFrame         | Filter data using a boolean array                                         |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`pipe`               | Series/DataFrame       | Series/DataFrame         | Chain UDFs together to apply to Series or Dataframe                       |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns
+a scalar value on each input.
+
+Documentation can be found at :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found at :meth:`~DataFrame.transform`.
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    }).set_index("x")
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        x = group.index.to_frame()
+        y = group
+        model = LinearRegression()
+        model.fit(x, y)
+        pred = model.predict(x)
+        return pred
+
+    result = df.groupby('group').transform(fit_model)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'AA': [1, 2, 3],
+        'BB': [4, 5, 6],
+        'C': [7, 8, 9],
+        'D': [10, 11, 12]
+    })
+
+    # Function that filters out columns where the name is longer than 1 character
+    def is_long_name(column_name):
+        return len(column_name) > 1
+
+    df_filtered = df.filter(items=[col for col in df.columns if is_long_name(col)])
+    print(df_filtered)
+
+Since filter does not directly accept a UDF, you have to apply the UDF indirectly,
+for example, by using list comprehensions.
+
+Documentation can be found at :meth:`~DataFrame.filter`.
+
+:meth:`DataFrame.map`
+~~~~~~~~~~~~~~~~~~~~~
+
+:meth:`DataFrame.map` is used specifically to apply element-wise UDFs and is better
+for this purpose compared to :meth:`DataFrame.apply` because of its better performance.
+
+When to use: Use map for applying element-wise UDFs to DataFrames or Series.
+
+Documentation can be found at :meth:`~DataFrame.map`.
+
+:meth:`DataFrame.pipe`
+~~~~~~~~~~~~~~~~~~~~~~
+
+The pipe method is useful for chaining operations together into a clean and readable pipeline.
+It is a helpful tool for organizing complex data processing workflows.
+
+When to use: Use pipe when you need to create a pipeline of operations and want to keep the code readable and maintainable.
+
+Documentation can be found at :meth:`~DataFrame.pipe`.
+
+
+Performance
+-----------
+
+While UDFs provide flexibility, their use is currently discouraged as they can introduce
+performance issues, especially when written in pure Python. To improve efficiency,
+consider using built-in ``NumPy`` or ``pandas`` functions instead of UDFs
+for common operations.
+
+.. note::
+    If performance is critical, explore **vectorizated operations** before resorting
+    to UDFs.
+
+Vectorized Operations
+~~~~~~~~~~~~~~~~~~~~~
+
+Below is a comparison of using UDFs versus using Vectorized Operations:
+
+.. code-block:: python
+
+    # User-defined function
+    def calc_ratio(row):
+        return 100 * (row["one"] / row["two"])
+
+    df["new_col"] = df.apply(calc_ratio, axis=1)
+
+    # Vectorized Operation
+    df["new_col2"] = 100 * (df["one"] / df["two"])
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: Maybe worth mentioning and comparing also `.pipe`, which is both vectorized and a udf?
created_at: 2025-04-20T12:25:11Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
path: doc/source/user_guide/user_defined_functions.rst
user: login: rhshadrach
body: > Personally, I'd rather not have that section, and have that content here.

If we move the main document of `apply` here, then I am quite opposed to calling this a page on UDFs as apply does  more than just take UDFs. By documenting `apply("sum")` et al here, it seems to me we make this page far less clear than leaving it as solely UDFs.

In any case, is that something you think should be tackled in this PR? This PR started as

> A dedicated page in the users guide that guides users on when to use udf, a general idea of the API, the differences between the different methods, the options available... seems a better idea.

I do not think we should morph it into moving around documentation from other places, especially when there are disagreements.

> Users having to navigate that whole API to find out themselves that map, apply and pipe are kind of the same just changing the input of the udf, doesn't seem ideal.

Which is why I think this page should be a comparison of UDF methods (as it mostly is now), while pointing to more thorough documentation elsewhere in the User Guide.

created_at: 2025-04-20T12:53:58Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,275 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+**************************************
+Introduction to User-Defined Functions
+**************************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and change the data differently, depending on the method used.
+
+Why Not To Use User-Defined Functions
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: Fair enough, I think I understand your point better now. Maybe I'd like to improve a bit the apply/maps docs in essential, but that's unrelated to this PR. And happy to move forward here focussing on the UDFs and not on the methods, as you describe.
created_at: 2025-04-20T13:22:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,336 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Apply a function to each element                                          |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Apply a function while preserving shape; raises error if shape changes    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`filter`             | -                      | -                        | Return rows that satisfy a boolean condition                              |
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: ```suggestion
| :meth:`filter`             | Series or DataFrame | Boolean         | Only accepts UDFs in group by. Function it's called for each group, and the group is removed from the result if the function returns ``False``                              |
```
created_at: 2025-05-03T13:46:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,336 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Apply a function to each element                                          |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Apply a function while preserving shape; raises error if shape changes    |
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: ```suggestion
| :meth:`transform`     (axis=0)      | Column (Series)       | Column(Series)     | Same as :meth:`apply`, but it raises an exception if the function changes the shape of the data    |
```

`apply` and `transform` are almost the same, I'd also have it twice for axis=0 and axis=1 like apply.
created_at: 2025-05-03T13:49:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,336 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Apply a function to each element                                          |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Apply a function while preserving shape; raises error if shape changes    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`filter`             | -                      | -                        | Return rows that satisfy a boolean condition                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`pipe`               | Series/DataFrame       | Series/DataFrame         | Chain functions together to apply to Series or Dataframe                  |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: Personally I'd only keep the first table. I think the description can already explain anything not clear present in the second table.

And I'd make this section `choosing the right method` part of the `Methods that support User-Defined Functions`. To me it feels like we're having two sections for mostly the same.
created_at: 2025-05-03T14:04:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,336 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Apply a function to each element                                          |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Apply a function while preserving shape; raises error if shape changes    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`filter`             | -                      | -                        | Return rows that satisfy a boolean condition                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`pipe`               | Series/DataFrame       | Series/DataFrame         | Chain functions together to apply to Series or Dataframe                  |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns
+a scalar value on each input.
+
+Documentation can be found at :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found at :meth:`~DataFrame.transform`.
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    }).set_index("x")
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        x = group.index.to_frame()
+        y = group
+        model = LinearRegression()
+        model.fit(x, y)
+        pred = model.predict(x)
+        return pred
+
+    result = df.groupby('group').transform(fit_model)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'AA': [1, 2, 3],
+        'BB': [4, 5, 6],
+        'C': [7, 8, 9],
+        'D': [10, 11, 12]
+    })
+
+    # Function that filters out columns where the name is longer than 1 character
+    def is_long_name(column_name):
+        return len(column_name) > 1
+
+    df_filtered = df.filter(items=[col for col in df.columns if is_long_name(col)])
+    print(df_filtered)
+
+Since filter does not directly accept a UDF, you have to apply the UDF indirectly,
+for example, by using list comprehensions.
+
+Documentation can be found at :meth:`~DataFrame.filter`.
+
+:meth:`DataFrame.map`
+~~~~~~~~~~~~~~~~~~~~~
+
+:meth:`DataFrame.map` is used specifically to apply element-wise UDFs and is better
+for this purpose compared to :meth:`DataFrame.apply` because of its better performance.
+
+When to use: Use map for applying element-wise UDFs to DataFrames or Series.
+
+Documentation can be found at :meth:`~DataFrame.map`.
+
+:meth:`DataFrame.pipe`
+~~~~~~~~~~~~~~~~~~~~~~
+
+The pipe method is useful for chaining operations together into a clean and readable pipeline.
+It is a helpful tool for organizing complex data processing workflows.
+
+When to use: Use pipe when you need to create a pipeline of operations and want to keep the code readable and maintainable.
+
+Documentation can be found at :meth:`~DataFrame.pipe`.
+
+
+Performance
+-----------
+
+While UDFs provide flexibility, their use is generally discouraged as they can introduce
+performance issues, especially when written in pure Python. To improve efficiency,
+consider using built-in ``NumPy`` or ``pandas`` functions instead of UDFs
+for common operations.
+
+.. note::
+    If performance is critical, explore **vectorizated operations** before resorting
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: vectorized?
created_at: 2025-05-03T14:32:54Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,336 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Apply a function to each element                                          |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Apply a function while preserving shape; raises error if shape changes    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`filter`             | -                      | -                        | Return rows that satisfy a boolean condition                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`pipe`               | Series/DataFrame       | Series/DataFrame         | Chain functions together to apply to Series or Dataframe                  |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: I'd remove this for all methods, as the title section and every mention of `apply` will already be a link.
created_at: 2025-05-03T14:35:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,336 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Apply a function to each element                                          |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Apply a function while preserving shape; raises error if shape changes    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`filter`             | -                      | -                        | Return rows that satisfy a boolean condition                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`pipe`               | Series/DataFrame       | Series/DataFrame         | Chain functions together to apply to Series or Dataframe                  |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns
+a scalar value on each input.
+
+Documentation can be found at :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found at :meth:`~DataFrame.transform`.
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    }).set_index("x")
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        x = group.index.to_frame()
+        y = group
+        model = LinearRegression()
+        model.fit(x, y)
+        pred = model.predict(x)
+        return pred
+
+    result = df.groupby('group').transform(fit_model)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'AA': [1, 2, 3],
+        'BB': [4, 5, 6],
+        'C': [7, 8, 9],
+        'D': [10, 11, 12]
+    })
+
+    # Function that filters out columns where the name is longer than 1 character
+    def is_long_name(column_name):
+        return len(column_name) > 1
+
+    df_filtered = df.filter(items=[col for col in df.columns if is_long_name(col)])
+    print(df_filtered)
+
+Since filter does not directly accept a UDF, you have to apply the UDF indirectly,
+for example, by using list comprehensions.
+
+Documentation can be found at :meth:`~DataFrame.filter`.
+
+:meth:`DataFrame.map`
+~~~~~~~~~~~~~~~~~~~~~
+
+:meth:`DataFrame.map` is used specifically to apply element-wise UDFs and is better
+for this purpose compared to :meth:`DataFrame.apply` because of its better performance.
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: I don't think map is more performant than apply, in general apply will be more performant. I'd remove this as a reason, as it depends a lot on the use case, and I don't think this is a general rule.
created_at: 2025-05-03T14:38:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,336 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Apply a function to each element                                          |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Apply a function while preserving shape; raises error if shape changes    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`filter`             | -                      | -                        | Return rows that satisfy a boolean condition                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`pipe`               | Series/DataFrame       | Series/DataFrame         | Chain functions together to apply to Series or Dataframe                  |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns
+a scalar value on each input.
+
+Documentation can be found at :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found at :meth:`~DataFrame.transform`.
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    }).set_index("x")
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        x = group.index.to_frame()
+        y = group
+        model = LinearRegression()
+        model.fit(x, y)
+        pred = model.predict(x)
+        return pred
+
+    result = df.groupby('group').transform(fit_model)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'AA': [1, 2, 3],
+        'BB': [4, 5, 6],
+        'C': [7, 8, 9],
+        'D': [10, 11, 12]
+    })
+
+    # Function that filters out columns where the name is longer than 1 character
+    def is_long_name(column_name):
+        return len(column_name) > 1
+
+    df_filtered = df.filter(items=[col for col in df.columns if is_long_name(col)])
+    print(df_filtered)
+
+Since filter does not directly accept a UDF, you have to apply the UDF indirectly,
+for example, by using list comprehensions.
+
+Documentation can be found at :meth:`~DataFrame.filter`.
+
+:meth:`DataFrame.map`
+~~~~~~~~~~~~~~~~~~~~~
+
+:meth:`DataFrame.map` is used specifically to apply element-wise UDFs and is better
+for this purpose compared to :meth:`DataFrame.apply` because of its better performance.
path: doc/source/user_guide/user_defined_functions.rst
user: login: arthurlw
body: Got it. Just curious: Why is applying generally more performant than mapping in practice? I thought map was faster for simple element-wise UDFs.
created_at: 2025-05-04T18:42:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,336 @@
+.. _user_defined_functions:
+
+{{ header }}
+
+*****************************
+User-Defined Functions (UDFs)
+*****************************
+
+In pandas, User-Defined Functions (UDFs) provide a way to extend the library’s
+functionality by allowing users to apply custom computations to their data. While
+pandas comes with a set of built-in functions for data manipulation, UDFs offer
+flexibility when built-in methods are not sufficient. These functions can be
+applied at different levels: element-wise, row-wise, column-wise, or group-wise,
+and behave differently, depending on the method used.
+
+Here’s a simple example to illustrate a UDF applied to a Series:
+
+.. ipython:: python
+
+    s = pd.Series([1, 2, 3])
+
+    # Simple UDF that adds 1 to a value
+    def add_one(x):
+        return x + 1
+
+    # Apply the function element-wise using .map
+    s.map(add_one)
+
+You can also apply UDFs to an entire DataFrame. For example:
+
+.. ipython:: python
+
+    df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]})
+
+    # UDF that takes a row and returns the sum of columns A and B
+    def sum_row(row):
+        return row["A"] + row["B"]
+
+    # Apply the function row-wise (axis=1 means apply across columns per row)
+    df.apply(sum_row, axis=1)
+
+
+Why Not To Use User-Defined Functions
+-------------------------------------
+
+While UDFs provide flexibility, they come with significant drawbacks, primarily
+related to performance and behavior. When using UDFs, pandas must perform inference
+on the result, and that inference could be incorrect. Furthermore, unlike vectorized operations,
+UDFs are slower because pandas can't optimize their computations, leading to
+inefficient processing.
+
+.. note::
+    In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
+
+Despite their drawbacks, UDFs can be helpful when:
+
+* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
+  built-in methods cannot handle.
+* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
+* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
+
+For example:
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    # Sample data
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    })
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        model = LinearRegression()
+        model.fit(group[['x']], group['y'])
+        group['y_pred'] = model.predict(group[['x']])
+        return group
+
+    result = df.groupby('group').apply(fit_model)
+
+
+Methods that support User-Defined Functions
+-------------------------------------------
+
+User-Defined Functions can be applied across various pandas methods:
+
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| Method                     | Function Input         | Function Output          | Description                                                               |
++============================+========================+==========================+===========================================================================+
+| :meth:`map`                | Scalar                 | Scalar                   | Apply a function to each element                                          |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=0)     | Column (Series)        | Column (Series)          | Apply a function to each column                                           |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`apply` (axis=1)     | Row (Series)           | Row (Series)             | Apply a function to each row                                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`agg`                | Series/DataFrame       | Scalar or Series         | Aggregate and summarizes values, e.g., sum or custom reducer              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`transform`          | Series/DataFrame       | Same shape as input      | Apply a function while preserving shape; raises error if shape changes    |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`filter`             | -                      | -                        | Return rows that satisfy a boolean condition                              |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+| :meth:`pipe`               | Series/DataFrame       | Series/DataFrame         | Chain functions together to apply to Series or Dataframe                  |
++----------------------------+------------------------+--------------------------+---------------------------------------------------------------------------+
+
+.. note::
+    Some of these methods are can also be applied to groupby, resample, and various window objects.
+    See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`,
+    and :ref:`ewm()<window>` for details.
+
+
+Choosing the Right Method
+-------------------------
+When applying UDFs in pandas, it is essential to select the appropriate method based
+on your specific task. Each method has its strengths and is designed for different use
+cases. Understanding the purpose and behavior of each method will help you make informed
+decisions, ensuring more efficient and maintainable code.
+
+Below is a table overview of all methods that accept UDFs:
+
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| Method           | Purpose                              | Supports UDFs             | Keeps Shape        | Recommended Use Case                     |
++==================+======================================+===========================+====================+==========================================+
+| :meth:`apply`    | General-purpose function             | Yes                       | Yes (when axis=1)  | Custom row-wise or column-wise operations|
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`agg`      | Aggregation                          | Yes                       | No                 | Custom aggregation logic                 |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`transform`| Transform without reducing dimensions| Yes                       | Yes                | Broadcast element-wise transformations   |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`map`      | Element-wise mapping                 | Yes                       | Yes                | Simple element-wise transformations      |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`pipe`     | Functional chaining                  | Yes                       | Yes                | Building clean operation pipelines       |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+| :meth:`filter`   | Row/Column selection                 | Not directly              | Yes                | Subsetting based on conditions           |
++------------------+--------------------------------------+---------------------------+--------------------+------------------------------------------+
+
+:meth:`DataFrame.apply`
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
+it is slower than vectorized operations and should be used only when you need operations
+that cannot be achieved with built-in pandas functions.
+
+When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method or UDF method is available,
+but consider optimizing performance with vectorized operations wherever possible.
+
+Documentation can be found at :meth:`~DataFrame.apply`.
+
+:meth:`DataFrame.agg`
+~~~~~~~~~~~~~~~~~~~~~
+
+If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
+specifically designed for aggregation operations.
+
+When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns
+a scalar value on each input.
+
+Documentation can be found at :meth:`~DataFrame.agg`.
+
+:meth:`DataFrame.transform`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
+It is generally faster than apply because it can take advantage of pandas' internal optimizations.
+
+When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
+
+Documentation can be found at :meth:`~DataFrame.transform`.
+
+.. code-block:: python
+
+    from sklearn.linear_model import LinearRegression
+
+    df = pd.DataFrame({
+        'group': ['A', 'A', 'A', 'B', 'B', 'B'],
+        'x': [1, 2, 3, 1, 2, 3],
+        'y': [2, 4, 6, 1, 2, 1.5]
+    }).set_index("x")
+
+    # Function to fit a model to each group
+    def fit_model(group):
+        x = group.index.to_frame()
+        y = group
+        model = LinearRegression()
+        model.fit(x, y)
+        pred = model.predict(x)
+        return pred
+
+    result = df.groupby('group').transform(fit_model)
+
+:meth:`DataFrame.filter`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
+columns or row. It is useful when you want to extract specific columns or rows that
+match particular conditions.
+
+When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
+
+.. note::
+    :meth:`DataFrame.filter` does not accept UDFs, but can accept
+    list comprehensions that have UDFs applied to them.
+
+.. ipython:: python
+
+    # Sample DataFrame
+    df = pd.DataFrame({
+        'AA': [1, 2, 3],
+        'BB': [4, 5, 6],
+        'C': [7, 8, 9],
+        'D': [10, 11, 12]
+    })
+
+    # Function that filters out columns where the name is longer than 1 character
+    def is_long_name(column_name):
+        return len(column_name) > 1
+
+    df_filtered = df.filter(items=[col for col in df.columns if is_long_name(col)])
+    print(df_filtered)
+
+Since filter does not directly accept a UDF, you have to apply the UDF indirectly,
+for example, by using list comprehensions.
+
+Documentation can be found at :meth:`~DataFrame.filter`.
+
+:meth:`DataFrame.map`
+~~~~~~~~~~~~~~~~~~~~~
+
+:meth:`DataFrame.map` is used specifically to apply element-wise UDFs and is better
+for this purpose compared to :meth:`DataFrame.apply` because of its better performance.
path: doc/source/user_guide/user_defined_functions.rst
user: login: datapythonista
body: In map, without anything special, pandas should necessarily loop in Python over the elements, casting each to a PyObject. In apply the same can be true, but since a whole column is given to the user, the user could also use vectorization. Like, a simple function doing a `x + 1` would be fast when using apply and slow when using map. That's why I wouldn't say map is faster.
created_at: 2025-05-04T18:48:10Z
author_association: MEMBER

PR comments:
user: login: arthurlw
created_at: 2025-03-28T19:34:49Z
body: Currently writing this, so I would appreciate any feedback on it!
author_association: MEMBER
user: login: arthurlw
created_at: 2025-03-29T17:10:53Z
body: Hi @rhshadrach thanks for the feedback! I agree with you and will push updates soon
author_association: MEMBER
user: login: datapythonista
created_at: 2025-05-04T21:13:57Z
body: /preview
author_association: MEMBER
user: login: github-actions[bot]
created_at: 2025-05-04T21:14:37Z
body: Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61195/
author_association: CONTRIBUTOR
user: login: datapythonista
created_at: 2025-05-04T21:17:04Z
body: @arthurlw I generated a preview of the rendered docs in this PR. If you want to have a look to see if everything looks as expected: https://pandas.pydata.org/preview/pandas-dev/pandas/61195/docs/user_guide/user_defined_functions.html

@rhshadrach can you have a look and see if this can be merged? Even if we want to make some improvements and extend this in the future, I think this PR is already a great first version. So, whenever there is no blocker, probably easier to get this merged and iterate in follow up PRs if needed.
author_association: MEMBER
user: login: arthurlw
created_at: 2025-05-05T01:21:00Z
body: Thanks for the preview! I think adding an example that combines groupby and filter (which takes a UDF) could be beneficial. That said, I do think it might duplicate some existing docs, so not sure if it's worth including here.
author_association: MEMBER
user: login: rhshadrach
created_at: 2025-05-08T15:54:56Z
body: Just noting this is still on my radar, should be able to get to it in the next 3 days.
author_association: MEMBER
user: login: rhshadrach
created_at: 2025-05-18T19:31:54Z
body: Thanks @arthurlw for all the work here!
author_association: MEMBER
user: login: arthurlw
created_at: 2025-05-19T08:03:38Z
body: Thanks for the guidance on this UDF User Guide @datapythonista @rhshadrach!

I'm interested in diving deeper into this area of pandas. Are there any related issues, features, or improvements you think could be tackled next?

Thanks in advance!
author_association: MEMBER
user: login: datapythonista
created_at: 2025-05-19T13:11:31Z
body: @arthurlw I created #61458 and assigned you to it. I think it's a good one to work in pandas udf. Good to learn more about the status quo, with reasonable complexity, and very useful to the project since it will make the code much clearer for future changes. Also, if you work on this issue, I think you'll realize of inconsistencies, missing documentation, and other related tasks that may be good to work on too. For example, it could be useful to document the executor interface in the documentation about extending pandas. So, third-party library authors can easily learn how to create an execution engine for pandas map/apply.

Please let me know if this is not what you're looking for, I can try to think of something else. Or if you have any question (better to ask them in the other issue).
author_association: MEMBER