Repository: Eventual-Inc/Daft
Pull request #4346: feat: Add `Expr.skew`
Author: login: srilman
Labels: name: feat
Created: 2025-05-14T01:36:37Z | Merged: 2025-05-14T20:30:51Z

## Changes Made

Adds `Expr.skew`. Implements it like Polars and Spark (Pandas and DuckDB implement a slightly different algorithm with normalization, Polars has a flag to choose between the options). `Series.skew` implements it normally and `populate_aggregation_stages` has a specialized version.

## Related Issues

Closes #4332.

## Checklist

- [ ] Documented in API Docs (if applicable)
- [ ] Documented in User Guide (if applicable)
- [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation
- [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)


Linked issue #4332: `Expr.skew`
### Is your feature request related to a problem?

Daft support in Narwhals

### Describe the solution you'd like

`daft.col('a').skew`

### Describe alternatives you've considered

_No response_

### Additional Context

It's a fairly common function, e.g. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.skewness.html / Polars has it

### Would you like to implement a fix?

No
Linked issue comments: user: login: srilman
created_at: 2025-05-15T18:56:43Z
body: @MarcoGorelli Should be available in 0.4.15!
author_association: CONTRIBUTOR

Diff:
diff --git a/daft/daft/__init__.pyi b/daft/daft/__init__.pyi
index b75ff19a36..33bb155238 100644
--- a/daft/daft/__init__.pyi
+++ b/daft/daft/__init__.pyi
@@ -1064,6 +1064,7 @@ class PyExpr:
     def bool_and(self) -> PyExpr: ...
     def bool_or(self) -> PyExpr: ...
     def any_value(self, ignore_nulls: bool) -> PyExpr: ...
+    def skew(self) -> PyExpr: ...
     def agg_list(self) -> PyExpr: ...
     def agg_set(self) -> PyExpr: ...
     def agg_concat(self) -> PyExpr: ...
diff --git a/daft/dataframe/dataframe.py b/daft/dataframe/dataframe.py
index 6157cac695..4592393eaf 100644
--- a/daft/dataframe/dataframe.py
+++ b/daft/dataframe/dataframe.py
@@ -2487,6 +2487,8 @@ def _map_agg_string_to_expr(self, expr: Expression, op: str) -> Expression:
             return expr.agg_set()
         elif op == "concat":
             return expr.agg_concat()
+        elif op == "skew":
+            return expr.skew()
 
         raise NotImplementedError(f"Aggregation {op} is not implemented.")
 
@@ -3686,6 +3688,14 @@ def count(self, *cols: ColumnInputType) -> "DataFrame":
         """
         return self.df._apply_agg_fn(Expression.count, cols, self.group_by)
 
+    def skew(self, *cols: ColumnInputType) -> "DataFrame":
+        """Performs grouped skew on this GroupedDataFrame.
+
+        Returns:
+            DataFrame: DataFrame with the grouped skew per column.
+        """
+        return self.df._apply_agg_fn(Expression.skew, cols, self.group_by)
+
     def agg_list(self, *cols: ColumnInputType) -> "DataFrame":
         """Performs grouped list on this GroupedDataFrame.
 
diff --git a/daft/expressions/expressions.py b/daft/expressions/expressions.py
index 42814d2bc0..06b631fb69 100644
--- a/daft/expressions/expressions.py
+++ b/daft/expressions/expressions.py
@@ -1190,6 +1190,11 @@ def any_value(self, ignore_nulls=False) -> Expression:
         expr = self._expr.any_value(ignore_nulls)
         return Expression._from_pyexpr(expr)
 
+    def skew(self) -> Expression:
+        """Calculates the skewness of the values from the expression."""
+        expr = self._expr.skew()
+        return Expression._from_pyexpr(expr)
+
     def agg_list(self) -> Expression:
         """Aggregates the values in the expression into a list."""
         expr = self._expr.agg_list()
diff --git a/src/daft-core/src/array/ops/mod.rs b/src/daft-core/src/array/ops/mod.rs
index 3214301451..b7a47b33b8 100644
--- a/src/daft-core/src/array/ops/mod.rs
+++ b/src/daft-core/src/array/ops/mod.rs
@@ -49,6 +49,7 @@ mod search_sorted;
 mod shift;
 mod sign;
 mod sketch_percentile;
+mod skew;
 mod sort;
 pub(crate) mod sparse_tensor;
 mod sqrt;
@@ -255,3 +256,9 @@ pub trait DaftBoolAggable {
     fn grouped_bool_and(&self, groups: &GroupIndices) -> Self::Output;
     fn grouped_bool_or(&self, groups: &GroupIndices) -> Self::Output;
 }
+
+pub trait DaftSkewAggable {
+    type Output;
+    fn skew(&self) -> Self::Output;
+    fn grouped_skew(&self, groups: &GroupIndices) -> Self::Output;
+}
diff --git a/src/daft-core/src/array/ops/skew.rs b/src/daft-core/src/array/ops/skew.rs
new file mode 100644
index 0000000000..5921c59c5a
--- /dev/null
+++ b/src/daft-core/src/array/ops/skew.rs
@@ -0,0 +1,36 @@
+use arrow2::array::PrimitiveArray;
+use common_error::DaftResult;
+
+use crate::{
+    array::{
+        ops::{DaftSkewAggable, GroupIndices},
+        DataArray,
+    },
+    datatypes::Float64Type,
+    utils::stats,
+};
+
+impl DaftSkewAggable for DataArray<Float64Type> {
+    type Output = DaftResult<Self>;
+
+    fn skew(&self) -> Self::Output {
+        let stats = stats::calculate_stats(self)?;
+        let values = self.into_iter().flatten().copied();
+        let skew = stats::calculate_skew(stats, values);
+        let field = self.field.clone();
+        let data = PrimitiveArray::<f64>::from([skew]).boxed();
+        Self::new(field, data)
+    }
+
+    fn grouped_skew(&self, groups: &GroupIndices) -> Self::Output {
+        let grouped_skew_iter = stats::grouped_stats(self, groups)?.map(|(stats, group)| {
+            let values = group.iter().filter_map(|&index| self.get(index as _));
+
+            stats::calculate_skew(stats, values)
+        });
+
+        let field = self.field.clone();
+        let data = PrimitiveArray::<f64>::from_iter(grouped_skew_iter).boxed();
+        Self::new(field, data)
+    }
+}
diff --git a/src/daft-core/src/datatypes/agg_ops.rs b/src/daft-core/src/datatypes/agg_ops.rs
index 78243414a2..27ad87a870 100644
--- a/src/daft-core/src/datatypes/agg_ops.rs
+++ b/src/daft-core/src/datatypes/agg_ops.rs
@@ -66,3 +66,15 @@ pub fn try_stddev_aggregation_supertype(dtype: &DataType) -> DaftResult<DataType
         ))),
     }
 }
+
+/// Get the data type that the skew of a column of the given data type should be casted to.
+pub fn try_skew_aggregation_supertype(dtype: &DataType) -> DaftResult<DataType> {
+    match dtype {
+        d if d.is_numeric() => Ok(DataType::Float64),
+        DataType::Decimal128(..) => Ok(DataType::Float64),
+        _ => Err(DaftError::TypeError(format!(
+            "Skew is not supported for: {}",
+            dtype
+        ))),
+    }
+}
diff --git a/src/daft-core/src/datatypes/mod.rs b/src/daft-core/src/datatypes/mod.rs
index 2e0036970a..9c6cfd7d1f 100644
--- a/src/daft-core/src/datatypes/mod.rs
+++ b/src/daft-core/src/datatypes/mod.rs
@@ -7,7 +7,8 @@ pub mod prelude;
 use std::ops::{Add, Div, Mul, Rem, Sub};
 
 pub use agg_ops::{
-    try_mean_aggregation_supertype, try_stddev_aggregation_supertype, try_sum_supertype,
+    try_mean_aggregation_supertype, try_skew_aggregation_supertype,
+    try_stddev_aggregation_supertype, try_sum_supertype,
 };
 use arrow2::{
     compute::comparison::Simd8,
diff --git a/src/daft-core/src/series/ops/agg.rs b/src/daft-core/src/series/ops/agg.rs
index 750b5966bb..b348fa0287 100644
--- a/src/daft-core/src/series/ops/agg.rs
+++ b/src/daft-core/src/series/ops/agg.rs
@@ -6,7 +6,7 @@ use crate::{
         growable::make_growable,
         ops::{
             DaftApproxSketchAggable, DaftCountAggable, DaftHllMergeAggable, DaftMeanAggable,
-            DaftSetAggable, DaftStddevAggable, DaftSumAggable, GroupIndices,
+            DaftSetAggable, DaftSkewAggable as _, DaftStddevAggable, DaftSumAggable, GroupIndices,
         },
         ListArray,
     },
@@ -338,6 +338,22 @@ impl Series {
             ))),
         }
     }
+
+    pub fn skew(&self, groups: Option<&GroupIndices>) -> DaftResult<Self> {
+        let target_type = try_skew_aggregation_supertype(self.data_type())?;
+        if !matches!(target_type, DataType::Float64) {
+            return Err(DaftError::not_implemented(format!(
+                "Skew is not implement for {target_type}, source type: {}",
+                self.data_type()
+            )));
+        }
+
+        let casted = self.cast(&DataType::Float64)?;
+        let casted = casted.f64()?;
+        Ok(groups
+            .map_or_else(|| casted.skew(), |groups| casted.grouped_skew(groups))?
+            .into_series())
+    }
 }
 
 impl DaftSetAggable for Series {
diff --git a/src/daft-core/src/utils/stats.rs b/src/daft-core/src/utils/stats.rs
index de43b186ea..9e7e046c2f 100644
--- a/src/daft-core/src/utils/stats.rs
+++ b/src/daft-core/src/utils/stats.rs
@@ -80,3 +80,18 @@ pub fn calculate_stddev(stats: Stats, values: impl Iterator<Item = f64>) -> Opti
         (sum_of_squares / stats.count).sqrt()
     })
 }
+
+pub fn calculate_skew(stats: Stats, values: impl Iterator<Item = f64>) -> Option<f64> {
+    let count = stats.count;
+    stats.mean.map(|mean| {
+        // In order to use the same iterator for 2 different calculations
+        let (m3, m2) = values.fold((0., 0.), |(m3_acc, m2_acc), v| {
+            (
+                m3_acc + (v - mean).powi(3),
+                (v - mean).mul_add(v - mean, m2_acc),
+            )
+        });
+
+        (m3 / count) / (m2 / count).powi(3).sqrt()
+    })
+}
diff --git a/src/daft-dsl/src/expr/mod.rs b/src/daft-dsl/src/expr/mod.rs
index 8f41df8831..c88261df6d 100644
--- a/src/daft-dsl/src/expr/mod.rs
+++ b/src/daft-dsl/src/expr/mod.rs
@@ -20,8 +20,8 @@ use common_hashable_float_wrapper::FloatWrapper;
 use common_treenode::{Transformed, TreeNode};
 use daft_core::{
     datatypes::{
-        try_mean_aggregation_supertype, try_stddev_aggregation_supertype, try_sum_supertype,
-        InferDataType,
+        try_mean_aggregation_supertype, try_skew_aggregation_supertype,
+        try_stddev_aggregation_supertype, try_sum_supertype, InferDataType,
     },
     join::JoinSide,
     prelude::*,
@@ -351,6 +351,9 @@ pub enum AggExpr {
     #[display("list({_0})")]
     Concat(ExprRef),
 
+    #[display("skew({_0}")]
+    Skew(ExprRef),
+
     #[display("{}", function_display_without_formatter(func, inputs)?)]
     MapGroups {
         func: FunctionExpr,
@@ -442,7 +445,8 @@ impl AggExpr {
             | Self::AnyValue(expr, _)
             | Self::List(expr)
             | Self::Set(expr)
-            | Self::Concat(expr) => expr.name(),
+            | Self::Concat(expr)
+            | Self::Skew(expr) => expr.name(),
             Self::MapGroups { func: _, inputs } => inputs.first().unwrap().name(),
         }
     }
@@ -530,6 +534,10 @@ impl AggExpr {
                 let child_id = expr.semantic_id(schema);
                 FieldID::new(format!("{child_id}.local_concat()"))
             }
+            Self::Skew(expr) => {
+                let child_id = expr.semantic_id(schema);
+                FieldID::new(format!("{child_id}.local_skew()"))
+            }
             Self::MapGroups { func, inputs } => function_semantic_id(func, inputs, schema),
         }
     }
@@ -552,7 +560,8 @@ impl AggExpr {
             | Self::AnyValue(expr, _)
             | Self::List(expr)
             | Self::Set(expr)
-            | Self::Concat(expr) => vec![expr.clone()],
+            | Self::Concat(expr)
+            | Self::Skew(expr) => vec![expr.clone()],
             Self::MapGroups { func: _, inputs } => inputs.clone(),
         }
     }
@@ -578,6 +587,7 @@ impl AggExpr {
             Self::List(_) => Self::List(first_child()),
             Self::Set(_expr) => Self::Set(first_child()),
             Self::Concat(_) => Self::Concat(first_child()),
+            Self::Skew(_) => Self::Skew(first_child()),
             Self::MapGroups { func, inputs: _ } => Self::MapGroups {
                 func: func.clone(),
                 inputs: children,
@@ -711,6 +721,15 @@ impl AggExpr {
                     ))),
                 }
             }
+
+            Self::Skew(expr) => {
+                let field = expr.to_field(schema)?;
+                Ok(Field::new(
+                    field.name.as_str(),
+                    try_skew_aggregation_supertype(&field.dtype)?,
+                ))
+            }
+
             Self::MapGroups { func, inputs } => func.to_field(inputs.as_slice(), schema, func),
         }
     }
@@ -977,6 +996,10 @@ impl Expr {
         Self::Agg(AggExpr::AnyValue(self, ignore_nulls)).into()
     }
 
+    pub fn skew(self: ExprRef) -> ExprRef {
+        Self::Agg(AggExpr::Skew(self)).into()
+    }
+
     pub fn agg_list(self: ExprRef) -> ExprRef {
         Self::Agg(AggExpr::List(self)).into()
     }
diff --git a/src/daft-dsl/src/python.rs b/src/daft-dsl/src/python.rs
index a54839796f..8cc938f5e6 100644
--- a/src/daft-dsl/src/python.rs
+++ b/src/daft-dsl/src/python.rs
@@ -394,6 +394,10 @@ impl PyExpr {
         Ok(self.expr.clone().any_value(ignore_nulls).into())
     }
 
+    pub fn skew(&self) -> PyResult<Self> {
+        Ok(self.expr.clone().skew().into())
+    }
+
     pub fn agg_list(&self) -> PyResult<Self> {
         Ok(self.expr.clone().agg_list().into())
     }
diff --git a/src/daft-logical-plan/src/ops/project.rs b/src/daft-logical-plan/src/ops/project.rs
index af34359cc3..39427629dd 100644
--- a/src/daft-logical-plan/src/ops/project.rs
+++ b/src/daft-logical-plan/src/ops/project.rs
@@ -567,6 +567,10 @@ fn replace_column_with_semantic_id_aggexpr(
             replace_column_with_semantic_id(child.clone(), subexprs_to_replace, schema)
                 .map_yes_no(AggExpr::Concat, |_| e)
         }
+        AggExpr::Skew(ref child) => {
+            replace_column_with_semantic_id(child.clone(), subexprs_to_replace, schema)
+                .map_yes_no(AggExpr::Skew, |_| e)
+        }
         AggExpr::MapGroups { func, inputs } => {
             let transforms = inputs
                 .iter()
diff --git a/src/daft-physical-plan/src/physical_planner/translate.rs b/src/daft-physical-plan/src/physical_planner/translate.rs
index 0f5dfc4986..9485bb9937 100644
--- a/src/daft-physical-plan/src/physical_planner/translate.rs
+++ b/src/daft-physical-plan/src/physical_planner/translate.rs
@@ -11,7 +11,7 @@ use common_scan_info::{PhysicalScanInfo, ScanState, SPLIT_AND_MERGE_PASS};
 use daft_core::{join::JoinSide, prelude::*};
 use daft_dsl::{
     estimated_selectivity, functions::agg::merge_mean, is_partition_compatible,
-    join::normalize_join_keys, resolved_col, AggExpr, ApproxPercentileParams, Expr, ExprRef,
+    join::normalize_join_keys, lit, resolved_col, AggExpr, ApproxPercentileParams, Expr, ExprRef,
     SketchType,
 };
 use daft_functions::{
@@ -607,6 +607,7 @@ pub fn extract_agg_expr(expr: &ExprRef) -> DaftResult<AggExpr> {
                 AggExpr::List(e) => AggExpr::List(Expr::Alias(e, name.clone()).into()),
                 AggExpr::Set(e) => AggExpr::Set(Expr::Alias(e, name.clone()).into()),
                 AggExpr::Concat(e) => AggExpr::Concat(Expr::Alias(e, name.clone()).into()),
+                AggExpr::Skew(e) => AggExpr::Skew(Expr::Alias(e, name.clone()).into()),
                 AggExpr::MapGroups { func, inputs } => AggExpr::MapGroups {
                     func,
                     inputs: inputs
@@ -938,6 +939,92 @@ pub fn populate_aggregation_stages(
                     ));
                 final_exprs.push(resolved_col(concat_of_concat_id.clone()).alias(output_name));
             }
+            AggExpr::Skew(se) => {
+                // See https://github.com/duckdb/duckdb/blob/93fda3591f4298414fa362c59219c09e03f718ab/extension/core_functions/aggregate/distributive/skew.cpp#L16
+                // Not exactly the same since they normalize by N - 1
+                let se = se.clone().cast(&DataType::Float64);
+
+                // Global count, sum, squared_sum, and cubed_sum are required for final expr
+
+                // First stage: count, sum, squared_sum, cubed_sum
+                let count_id = add_to_stage(
+                    |se| AggExpr::Count(se, CountMode::Valid),
+                    se.clone(),
+                    schema,
+                    &mut first_stage_aggs,
+                );
+                let sum_id = add_to_stage(AggExpr::Sum, se.clone(), schema, &mut first_stage_aggs);
+                let squared_sum_id = add_to_stage(
+                    |se| AggExpr::Sum(se.clone().mul(se)),
+                    se.clone(),
+                    schema,
+                    &mut first_stage_aggs,
+                );
+                let cubed_sum_id = add_to_stage(
+                    |se| AggExpr::Sum(se.clone().mul(se.clone()).mul(se)),
+                    se.clone(),
+                    schema,
+                    &mut first_stage_aggs,
+                );
+
+                // Second stage: sum(count), sum(sum), sum(squared_sum), sum(cubed_sum)
+                let sum_count_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(count_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+                let sum_sum_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(sum_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+                let sum_squared_sum_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(squared_sum_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+                let sum_cubed_sum_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(cubed_sum_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+
+                // Final projection: Given
+                // - sum(count) = N
+                // - sum(sum) = S
+                // - sum(squared_sum) = S2
+                // - sum(cubed_sum) = S3
+                //         S3 - 3 * S2 * S / N + 2 * S^3 / N^2
+                // Skew = -------------------------------------
+                //          N * ((S2 - S^2 / N) / N) ^ (3/2)
+
+                let n = resolved_col(sum_count_id);
+                let s = resolved_col(sum_sum_id);
+                let s2 = resolved_col(sum_squared_sum_id);
+                let s3 = resolved_col(sum_cubed_sum_id);
+
+                let denom_base = s2
+                    .clone()
+                    .sub(s.clone().mul(s.clone()).div(n.clone()))
+                    .div(n.clone());
+                let denom = sqrt::sqrt(denom_base.clone().mul(denom_base.clone()).mul(denom_base));
+
+                let numerator = s3.sub(lit(3).mul(s2).mul(s.clone()).div(n.clone())).add(
+                    lit(2)
+                        .mul(s.clone())
+                        .mul(s.clone())
+                        .mul(s)
+                        .div(n.clone())
+                        .div(n.clone()),
+                );
+
+                let result = numerator.div(denom).div(n).alias(output_name);
+                final_exprs.push(result);
+            }
             AggExpr::MapGroups { func, inputs } => {
                 let func_id = agg_expr.semantic_id(schema).id;
                 // No first stage aggregation for MapGroups, do all the work in the second stage.
diff --git a/src/daft-recordbatch/src/lib.rs b/src/daft-recordbatch/src/lib.rs
index f338ded036..f19c25693e 100644
--- a/src/daft-recordbatch/src/lib.rs
+++ b/src/daft-recordbatch/src/lib.rs
@@ -599,6 +599,9 @@ impl RecordBatch {
             AggExpr::Concat(expr) => self
                 .eval_expression(&BoundExpr::new_unchecked(expr.clone()))?
                 .agg_concat(groups),
+            AggExpr::Skew(expr) => self
+                .eval_expression(&BoundExpr::new_unchecked(expr.clone()))?
+                .skew(groups),
             AggExpr::MapGroups { .. } => Err(DaftError::ValueError(
                 "MapGroups not supported via aggregation, use map_groups instead".to_string(),
             )),
diff --git a/src/daft-sql/src/modules/aggs.rs b/src/daft-sql/src/modules/aggs.rs
index b642e0ab53..4806091dbf 100644
--- a/src/daft-sql/src/modules/aggs.rs
+++ b/src/daft-sql/src/modules/aggs.rs
@@ -159,6 +159,7 @@ fn to_expr(expr: &AggExpr, args: &[ExprRef]) -> SQLPlannerResult<ExprRef> {
         AggExpr::Concat(_) => unsupported_sql_err!("concat"),
         AggExpr::MapGroups { .. } => unsupported_sql_err!("map_groups"),
         AggExpr::Set(_) => unsupported_sql_err!("set"),
+        AggExpr::Skew(_) => unsupported_sql_err!("skew"),
     }
 }
 
diff --git a/tests/dataframe/test_aggregations.py b/tests/dataframe/test_aggregations.py
index 1b223b3fb6..c27fa119b7 100644
--- a/tests/dataframe/test_aggregations.py
+++ b/tests/dataframe/test_aggregations.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import math
 from dataclasses import dataclass
 
 import numpy as np
@@ -626,6 +627,27 @@ def test_agg_any_value_ignore_nulls(make_df, repartition_nparts, with_morsel_siz
     assert mapping == {1: 2, 2: 4, 3: None}
 
 
+@pytest.mark.parametrize("repartition_nparts", [1, 2, 4])
+def test_agg_skew(make_df, repartition_nparts, with_morsel_size):
+    daft_df = make_df(
+        {
+            "group": [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4],
+            "values": [1, 3, 2, 4, None, 4, 1, 1, 2, None, None, None],
+        },
+        repartition=repartition_nparts,
+    )
+    daft_df = daft_df.groupby("group").agg(col("values").skew().alias("skew"))
+
+    daft_df.collect()
+    res = daft_df.to_pydict()
+    mapping = {res["group"][i]: res["skew"][i] for i in range(len(res["group"]))}
+    assert len(mapping) == 4
+    assert mapping[1] == 0.0
+    assert math.isnan(mapping[2])
+    assert math.isclose(mapping[3], 0.70710678, rel_tol=1e-5)
+    assert mapping[4] is None
+
+
 @pytest.mark.parametrize("repartition_nparts", [1, 2, 4])
 def test_agg_with_non_agg_expr_global(make_df, repartition_nparts, with_morsel_size):
     daft_df = make_df(


Reviews:
user: login: srilman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T01:37:53Z
user: login: srilman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T01:38:46Z
user: login: srilman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T01:39:47Z
user: login: srilman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T01:40:09Z
user: login: universalmind303
body: lgtm!


Definitely not something we need to do here, but it might be worth thinking about how we can further simplify the agg exprs like we've been doing with scalar exprs.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-14T14:51:25Z

Review comments:
diff_hunk: @@ -0,0 +1,36 @@
+use arrow2::array::PrimitiveArray;
+use common_error::DaftResult;
+
+use crate::{
+    array::{
+        ops::{DaftSkewAggable, GroupIndices},
+        DataArray,
+    },
+    datatypes::Float64Type,
+    utils::stats,
+};
+
+impl DaftSkewAggable for DataArray<Float64Type> {
+    type Output = DaftResult<Self>;
+
+    fn skew(&self) -> Self::Output {
+        let stats = stats::calculate_stats(self)?;
+        let values = self.into_iter().flatten().copied();
+        let skew = stats::calculate_skew(stats, values);
+        let field = self.field.clone();
+        let data = PrimitiveArray::<f64>::from([skew]).boxed();
+        Self::new(field, data)
+    }
+
+    fn grouped_skew(&self, groups: &GroupIndices) -> Self::Output {
+        let grouped_skew_iter = stats::grouped_stats(self, groups)?.map(|(stats, group)| {
path: src/daft-core/src/array/ops/skew.rs
user: login: srilman
body: Potentially dumb question, but why do we need two variants. Seems like the `grouped_` variant can always be derived from the regular.
created_at: 2025-05-14T01:37:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -80,3 +80,18 @@ pub fn calculate_stddev(stats: Stats, values: impl Iterator<Item = f64>) -> Opti
         (sum_of_squares / stats.count).sqrt()
     })
 }
+
+pub fn calculate_skew(stats: Stats, values: impl Iterator<Item = f64>) -> Option<f64> {
+    let count = stats.count;
+    stats.mean.map(|mean| {
+        // In order to use the same iterator for 2 different calculations
+        let (m3, m2) = values.fold((0., 0.), |(m3_acc, m2_acc), v| {
+            (
+                m3_acc + (v - mean).powi(3),
+                (v - mean).mul_add(v - mean, m2_acc),
path: src/daft-core/src/utils/stats.rs
user: login: srilman
body: Let me know if there's a better way to do this. Couldn't figure out a good way to reuse / copy the iterator so just calculated both aggregations in the same iterator
created_at: 2025-05-14T01:38:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -938,6 +939,92 @@ pub fn populate_aggregation_stages(
                     ));
                 final_exprs.push(resolved_col(concat_of_concat_id.clone()).alias(output_name));
             }
+            AggExpr::Skew(se) => {
+                // See https://github.com/duckdb/duckdb/blob/93fda3591f4298414fa362c59219c09e03f718ab/extension/core_functions/aggregate/distributive/skew.cpp#L16
+                // Not exactly the same since they normalize by N - 1
+                let se = se.clone().cast(&DataType::Float64);
+
+                // Global count, sum, squared_sum, and cubed_sum are required for final expr
+
+                // First stage: count, sum, squared_sum, cubed_sum
+                let count_id = add_to_stage(
+                    |se| AggExpr::Count(se, CountMode::Valid),
+                    se.clone(),
+                    schema,
+                    &mut first_stage_aggs,
+                );
+                let sum_id = add_to_stage(AggExpr::Sum, se.clone(), schema, &mut first_stage_aggs);
+                let squared_sum_id = add_to_stage(
+                    |se| AggExpr::Sum(se.clone().mul(se)),
+                    se.clone(),
+                    schema,
+                    &mut first_stage_aggs,
+                );
+                let cubed_sum_id = add_to_stage(
+                    |se| AggExpr::Sum(se.clone().mul(se.clone()).mul(se)),
+                    se.clone(),
+                    schema,
+                    &mut first_stage_aggs,
+                );
+
+                // Second stage: sum(count), sum(sum), sum(squared_sum), sum(cubed_sum)
+                let sum_count_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(count_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+                let sum_sum_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(sum_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+                let sum_squared_sum_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(squared_sum_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+                let sum_cubed_sum_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(cubed_sum_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+
+                // Final projection: Given
+                // - sum(count) = N
+                // - sum(sum) = S
+                // - sum(squared_sum) = S2
+                // - sum(cubed_sum) = S3
+                //         S3 - 3 * S2 * S / N + 2 * S^3 / N^2
+                // Skew = -------------------------------------
+                //          N * ((S2 - S^2 / N) / N) ^ (3/2)
+
+                let n = resolved_col(sum_count_id);
+                let s = resolved_col(sum_sum_id);
+                let s2 = resolved_col(sum_squared_sum_id);
+                let s3 = resolved_col(sum_cubed_sum_id);
+
+                let denom_base = s2
+                    .clone()
+                    .sub(s.clone().mul(s.clone()).div(n.clone()))
+                    .div(n.clone());
+                let denom = sqrt::sqrt(denom_base.clone().mul(denom_base.clone()).mul(denom_base));
+
+                let numerator = s3.sub(lit(3).mul(s2).mul(s.clone()).div(n.clone())).add(
+                    lit(2)
+                        .mul(s.clone())
+                        .mul(s.clone())
+                        .mul(s)
+                        .div(n.clone())
+                        .div(n.clone()),
+                );
+
+                let result = numerator.div(denom).div(n).alias(output_name);
+                final_exprs.push(result);
path: src/daft-physical-plan/src/physical_planner/translate.rs
user: login: srilman
body: I did it like this using the DuckDB algorithm instead of just accumulation and calling skew because I couldn't really figure out how to implement a `list_skew` on ListArrays. Kept running into issues 
created_at: 2025-05-14T01:39:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -938,6 +939,92 @@ pub fn populate_aggregation_stages(
                     ));
                 final_exprs.push(resolved_col(concat_of_concat_id.clone()).alias(output_name));
             }
+            AggExpr::Skew(se) => {
+                // See https://github.com/duckdb/duckdb/blob/93fda3591f4298414fa362c59219c09e03f718ab/extension/core_functions/aggregate/distributive/skew.cpp#L16
+                // Not exactly the same since they normalize by N - 1
+                let se = se.clone().cast(&DataType::Float64);
+
+                // Global count, sum, squared_sum, and cubed_sum are required for final expr
+
+                // First stage: count, sum, squared_sum, cubed_sum
+                let count_id = add_to_stage(
+                    |se| AggExpr::Count(se, CountMode::Valid),
+                    se.clone(),
+                    schema,
+                    &mut first_stage_aggs,
+                );
+                let sum_id = add_to_stage(AggExpr::Sum, se.clone(), schema, &mut first_stage_aggs);
+                let squared_sum_id = add_to_stage(
+                    |se| AggExpr::Sum(se.clone().mul(se)),
+                    se.clone(),
+                    schema,
+                    &mut first_stage_aggs,
+                );
+                let cubed_sum_id = add_to_stage(
+                    |se| AggExpr::Sum(se.clone().mul(se.clone()).mul(se)),
+                    se.clone(),
+                    schema,
+                    &mut first_stage_aggs,
+                );
+
+                // Second stage: sum(count), sum(sum), sum(squared_sum), sum(cubed_sum)
+                let sum_count_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(count_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+                let sum_sum_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(sum_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+                let sum_squared_sum_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(squared_sum_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+                let sum_cubed_sum_id = add_to_stage(
+                    AggExpr::Sum,
+                    resolved_col(cubed_sum_id),
+                    schema,
+                    &mut second_stage_aggs,
+                );
+
+                // Final projection: Given
+                // - sum(count) = N
+                // - sum(sum) = S
+                // - sum(squared_sum) = S2
+                // - sum(cubed_sum) = S3
+                //         S3 - 3 * S2 * S / N + 2 * S^3 / N^2
+                // Skew = -------------------------------------
+                //          N * ((S2 - S^2 / N) / N) ^ (3/2)
+
+                let n = resolved_col(sum_count_id);
+                let s = resolved_col(sum_sum_id);
+                let s2 = resolved_col(sum_squared_sum_id);
+                let s3 = resolved_col(sum_cubed_sum_id);
+
+                let denom_base = s2
+                    .clone()
+                    .sub(s.clone().mul(s.clone()).div(n.clone()))
+                    .div(n.clone());
+                let denom = sqrt::sqrt(denom_base.clone().mul(denom_base.clone()).mul(denom_base));
+
+                let numerator = s3.sub(lit(3).mul(s2).mul(s.clone()).div(n.clone())).add(
+                    lit(2)
+                        .mul(s.clone())
+                        .mul(s.clone())
+                        .mul(s)
+                        .div(n.clone())
+                        .div(n.clone()),
+                );
+
+                let result = numerator.div(denom).div(n).alias(output_name);
+                final_exprs.push(result);
path: src/daft-physical-plan/src/physical_planner/translate.rs
user: login: srilman
body: If you happen to think that would be easier or have guidance for the next time, just let me know
created_at: 2025-05-14T01:40:09Z
author_association: CONTRIBUTOR

PR comments:
user: login: srilman
created_at: 2025-05-14T01:40:56Z
body: @colin-ho Let me know if someone else would be better to have review. Just tagging you for review initially
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-05-14T20:25:31Z
body: ## [Codecov](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) Report
Attention: Patch coverage is `90.28571%` with `17 lines` in your changes missing coverage. Please review.
> Project coverage is 78.56%. Comparing base [(`8c10e5b`)](https://app.codecov.io/gh/Eventual-Inc/Daft/commit/8c10e5b5493ba5a4278eed60a01665e6c6ddebaf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) to head [(`ac9aa3f`)](https://app.codecov.io/gh/Eventual-Inc/Daft/commit/ac9aa3f41fcf20b638701062281d4beb81054725?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc).
> Report is 7 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Patch % | Lines |
|---|---|---|
| [src/daft-core/src/datatypes/agg\_ops.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-core%2Fsrc%2Fdatatypes%2Fagg_ops.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29yZS9zcmMvZGF0YXR5cGVzL2FnZ19vcHMucnM=) | 33.33% | [6 Missing :warning: ](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) |
| [src/daft-core/src/series/ops/agg.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-core%2Fsrc%2Fseries%2Fops%2Fagg.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29yZS9zcmMvc2VyaWVzL29wcy9hZ2cucnM=) | 71.42% | [4 Missing :warning: ](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) |
| [daft/dataframe/dataframe.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=daft%2Fdataframe%2Fdataframe.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9kYXRhZnJhbWUvZGF0YWZyYW1lLnB5) | 25.00% | [3 Missing :warning: ](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) |
| [src/daft-logical-plan/src/ops/project.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Fops%2Fproject.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9vcHMvcHJvamVjdC5ycw==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) |
| [src/daft-sql/src/modules/aggs.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-sql%2Fsrc%2Fmodules%2Faggs.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtc3FsL3NyYy9tb2R1bGVzL2FnZ3MucnM=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) |

<details><summary>Additional details and impacted files</summary>


[![Impacted file tree graph](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346/graphs/tree.svg?width=650&height=150&src=pr&token=J430QVFE89&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)

```diff
@@            Coverage Diff             @@
##             main    #4346      +/-   ##
==========================================
+ Coverage   78.21%   78.56%   +0.34%     
==========================================
  Files         814      820       +6     
  Lines      110992   109850    -1142     
==========================================
- Hits        86811    86299     -512     
+ Misses      24181    23551     -630     
```

| [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Coverage Δ | |
|---|---|---|
| [daft/expressions/expressions.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=daft%2Fexpressions%2Fexpressions.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9leHByZXNzaW9ucy9leHByZXNzaW9ucy5weQ==) | `95.91% <100.00%> (+0.01%)` | :arrow_up: |
| [src/daft-core/src/array/ops/skew.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-core%2Fsrc%2Farray%2Fops%2Fskew.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29yZS9zcmMvYXJyYXkvb3BzL3NrZXcucnM=) | `100.00% <100.00%> (ø)` | |
| [src/daft-core/src/datatypes/mod.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-core%2Fsrc%2Fdatatypes%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29yZS9zcmMvZGF0YXR5cGVzL21vZC5ycw==) | `40.00% <ø> (ø)` | |
| [src/daft-core/src/utils/stats.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-core%2Fsrc%2Futils%2Fstats.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29yZS9zcmMvdXRpbHMvc3RhdHMucnM=) | `98.43% <100.00%> (+0.43%)` | :arrow_up: |
| [src/daft-dsl/src/expr/mod.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-dsl%2Fsrc%2Fexpr%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtZHNsL3NyYy9leHByL21vZC5ycw==) | `79.63% <100.00%> (+0.21%)` | :arrow_up: |
| [src/daft-dsl/src/python.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-dsl%2Fsrc%2Fpython.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtZHNsL3NyYy9weXRob24ucnM=) | `90.38% <100.00%> (-1.49%)` | :arrow_down: |
| [...ft-physical-plan/src/physical\_planner/translate.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-physical-plan%2Fsrc%2Fphysical_planner%2Ftranslate.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtcGh5c2ljYWwtcGxhbi9zcmMvcGh5c2ljYWxfcGxhbm5lci90cmFuc2xhdGUucnM=) | `93.93% <100.00%> (+0.39%)` | :arrow_up: |
| [src/daft-recordbatch/src/lib.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-recordbatch%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtcmVjb3JkYmF0Y2gvc3JjL2xpYi5ycw==) | `85.81% <100.00%> (+0.05%)` | :arrow_up: |
| [src/daft-sql/src/modules/aggs.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=src%2Fdaft-sql%2Fsrc%2Fmodules%2Faggs.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtc3FsL3NyYy9tb2R1bGVzL2FnZ3MucnM=) | `68.04% <0.00%> (-0.71%)` | :arrow_down: |
| [daft/dataframe/dataframe.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree&filepath=daft%2Fdataframe%2Fdataframe.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9kYXRhZnJhbWUvZGF0YWZyYW1lLnB5) | `85.67% <25.00%> (-0.23%)` | :arrow_down: |
| ... and [3 more](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | |

... and [24 files with indirect coverage changes](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4346/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)

</details>
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: srilman
created_at: 2025-05-14T20:31:46Z
body: > Definitely not something we need to do here, but it might be worth thinking about how we can further simplify the agg exprs like we've been doing with scalar exprs.

Agree, yeah especially making it a bit more flexible for future additions.
author_association: CONTRIBUTOR