Repository: apache/datafusion (language: rust)

Issue:
Datafusion binary size has been getting bigger
### Is your feature request related to a problem or challenge?

The size of datafusion's binary has grown significantly in the last few releases

This likely leads to higher compile times as well as larger overall binary size

| version | size of `datafusion-cli` binary |
|--------|--------|
|  `main` at 57d1309ec0830738af79c1885e514e19f324b1aa | 92M |
| `43.0.0` | 87M |
| `42.0.0` | 83M |
| `41.0.0` | 72M |
| `40.0.0` | 69M | 
| `39.0.0` | 68M | 


The sizes are measured like this:

```shell
git checkout version
cd datafusion-cli
cargo build --release
du -h target/release/datafusion-cli
```

Also, people such as @g3blv have noticed that the WASM build has increased 50%:
https://github.com/apache/datafusion/discussions/9834#discussioncomment-11574581

### Describe the solution you'd like

I would like to reduce the binary size of DataFusion if possible

At least I would like to understand where the code size comes from and offer hints about how to reduce the size if needed

### Describe alternatives you've considered

A common source of code size is templated functions (as that generates multiple copies of the same function(s)). 


Here is some fascianting information from running `cargo bloat -p datafusion`

```shell
 File  .text    Size                          Crate Name
 0.1%   0.3% 79.7KiB                         blake2 blake2::Blake2bVarCore::compress
 0.1%   0.2% 70.7KiB                         blake2 blake2::Blake2sVarCore::compress
 0.1%   0.2% 67.1KiB                      sqlparser <sqlparser::ast::Statement as core::fmt::Display>::fmt
 0.1%   0.2% 61.4KiB                         blake3 _blake3_hash4_neon
 0.1%   0.2% 56.4KiB                      chrono_tz <chrono_tz::timezones::Tz as chrono_tz::timezone_impl::TimeSpans>::timespans
 0.1%   0.2% 44.7KiB                     arrow_cast <i64 as lexical_write_integer::api::ToLexical>::to_lexical
 0.1%   0.1% 42.8KiB                     arrow_cast arrow_cast::cast::cast_with_options
 0.0%   0.1% 35.9KiB                           rand <rand_chacha::chacha::ChaCha12Core as rand_core::block::BlockRngCore>::generate
 0.0%   0.1% 34.9KiB                     arrow_cast lexical_parse_float::slow::parse_mantissa
 0.0%   0.1% 33.1KiB                     arrow_cast lexical_parse_float::parse::parse_complete
 0.0%   0.1% 33.1KiB                     arrow_cast lexical_parse_float::parse::parse_complete
 0.0%   0.1% 29.0KiB                 regex_automata regex_automata::hybrid::search::find_fwd
 0.0%   0.1% 27.6KiB                         blake3 blake3::portable::compress_in_place
 0.0%   0.1% 27.1KiB                   aho_corasick aho_corasick::automaton::try_find_fwd
 0.0%   0.1% 25.2KiB                      sqlparser <sqlparser::ast::Expr as core::fmt::Display>::fmt
 0.0%   0.1% 23.8KiB              datafusion_common datafusion_common::scalar::ScalarValue::iter_to_array
 0.0%   0.1% 23.7KiB              datafusion_common datafusion_common::scalar::ScalarValue::iter_to_array
 0.0%   0.1% 23.7KiB       datafusion_physical_expr datafusion_common::scalar::ScalarValue::iter_to_array
 0.0%   0.1% 23.7KiB datafusion_functions_aggregate datafusion_common::scalar::ScalarValue::iter_to_array
 0.0%   0.1% 22.0KiB                     arrow_cast <u64 as lexical_write_integer::api::ToLexical>::to_lexical
36.7%  97.4% 27.7MiB                                And 139272 smaller methods. Use -n N to show more.
37.7% 100.0% 28.4MiB                                .text section size, the file size is 75.4MiB
```

### Additional context

_No response_


Discussion / hints:
```
print_functions_docs
print_functions_config
```
binaries can be moved out from the main release
Some good experiments are https://github.com/johnthagen/min-sized-rust?tab=readme-ov-file#optimize-libstd-with-xargo

with this profile

```
[profile.release]
codegen-units = 1
strip = true
panic = "abort"
```

The cli size went from `94.3MiB` down to `48.6MiB` 🤔 
That is a very cool page 🤔 
```
[profile.release]
codegen-units = 1
strip = true
panic = "abort"
opt-level = "s"
```

Expanding on @comphead's idea adding opt-level = "s" reduced the size of the cli from 52MB on my machine (with his changes) to 37MB.
Thanks @Omega359 Opt-level is 3 by default for the release https://doc.rust-lang.org/cargo/reference/profiles.html#release
which focus on maximum runtime speed, I think it is important. However we can strip things to make executable smaller, I'll create a PR soon
FWIW I don't think the size of hte datafusion-cli binary is all that critical per se (maybe we can adjust / optimize the size of what is distributed on homebrew)

What I was hoping to review / improve with this PR is the size of the code in general (and review if there were places that were unecessairly causing code bloat that weren't also adding value
At a high level, I think this ticket has 2 parts:
1. Figure out what is contributing to code size increase
2. Then perhaps figure out how to make it better

I think the most valuable (and hardest) part is 1 (figuring out what to do)

To do so I recommend doing an ["Ablation Study"](https://en.wikipedia.org/wiki/Ablation_(artificial_intelligence))

> An ablation study aims to determine the contribution of a component to an AI system by removing the component, and then analyzing the resultant performance of the system.[[2]](https://en.wikipedia.org/wiki/Ablation_(artificial_intelligence)#cite_note-2)

This is a fancy way of saying "remove parts of the system and see how much impact it makes on binary size"

# Suggested things to try
I suggest initially simply trying with different datafusion crate features and see how much extra code each contributes to the binary size.

A follow on idea would be to comment out some of the  features that require lots of generic code such as

* REmove (all implementations of `GroupValues`: https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/aggregates/group_values/mod.rs
* Remvoe all implementations of `GroupAccumulator`: https://github.com/apache/datafusion/blob/469f18be1c594b07e4b235f3404419792ed3c24f/datafusion/expr-common/src/groups_accumulator.rs#L108


## Example of Ablation Study for the `parquet` feature
For example, to test the impact of the `parquet` feature, I tested the size of the binary with and without parquet support

```shell
cargo build --release -p datafusion-cli
# get size of datafusion-cli in kb
du -k target/release/datafusion-cli
```

Here is what I got

| type | size in mb | size in kb |
|--------|--------|--------|
| default | 58 | 58440 |
| without `parquet` feature | 53 | 54248 |

So I conclude that the parquet feature adds approximately 5mb to the binary size

To remove parquet support I hacked out the dependency on parquet. Since this was just to test the impact as long as it compiles that is good enough. No need to be pretty. Here is the 
- https://github.com/apache/datafusion/pull/14666


That is really awesome write up @alamb 
Tbh I wanted to do exactly the same but I even had no idea this approach has so smart definition `Ablative Study`.


> That is really awesome write up [@alamb](https://github.com/alamb) Tbh I wanted to do exactly the same but I even had no idea this approach has so smart definition `Ablative Study`.

LOL like most great things I think I first heard this term used in a paper by Viktor Leis and Thomas Neumann 
btw after changes in 45.0.0 the image size is 49M 🎉 
> btw after changes in 45.0.0 the image size is 49M 🎉

Nice! Do you know what changed? 

Indeed I checked on my mac after doing `cargo build --release` and the size is 58MB:

```shell
andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion$ du -s -h target/release/datafusion-cli
 58M	target/release/datafusion-cli
```
Tbh I dont have accurate answer for this, I found it when played with different feature set on latest DF, but I remember some packages were moved out of the core or similar.

Experimenting with different feature set and --no-default-features keeps the binary size the same.

One more optimization is to strip symbols and debug info which takes some space considering our code size and it saves 20% more going down to 50M


Fix patch:
diff --git a/Cargo.toml b/Cargo.toml
index f7d39aeb3003..d0710113d47b 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -159,19 +159,18 @@ url = "2.5.4"
 [profile.release]
 codegen-units = 1
 lto = true
+strip = true
 
 # the release profile takes a long time to build so we can use this profile during development to save time
 # cargo build --profile release-nonlto
 [profile.release-nonlto]
 codegen-units = 16
-debug = false
 debug-assertions = false
 incremental = false
 inherits = "release"
 lto = false
 opt-level = 3
 overflow-checks = false
-panic = 'unwind'
 rpath = false
 
 [profile.ci]


Test patch: