Repository: pydantic/pydantic

Issue:
Change default cython flags to have smaller compiled files
Pip install results in a much larger lib artefact compared to conda install (both in a conda environment):
- `pip install pydantic`: 80M
- `conda install pydantic -c conda-forge`: 6.6M

I don't know if this is expected or not. I encountered this while trying to minimize the size of a Docker container I'm building, and was surprised that pydantic took up 80M when installed with pip.

I wasn't sure to file this as bug or not, but given the extreme difference in size I thought there might be something going wrong with the pip install.

I've added a full list of the files in `site-packages`:
## Pip install file sizes 
```
> du -sh /opt/conda/lib/python3.8/site-packages/pydantic/* | sort -h
0	/opt/conda/lib/python3.8/site-packages/pydantic/py.typed
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/json.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/version.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/generics.py
12K	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.py
12K	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.py
12K	/opt/conda/lib/python3.8/site-packages/pydantic/typing.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/errors.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/networks.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/color.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/validators.py
24K	/opt/conda/lib/python3.8/site-packages/pydantic/utils.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/types.py
32K	/opt/conda/lib/python3.8/site-packages/pydantic/fields.py
40K	/opt/conda/lib/python3.8/site-packages/pydantic/main.py
40K	/opt/conda/lib/python3.8/site-packages/pydantic/schema.py
196K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so
332K	/opt/conda/lib/python3.8/site-packages/pydantic/__pycache__
516K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.cpython-38-x86_64-linux-gnu.so
552K	/opt/conda/lib/python3.8/site-packages/pydantic/version.cpython-38-x86_64-linux-gnu.so
684K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.cpython-38-x86_64-linux-gnu.so
948K	/opt/conda/lib/python3.8/site-packages/pydantic/json.cpython-38-x86_64-linux-gnu.so
1.4M	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.cpython-38-x86_64-linux-gnu.so
1.5M	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.cpython-38-x86_64-linux-gnu.so
1.6M	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.cpython-38-x86_64-linux-gnu.so
1.7M	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.cpython-38-x86_64-linux-gnu.so
2.0M	/opt/conda/lib/python3.8/site-packages/pydantic/typing.cpython-38-x86_64-linux-gnu.so
2.1M	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.cpython-38-x86_64-linux-gnu.so
2.7M	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.cpython-38-x86_64-linux-gnu.so
3.4M	/opt/conda/lib/python3.8/site-packages/pydantic/color.cpython-38-x86_64-linux-gnu.so
4.0M	/opt/conda/lib/python3.8/site-packages/pydantic/networks.cpython-38-x86_64-linux-gnu.so
5.1M	/opt/conda/lib/python3.8/site-packages/pydantic/validators.cpython-38-x86_64-linux-gnu.so
5.2M	/opt/conda/lib/python3.8/site-packages/pydantic/utils.cpython-38-x86_64-linux-gnu.so
6.2M	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.cpython-38-x86_64-linux-gnu.so
6.4M	/opt/conda/lib/python3.8/site-packages/pydantic/errors.cpython-38-x86_64-linux-gnu.so
6.4M	/opt/conda/lib/python3.8/site-packages/pydantic/fields.cpython-38-x86_64-linux-gnu.so
8.0M	/opt/conda/lib/python3.8/site-packages/pydantic/schema.cpython-38-x86_64-linux-gnu.so
8.5M	/opt/conda/lib/python3.8/site-packages/pydantic/main.cpython-38-x86_64-linux-gnu.so
11M	/opt/conda/lib/python3.8/site-packages/pydantic/types.cpython-38-x86_64-linux-gnu.so
```

## Conda install file sizes:
```
> du -sh /opt/conda/lib/python3.8/site-packages/pydantic/* | sort -h
0	/opt/conda/lib/python3.8/site-packages/pydantic/py.typed
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/json.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/version.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/generics.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/typing.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/errors.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/networks.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/validators.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/color.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/utils.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/types.py
32K	/opt/conda/lib/python3.8/site-packages/pydantic/fields.py
36K	/opt/conda/lib/python3.8/site-packages/pydantic/main.py
36K	/opt/conda/lib/python3.8/site-packages/pydantic/schema.py
48K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so
80K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.cpython-38-x86_64-linux-gnu.so
84K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.cpython-38-x86_64-linux-gnu.so
88K	/opt/conda/lib/python3.8/site-packages/pydantic/version.cpython-38-x86_64-linux-gnu.so
116K	/opt/conda/lib/python3.8/site-packages/pydantic/json.cpython-38-x86_64-linux-gnu.so
144K	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.cpython-38-x86_64-linux-gnu.so
156K	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.cpython-38-x86_64-linux-gnu.so
164K	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.cpython-38-x86_64-linux-gnu.so
188K	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.cpython-38-x86_64-linux-gnu.so
188K	/opt/conda/lib/python3.8/site-packages/pydantic/typing.cpython-38-x86_64-linux-gnu.so
192K	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.cpython-38-x86_64-linux-gnu.so
280K	/opt/conda/lib/python3.8/site-packages/pydantic/errors.cpython-38-x86_64-linux-gnu.so
296K	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.cpython-38-x86_64-linux-gnu.so
308K	/opt/conda/lib/python3.8/site-packages/pydantic/networks.cpython-38-x86_64-linux-gnu.so
324K	/opt/conda/lib/python3.8/site-packages/pydantic/__pycache__
356K	/opt/conda/lib/python3.8/site-packages/pydantic/color.cpython-38-x86_64-linux-gnu.so
360K	/opt/conda/lib/python3.8/site-packages/pydantic/utils.cpython-38-x86_64-linux-gnu.so
452K	/opt/conda/lib/python3.8/site-packages/pydantic/fields.cpython-38-x86_64-linux-gnu.so
456K	/opt/conda/lib/python3.8/site-packages/pydantic/validators.cpython-38-x86_64-linux-gnu.so
472K	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.cpython-38-x86_64-linux-gnu.so
480K	/opt/conda/lib/python3.8/site-packages/pydantic/types.cpython-38-x86_64-linux-gnu.so
572K	/opt/conda/lib/python3.8/site-packages/pydantic/main.cpython-38-x86_64-linux-gnu.so
572K	/opt/conda/lib/python3.8/site-packages/pydantic/schema.cpython-38-x86_64-linux-gnu.so
```

# Bug

Output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())"` from conda install:
```
root@c6b3b43340a6:/# python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.6.1
            pydantic compiled: True
                 install path: /opt/conda/lib/python3.8/site-packages/pydantic
               python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]
                     platform: Linux-5.8.0-38-generic-x86_64-with-glibc2.10
     optional deps. installed: []
```

Output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())"` from pip install:
```
root@c6b3b43340a6:/# python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.7.3
            pydantic compiled: True
                 install path: /opt/conda/lib/python3.8/site-packages/pydantic
               python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]
                     platform: Linux-5.8.0-38-generic-x86_64-with-glibc2.10
     optional deps. installed: []
```

conda info:
```
     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.8.5.final.0
       virtual packages : __glibc=2.28=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/conda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.25.1 CPython/3.8.5 Linux/5.8.0-38-generic debian/10 glibc/2.28
                UID:GID : 0:0
             netrc file : None
           offline mode : False
```


Discussion:
I noticed there is a version difference, however, even with `conda install pydantic=1.7.3 -c conda-forge` I only get a 7.0M artefact.
Hello @peterroelants and thanks for reporting.
Your issue comes from compiled files with `cython` (the `.so` files).
Can you remove everything and try again with `SKIP_CYTHON=1 pip install pydantic`?
I uninstalled the conda pydantic and installed it again with `SKIP_CYTHON=1 pip install pydantic`. After pip install I get a 80M large `/opt/conda/lib/python3.8/site-packages/pydantic/` folder. The files seem to be the same as the one I posted in "Pip install file sizes" above.

Could it be that cython is not skipped?
Can you try clone the repo and build it with the `SKIP_CYTHON` env variable ?
If you download from `pypi`, the flag will be useless since it already contains the pre-compiles binaries!
Or you can simply remove all the `*.so` and use only the `*.py` files.

-----
On linux the compiled files are big because a lot of optimizations are done to improve perf but at the cost of quite some space.
We could also maybe improve the `cythonize` parameters. Currently we use only the `-O3` flag but maybe `-Oz`or others would be possible and better?
Note that the files in the conda build are the same, it also contains the `.so` files, they are just of smaller size.

What is the sequence of commands I need to run to build the repo?
You might be right that it's related to the cython flags, see this comment: https://github.com/cython/cython/issues/2102#issuecomment-397755796
I finally got my hands on a ubuntu and tested with default `CFLAGS` and `-Os -g0`.

```bash
(env) ~/pydantic-test/pydantic$ du -sh build
13M	build
(env) ~/pydantic-test/pydantic$ du -sh build-original
74M	build-original
```

You can try it out yourself by just cloning it, change in `setup.py` -> `os.environ['CFLAGS'] = '-Os -g0'`
and run `python setup.py build_ext`

Now I don't know if we should make the change directly in pydantic as `-O3` optimizes performance over size, which makes sense to me. And you can always cythonize yourself. Maybe there is a trade-off in the flags to make or just add some documentation
Thanks for verifying.

I see that `-Os` optimizes for size. From the [gcc docs](https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html): "`-Os` enables all `-O2` optimizations except those that often increase code size.".
`-g0` just seems to [turn of debugging](https://gcc.gnu.org/onlinedocs/gcc-5.3.0/gcc/Debugging-Options.html#Debugging-Options).

However, I haven't been able to reproduce your build results. `python setup.py build_ext` doesn't produce a `build` folder.
When using `python setup.py build install && pip install .` I'm only gettiung a 712K folder for pydantic in `site-packages`. I'm using conda from the `continuumio/miniconda3:4.9.2` container to run these experiments.
This is currently in "feedback wanted" state. What kind of feedback is wanted?

The big binaries are problematic enough in that they warranted a special warning here: https://awslabs.github.io/aws-lambda-powertools-python/utilities/parser/

Even if the optimized binaries were 2x faster, they wouldn't be worth the size increase in modern workloads (lambdas, containers etc) - and people needing to extract the last bit of performance should be able to compile their own version of pydantic

This has already been discussed here: https://github.com/samuelcolvin/pydantic/issues/1325.  The .so files were an issue in AWS Lambda already back then but now the situation is even worse as the files have grown a lot. (I actually already hit the limits in one of my AWS Lambda deployments and after investigating why the package was so big I found this discussion.) In order FastAPI to become the next recommended framework for serverless APIs this major dependency should be more lightweight (i.e. smaller). Installing packages straight from Git repository doesn't sound right.
First of all, I'm surprised and disappointed by how impassioned people get without taking the time to think about a work around for a problem. Indignance needs to be matched by investigation.

Remember pydantic isn't developed or sponsored by some big organisation, I don't get paid to develop it (well, what I get from github sponsors represents a lot less than the minimum wage). Myself and other worked hard to compile pydantic because some people cared a lot about performance.

---

If when installing pydantic you care more about space than performance, spend 3 minutes reading [the docs](https://pip.pypa.io/en/stable/reference/pip_install/#install-no-binary) on pip and you'll notice it's trivial to install pydantic without downloading binaries:

    pip install --no-binary pydantic pydantic

I believe pydantic is around 2x slower when not compiled.

---

In terms of reducing the size of the binaries and fixing this issue, please can someone who's concerned about this compile pydantic with different flags, run the benchmarks and provide a breakdown here of performance vs. size for different options.
Even better, you can custom compile it again during pip install with preferred flags to optimize size vs speed. For example:
```
CFLAGS="-Os -g0 -s" pip install --no-binary pydantic --global-option=build_ext --global-option=-j8 pydantic
```
Will result in a around 7MB pydantic install.

Note that this will need build tools (e.g. `build-essential` on debian/ubuntu) and Cython installed when pip installing.

I only discovered this option myself last week, and was in the process of writing a post on it since it seems to be generally unknown pip install option.
Don't get at least me wrong. I love Pydantic and I see there a lot of more potential than just as a main dependency to FastAPI. Actually I see huge momentum and potential when it comes to defining any kind of contracts between components and services building larger microservice architectures. With Pydantic you can easily share the DTOs between services and clients and by that avoid writing a lot of boilerplate and validation code yourself. Of course FastAPI is great as well as it provides asyncio but even bigger reason why I'm about to switch to FastAPI is that it comes with Pydantic.

I can imagine that FastAPI benefits from speed optimized Pydantic. But when we find projects from big players, like AWS, who use Pydantic because of its typed approach to data structures and validation, we realize that this really is not only some sub component of FastAPI but very interesting library as its own. So don't be surprised if you suddenly get a sponsor. This discussion also shows that there are projects that value smaller disk footprint so the feedback is really valid.

Thanks for the tip for --no-binary. However, this approach isn't perfect either as bigger projects have a lot of other dependencies as well and they may not want to prevent pip from installing binaries for every component. Pip resolves version dependencies best when provided with the list of all components at a time so it's not optimal to install Pydantic first with special way and then the rest of the packages in a usual way.
@jvuori  If I understand it correctly, you can still install multiple packages using the `--no-binary` option. The `--no-binary` takes in an argument itself, for example `--no-binary pydantic` will only disable installing binaries from pypi for pydantic, while `--no-binary :all:` will disable it for all packages you want to install.
@peterroelants Nice! And that can be added also to requirements.txt as its own line. This is very much needed with toolchains like AWS SAM which installs the packages purely based on requirements.txt and you may not be able to affect on any extra parameters it passes to pip.
@peterroelants this is a very good tip, hopefully people will find it when googling about this problem
Very happy to accept a PR to add some notes about this to the install page of the docs.
@samuelcolvin I've run some tests and these are the results:

|      | size| benchmark avg time |
| --- | --- | --- |
| default `pip install` | 45M | 70.7μs |
| `pip install --no-binary` | 796K | 101.4μs |
| `pip install --no-binary` with cython | 6.4M | 67.8μs |

These are run on the [validation benchmark task](https://github.com/samuelcolvin/pydantic/blob/master/benchmarks/run.py#L199).

What is strange to me is that "_`pip install --no-binary` with cython_" is much than "_default `pip install`_ ". I would have expected  them to have the same size.

- "_default `pip install`_ " is the regular pip install.
- `pip install --no-binary`  is the no-binary option you suggested without Cython
- The installation for "_`pip install --no-binary` with cython_" is:
```
# Setup build and other requirements
pip install --no-cache-dir \
    Cython==0.29.22 \
    devtools==0.6.1 \
    pip \
    python-dateutil \
    setuptools
# Install PyDantic
pip install \
    --no-cache-dir \
    --no-binary pydantic \
    --global-option=build_ext \
    --global-option=-j8 \
    pydantic==1.8.1
```


Additionally I also updated the documentation and suggested a small change to `setup.py` to allow for custom CFLAGS passed by the user: https://github.com/samuelcolvin/pydantic/pull/2517
@peterroelants how did you manage to have a 6.4M installation package with cython? I did exactly the same and got 40M. 

The only way I can get ~5M is on windows. Anything in Linux pushes me up to 40M. What am I doing wrong here?
@alexanderluiscampino That's weird, I hope I didn't make a mistake.

I ran my tests on a `python:3.9.2-slim-buster` Docker image with a Python env in `/venv`. The size should be the output from `du -sh /venv/lib/python3.9/site-packages/pydantic`

Hi @peterroelants  here are my size results, I tested every possible form in this thread, always get 40M

there results below are for the command:
```
CFLAGS="-Os -g0 -s" pip install --no-binary pydantic --global-option=build_ext --global-option=-j8 pydantic
``` 
suggested by you. But whichever way I go about it, always get 40M. I'm running on Docker as well, 

```
FROM mcr.microsoft.com/vscode/devcontainers/python:3.8
```

```
$ du -sh ./src/python/pydantic/* | sort -h
0       ./src/python/pydantic/py.typed
1.0K    ./src/python/pydantic/version.py
2.0K    ./src/python/pydantic/parse.py
2.5K    ./src/python/pydantic/__init__.py
2.5K    ./src/python/pydantic/annotated_types.py
2.5K    ./src/python/pydantic/tools.py
3.5K    ./src/python/pydantic/json.py
5.0K    ./src/python/pydantic/error_wrappers.py
7.5K    ./src/python/pydantic/datetime_parse.py
9.0K    ./src/python/pydantic/env_settings.py
9.5K    ./src/python/pydantic/dataclasses.py
10K     ./src/python/pydantic/decorator.py
12K     ./src/python/pydantic/generics.py
13K     ./src/python/pydantic/typing.py
14K     ./src/python/pydantic/_hypothesis_plugin.py
14K     ./src/python/pydantic/class_validators.py
15K     ./src/python/pydantic/networks.py
16K     ./src/python/pydantic/errors.py
17K     ./src/python/pydantic/color.py
20K     ./src/python/pydantic/validators.py
22K     ./src/python/pydantic/utils.py
28K     ./src/python/pydantic/mypy.py
29K     ./src/python/pydantic/types.py
38K     ./src/python/pydantic/fields.py
41K     ./src/python/pydantic/main.py
42K     ./src/python/pydantic/schema.py
142K    ./src/python/pydantic/__init__.cpython-39-x86_64-linux-gnu.so
328K    ./src/python/pydantic/__pycache__
388K    ./src/python/pydantic/parse.cpython-39-x86_64-linux-gnu.so
415K    ./src/python/pydantic/tools.cpython-39-x86_64-linux-gnu.so
438K    ./src/python/pydantic/annotated_types.cpython-39-x86_64-linux-gnu.so
446K    ./src/python/pydantic/version.cpython-39-x86_64-linux-gnu.so
604K    ./src/python/pydantic/json.cpython-39-x86_64-linux-gnu.so
835K    ./src/python/pydantic/datetime_parse.cpython-39-x86_64-linux-gnu.so
954K    ./src/python/pydantic/error_wrappers.cpython-39-x86_64-linux-gnu.so
1.1M    ./src/python/pydantic/dataclasses.cpython-39-x86_64-linux-gnu.so
1.1M    ./src/python/pydantic/env_settings.cpython-39-x86_64-linux-gnu.so
1.2M    ./src/python/pydantic/decorator.cpython-39-x86_64-linux-gnu.so
1.3M    ./src/python/pydantic/_hypothesis_plugin.cpython-39-x86_64-linux-gnu.so
1.3M    ./src/python/pydantic/errors.cpython-39-x86_64-linux-gnu.so
1.3M    ./src/python/pydantic/typing.cpython-39-x86_64-linux-gnu.so
1.7M    ./src/python/pydantic/class_validators.cpython-39-x86_64-linux-gnu.so
1.8M    ./src/python/pydantic/color.cpython-39-x86_64-linux-gnu.so
1.8M    ./src/python/pydantic/networks.cpython-39-x86_64-linux-gnu.so
2.5M    ./src/python/pydantic/utils.cpython-39-x86_64-linux-gnu.so
2.9M    ./src/python/pydantic/mypy.cpython-39-x86_64-linux-gnu.so
3.0M    ./src/python/pydantic/validators.cpython-39-x86_64-linux-gnu.so
3.4M    ./src/python/pydantic/types.cpython-39-x86_64-linux-gnu.so
3.6M    ./src/python/pydantic/fields.cpython-39-x86_64-linux-gnu.so
3.9M    ./src/python/pydantic/main.cpython-39-x86_64-linux-gnu.so
4.0M    ./src/python/pydantic/schema.cpython-39-x86_64-linux-gnu.so
```
Weird, I remember checking the `.so` files, they were there, only much smaller than the ones you are getting.
Btw, the `CFLAGS="-Os -g0 -s"` options won't actually work until https://github.com/samuelcolvin/pydantic/pull/2517 is merged since Pydantic currently overwrites CFLAGS.

Let me try to reproduce my results, and maybe clean up the testing scripts for sharing. It will probably take me a few days to find the time.
@alexanderluiscampino I tried reproducing my results and I have a Dockerfile for you to try at https://gist.github.com/peterroelants/e344ac416948296f7fcdc84a20ce6eb5

For me this results in a working Python environment with Pydantic env of **6.4M**:
```
docker run pydantic-test-env /bin/bash -c 'du -sh /usr/local/lib/python3.9/site-packages/pydantic'
6.4M	/usr/local/lib/python3.9/site-packages/pydantic
```


```
docker run pydantic-test-env /bin/bash -c 'du -sh /usr/local/lib/python3.9/site-packages/pydantic/* | sort -h'
0	/usr/local/lib/python3.9/site-packages/pydantic/py.typed
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/__init__.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/annotated_types.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/json.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/parse.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/tools.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/version.py
8.0K	/usr/local/lib/python3.9/site-packages/pydantic/datetime_parse.py
8.0K	/usr/local/lib/python3.9/site-packages/pydantic/error_wrappers.py
12K	/usr/local/lib/python3.9/site-packages/pydantic/dataclasses.py
12K	/usr/local/lib/python3.9/site-packages/pydantic/decorator.py
12K	/usr/local/lib/python3.9/site-packages/pydantic/env_settings.py
12K	/usr/local/lib/python3.9/site-packages/pydantic/generics.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/_hypothesis_plugin.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/class_validators.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/errors.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/networks.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/typing.py
20K	/usr/local/lib/python3.9/site-packages/pydantic/color.py
20K	/usr/local/lib/python3.9/site-packages/pydantic/validators.py
24K	/usr/local/lib/python3.9/site-packages/pydantic/utils.py
28K	/usr/local/lib/python3.9/site-packages/pydantic/mypy.py
32K	/usr/local/lib/python3.9/site-packages/pydantic/types.py
36K	/usr/local/lib/python3.9/site-packages/pydantic/__init__.cpython-39-x86_64-linux-gnu.so
40K	/usr/local/lib/python3.9/site-packages/pydantic/fields.py
44K	/usr/local/lib/python3.9/site-packages/pydantic/main.py
44K	/usr/local/lib/python3.9/site-packages/pydantic/schema.py
68K	/usr/local/lib/python3.9/site-packages/pydantic/annotated_types.cpython-39-x86_64-linux-gnu.so
68K	/usr/local/lib/python3.9/site-packages/pydantic/parse.cpython-39-x86_64-linux-gnu.so
72K	/usr/local/lib/python3.9/site-packages/pydantic/tools.cpython-39-x86_64-linux-gnu.so
72K	/usr/local/lib/python3.9/site-packages/pydantic/version.cpython-39-x86_64-linux-gnu.so
96K	/usr/local/lib/python3.9/site-packages/pydantic/json.cpython-39-x86_64-linux-gnu.so
128K	/usr/local/lib/python3.9/site-packages/pydantic/datetime_parse.cpython-39-x86_64-linux-gnu.so
148K	/usr/local/lib/python3.9/site-packages/pydantic/error_wrappers.cpython-39-x86_64-linux-gnu.so
168K	/usr/local/lib/python3.9/site-packages/pydantic/env_settings.cpython-39-x86_64-linux-gnu.so
172K	/usr/local/lib/python3.9/site-packages/pydantic/dataclasses.cpython-39-x86_64-linux-gnu.so
180K	/usr/local/lib/python3.9/site-packages/pydantic/decorator.cpython-39-x86_64-linux-gnu.so
188K	/usr/local/lib/python3.9/site-packages/pydantic/typing.cpython-39-x86_64-linux-gnu.so
196K	/usr/local/lib/python3.9/site-packages/pydantic/_hypothesis_plugin.cpython-39-x86_64-linux-gnu.so
212K	/usr/local/lib/python3.9/site-packages/pydantic/errors.cpython-39-x86_64-linux-gnu.so
236K	/usr/local/lib/python3.9/site-packages/pydantic/class_validators.cpython-39-x86_64-linux-gnu.so
248K	/usr/local/lib/python3.9/site-packages/pydantic/networks.cpython-39-x86_64-linux-gnu.so
276K	/usr/local/lib/python3.9/site-packages/pydantic/color.cpython-39-x86_64-linux-gnu.so
320K	/usr/local/lib/python3.9/site-packages/pydantic/utils.cpython-39-x86_64-linux-gnu.so
372K	/usr/local/lib/python3.9/site-packages/pydantic/__pycache__
384K	/usr/local/lib/python3.9/site-packages/pydantic/mypy.cpython-39-x86_64-linux-gnu.so
388K	/usr/local/lib/python3.9/site-packages/pydantic/types.cpython-39-x86_64-linux-gnu.so
444K	/usr/local/lib/python3.9/site-packages/pydantic/validators.cpython-39-x86_64-linux-gnu.so
472K	/usr/local/lib/python3.9/site-packages/pydantic/fields.cpython-39-x86_64-linux-gnu.so
544K	/usr/local/lib/python3.9/site-packages/pydantic/schema.cpython-39-x86_64-linux-gnu.so
572K	/usr/local/lib/python3.9/site-packages/pydantic/main.cpython-39-x86_64-linux-gnu.so
```
Thank you for your dedication @peterroelants !

I managed to replicate your results using your dockerfile. I'm assuming the difference between your method and the mine, is that you do a clean installation of pip and cython from the beginning, whereas I already started with a pip installation. Somehow, when I ran the pydantic installation script (w/ cython) it wasn't using it, and defaulted to the normal way of installing it. 
I'm still trying to get to the bottom of this, and will update here. 
@alexanderluiscampino I can't replicate this, the docker build doesn't complete successfully and fails on processing the install of pydantic:
```#6 62.96     building 'pydantic.types' extension
#6 62.96     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O3 -fPIC -I/usr/local/include/python3.9 -c pydantic/types.c -o build/temp.linux-aarch64-3.9/pydantic/types.o
#6 64.44     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/tools.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/tools.cpython-39-aarch64-linux-gnu.so
#6 64.51     building 'pydantic.typing' extension
#6 64.51     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O3 -fPIC -I/usr/local/include/python3.9 -c pydantic/typing.c -o build/temp.linux-aarch64-3.9/pydantic/typing.o
#6 71.70     gcc: fatal error: Killed signal terminated program cc1
#6 71.70     compilation terminated.
#6 71.77     building 'pydantic.utils' extension
#6 71.78     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O3 -fPIC -I/usr/local/include/python3.9 -c pydantic/utils.c -o build/temp.linux-aarch64-3.9/pydantic/utils.o
#6 72.89     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/networks.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/networks.cpython-39-aarch64-linux-gnu.so
#6 73.00     building 'pydantic.validators' extension
#6 73.00     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O3 -fPIC -I/usr/local/include/python3.9 -c pydantic/validators.c -o build/temp.linux-aarch64-3.9/pydantic/validators.o
#6 73.11     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/typing.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/typing.cpython-39-aarch64-linux-gnu.so
#6 73.16     building 'pydantic.version' extension
#6 73.16     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O3 -fPIC -I/usr/local/include/python3.9 -c pydantic/version.c -o build/temp.linux-aarch64-3.9/pydantic/version.o
#6 74.82     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/version.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/version.cpython-39-aarch64-linux-gnu.so
#6 76.17     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/mypy.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/mypy.cpython-39-aarch64-linux-gnu.so
#6 76.32     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/errors.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/errors.cpython-39-aarch64-linux-gnu.so
#6 81.10     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/utils.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/utils.cpython-39-aarch64-linux-gnu.so
#6 82.53     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/schema.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/schema.cpython-39-aarch64-linux-gnu.so
#6 83.37     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/fields.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/fields.cpython-39-aarch64-linux-gnu.so
#6 83.60     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/validators.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/validators.cpython-39-aarch64-linux-gnu.so
#6 84.33     gcc -pthread -shared -Wl,--strip-all -O3 build/temp.linux-aarch64-3.9/pydantic/types.o -L/usr/local/lib -o build/lib.linux-aarch64-3.9/pydantic/types.cpython-39-aarch64-linux-gnu.so
#6 84.35     error: command '/usr/bin/gcc' failed with exit code 1
#6 84.75     Running setup.py install for pydantic: finished with status 'error'
#6 84.78 ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3v2xdeud/pydantic_d3c32ea573474b5a9ffefdfa0bbb9476/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3v2xdeud/pydantic_d3c32ea573474b5a9ffefdfa0bbb9476/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext -j8 install --record /tmp/pip-record-bb95clcq/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.9/pydantic Check the logs for full command output.
#6 84.82 Exception information:
#6 84.82 Traceback (most recent call last):
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 806, in install
#6 84.82     success = install_legacy(
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/install/legacy.py", line 81, in install
#6 84.82     raise LegacyInstallFailure
#6 84.82 pip._internal.operations.install.legacy.LegacyInstallFailure
#6 84.82 
#6 84.82 During handling of the above exception, another exception occurred:
#6 84.82 
#6 84.82 Traceback (most recent call last):
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
#6 84.82     status = self.run(options, args)
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 204, in wrapper
#6 84.82     return func(self, options, args)
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 393, in run
#6 84.82     installed = install_given_reqs(
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/req/__init__.py", line 77, in install_given_reqs
#6 84.82     requirement.install(
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 824, in install
#6 84.82     six.reraise(*exc.parent)
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_vendor/six.py", line 703, in reraise
#6 84.82     raise value
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/install/legacy.py", line 69, in install
#6 84.82     runner(
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/utils/subprocess.py", line 274, in runner
#6 84.82     call_subprocess(
#6 84.82   File "/usr/local/lib/python3.9/site-packages/pip/_internal/utils/subprocess.py", line 244, in call_subprocess
#6 84.82     raise InstallationSubprocessError(proc.returncode, command_desc)
#6 84.82 pip._internal.exceptions.InstallationSubprocessError: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3v2xdeud/pydantic_d3c32ea573474b5a9ffefdfa0bbb9476/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3v2xdeud/pydantic_d3c32ea573474b5a9ffefdfa0bbb9476/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext -j8 install --record /tmp/pip-record-bb95clcq/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.9/pydantic Check the logs for full command output.
```
Seems to blow up on compiling pydantic.typing extension every time for me, did you change anything in the gist posted by @peterroelants to get it to work?

I've tried manually setting different versions of pip, setuptools and cython but same issue each time, I cannot figure out why this would work for others and not me (I am on Macbook with M1 chip, but its docker so 🤷 )


@dotorg-richard you mention you are running on a Macbook with M1 chip, this means that the base image pulled will be of a different architecture. The [`python:3.9.2-slim-buster`](https://hub.docker.com/_/python?tab=tags&page=1&ordering=last_updated&name=3.9.2-slim-buster) base image should support arm64.

Can you try the following?:

Make sure you have the latest Docker version and fulfil the system requirements as documented in "[Docker Desktop for Apple silicon](https://docs.docker.com/docker-for-mac/apple-silicon/)".

And then build the [exact same image](https://gist.github.com/peterroelants/e344ac416948296f7fcdc84a20ce6eb5) again using [docker buildx](https://docs.docker.com/buildx/working-with-buildx/):
```
docker buildx build --progress=plain --target pydantic_env --tag pydantic-test-env --file ./Dockerfile .
```

I'm curious if this would solve your problem. If not, could you share the results of running the following on your M1 machine:
```
docker run python:3.9.2-slim-buster /bin/bash -c "dpkg --print-architecture && uname -m"
```
@peterroelants I was feeling confident realizing I was still running the older Docker Preview and not the latest release but after installing the latest release and following the directions to install I still get the exact same issue as above, failing in the exact same place. 

The output you requested is:
```
arm64
aarch64
```

That's weird, I can build the same exact same Docker container I posted without any issues. And apparently so can @alexanderluiscampino?

Note from the output `error: command '/usr/bin/gcc' failed with exit code 1` that gcc seems to be failing. I'm running Ubuntu 20.04 on an Intel chipset myself. I tried running the build against an emulated arm64 but I didn't run into any issues.
I'm wondering if the M1 chipset is the cause of your issue.
@peterroelants I can confirm, able to build it with the dockerfile provided, running on windows.
@dotorg-richard I'm curious if a different base image would resolve your issue.

Could you try building the following Alpine-based-image: https://gist.github.com/peterroelants/e344ac416948296f7fcdc84a20ce6eb5#file-python-alpine-dockerfile with:
```
docker buildx build --progress=plain --load --target pydantic_env --tag pydantic-test-env-alpine --file ./python.alpine.Dockerfile .
```
@peterroelants thank you so much I'm happy to report that worked! Hopefully others with the M1 chip can use this alternate base image. Package size report is 6.2MB!


That's great to hear!

Interesting that the Debian based image resulted in issues on the M1, while the Alpine based images work.
I also experimented with plain Ubuntu images and there I got a Pydantic install that was 8 times as large as the ones in the Debian or Alpine Python images.
To clarify a bit, the excessive size is almost entirely due to the debug info, which is usually included by default (see the output of e.g. `python3.9-config --cflags`). So it's the `-g0` that does most of the "work" of the size reduction.

Perhaps given the large size overhead, pydantic could disable debug info by default? The debug info is useful for gdb debugging, crashes, benchmarking etc., but most library users probably won't have a use for it.

If disabling debuginfo entirely is not desirable, a middle ground can be to reduce the debug level from `-g2` (the default) to `-g1`. Citing the gcc manpage:

> Level 1 produces minimal information, enough for making backtraces in parts of the program that you don't plan to debug.  This includes descriptions of functions and external variables, and line number tables, but no information about local variables.

The size comparison is:

```
2.6M	g0/pydantic-1.8.1-cp39-cp39-linux_x86_64.whl
7.8M	g0/pydantic-1.8.1-cp39-cp39-linux_x86_64/

4.7M	g1/pydantic-1.8.1-cp39-cp39-linux_x86_64.whl
14M	g1/pydantic-1.8.1-cp39-cp39-linux_x86_64/

11M	g2/pydantic-1.8.1-cp39-cp39-linux_x86_64.whl
39M	g2/pydantic-1.8.1-cp39-cp39-linux_x86_64/
```
@bluetech How did you test this ([since pydantic overwrites cflags in setup.py](https://github.com/samuelcolvin/pydantic/pull/2517) afaik)?

Btw, depending on the system I'm running on I get different results when building Pydantic:
* Python Buster-Slim:
  - `6.4M	/usr/local/lib/python3.9/site-packages/pydantic`
  -  `python3.9-config --cflags`: `-I/usr/local/include/python3.9 -I/usr/local/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall`
* Python Alpine:
  - `6.7M	/usr/local/lib/python3.9/site-packages/pydantic`
  - `python3.9-config --cflags`: `-I/usr/local/include/python3.9 -I/usr/local/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall`
* Ubuntu hirsute Python from apt:
  - `39M	/usr/local/lib/python3.9/dist-packages/pydantic`
  - `python3.9-config --cflags`: `-I/usr/include/python3.9 -I/usr/include/python3.9  -Wno-unused-result -Wsign-compare -g -ffile-prefix-map=/build/python3.9-y4oXHL/python3.9-3.9.4=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall`
@peterroelants I cloned pydantic, edited the line you linked to, ran `pip wheel .`, extracted the wheel, ran `du -s`. That's as if pydantic itself makes the change.

> Btw, depending on the system I'm running on I get different results when building Pydantic:

That is to be expected, but the differences do not seem too big, the major thing is the debug info.
+1 for adding -g0 to CFLAGS
Bumping this issue - any plans on this?
This will be solved in V2, we're stopping using cython completely.

Instead the core validation logic is written in rust, meaning:
* Runtime performance has increased by roughly 15x
* the pydantic-core wheel is now ~1.5mb on Linux, <1mb on macos and windows
* The pydantic wheel should be a few hundred kb
Although looking at this again, I don't think I had realised that this could be solved by setting a debug flag, sorry.

Maybe we can fix this in v1.10.3 as I know that will be used for a while.
For anyone else looking for a post-install solution (i.e., you are fine with downloading the large binaries and you just want to save space), you can simply run `strip` on the shared libs to remove the debug info. Got me from ~50MB down to ~12MB.
> Although looking at this again, I don't think I had realised that this could be solved by setting a debug flag, sorry.
> 
> Maybe we can fix this in v1.10.3 as I know that will be used for a while.

Just to bump the discussion - we're also waiting (albeit passively) for some news on this issue. 
Should we expect a fix in v1.10.3, and if so, what is the ETA for this release?

Thank you!
happy to accept a fix for this for v1.10.3, otherwise I'll try to get to it.

In terms of ETA for V1.10.3, I had hoped to get it done over the last few weeks but COVID followed by bad cold combined with sick toddler has pretty much wiped me out.

I'll devote what time I have to it over the Christmas "break", but giving a firm deadline seems foolish at this point. Sorry for the delay.
> happy to accept a fix for this for v1.10.3, otherwise I'll try to get to it.
> 
> In terms of ETA for V1.10.3, I had hoped to get it done over the last few weeks but COVID followed by bad cold combined with sick toddler has pretty much wiped me out.
> 
> I'll devote what time I have to it over the Christmas "break", but giving a firm deadline seems foolish at this point. Sorry for the delay.

Sorry to hear that, get well soon!

No worries about the timeline, we'll be happy to just get a notification whenever this issue is referenced in the eventual patch release.
I'm afraid at this point we're too much of surface level users of Pydantic to offer a well thought out fix, so we'll wait for someone more qualified to chime in.

Thank you and take care.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -82,9 +82,9 @@ def extra(self):
         compiler_directives = {}
         if 'CYTHON_TRACE' in sys.argv:
             compiler_directives['linetrace'] = True
-        # Set CFLAG to all optimizations (-O3)
+        # Set CFLAG to all optimizations (-O3), add `-g0` to reduce size of binaries, see #2276
         # Any additional CFLAGS will be appended. Only the last optimization flag will have effect
-        os.environ['CFLAGS'] = '-O3 ' + os.environ.get('CFLAGS', '')
+        os.environ['CFLAGS'] = '-O3 -g0 ' + os.environ.get('CFLAGS', '')
         ext_modules = cythonize(
             'pydantic/*.py',
             exclude=['pydantic/generics.py'],


Test patch: