Question for community: We're considering adding `pydantic` as a base requirement to 🤗  `transformers`
Author: gante
Pull request: no
Labels: name: contributions-welcome

Hi everyone! 👋 

We want to expand our argument validation in `transformers` to improve the library's overall UX. No one wants to store a config object on the Hub with impossible parameterization or go through the code to find the admissible range for a certain input argument.

To that end, we're considering adding `pydantic>=2.0` as a base requirement to 🤗 `transformers`. 

Adding a base requirement is not a decision we want to make lightly -- it may place unwanted constraints on downstream projects. We can't anticipate all issues, so we're raising this issue to proactively find them. If `>2.0` is a general issue, we can do a try/except block with `import pydantic.v1 as pydantic`, and use `1.x` syntax.

Let us know your opinion about `pydantic`!

(Related PR: #35910)

Comments:
고맙습니다

Extra information:
- `pydantic` seems to be the most widely used annotator. Here's a graph with the GH star evolution of related libraries
<img width="500" alt="Image" src="https://github.com/user-attachments/assets/414039ff-3389-4d23-9854-2a28f33d25dd" />

- If we move forward, we'll be aiming at `pydantic>=2.0`. If we do so, it means we will not be compatible with packages that use `pydantic<2.0` -- there were breaking changes from `1.x`. If we find popular projects or a reasonable number of projects that are not planing to update from `1.x`, we can use a try/except block with `import pydantic.v1 as pydantic`. However, `1.x` is known to be much slower, and thus we would need to benchmark speed. 

After some search -- using `pydantic` v2 syntax would NOT be okay.

We can find a few libraries that have both `transformers` and `pydantic 1.x` in their requirements:
1. `transformers` + `pydantic<2.0` ([link](https://sourcegraph.com/search?q=context%3Aglobal+pydantic%3C2.0+AND+transformers&patternType=keyword&sm=0&df=%5B%22lang%22%2C%22Pip+Requirements%22%2C%22lang%3A%5C%22pip+requirements%5C%22%22%5D&__cc=1)). Notable examples with unpinned `transformers` version:
  a.  `microsoft/responsible-ai-toolbox`, 1.5k stars
  b. `ludwig-ai/ludwig`, 11.3k stars
2. `transformers` + `pydantic==1` ([link](https://sourcegraph.com/search?q=context%3Aglobal+pydantic%3D%3D1+AND+transformers&patternType=keyword&sm=0&df=%5B%22lang%22%2C%22Pip+Requirements%22%2C%22lang%3A%5C%22pip+requirements%5C%22%22%5D&__cc=1)). More than 1500 pip dependency matches (most of them with pinned `transformers` version)! Notable examples with unpinned `transformers` version:
  a. `wenge-research/YAYI`, 3.3k stars
  b. `THUDM/CogVideo`, 10.7k stars

If you can get it ready for v5, then such a change would not be too bad I think. Other libraries should pin <5 then.

I am not sure I completely understand how you would use it though. Could you give a small dummy code example? 

I think we would need to be super super cautious & conservative about this.

BTW ping @Wauplin and @LysandreJik given we decided against adding `pydantic` as a dependency in `huggingface_hub` (which in turn is a dependency of transformers)

worst case, can't we vendor a subset of pydantic implementation/code instead?

We can support both `Pydantic` v1 and v2 so that any Transformers dependent library that pins to either version will still work and pip can figure out the right version to use.

Since we expect the ecosystem to move entirely to v2, we want it to be easy to drop v1 in the future, to do that we can use the `Pydantic V2` namespaces internally, so if v2 is installed we call the native v2 methods directly and if v1 is installed we call their v1 equivalents under the hood.

We also don’t need to map every feature in Pydantic, we mostly need the validators, the base model configuration, Fields, and serialiazation methods.

Hi everyone 👋 

We've heard your feedback (here and in other social media), and it became clear that `pydantic` would be problematic for some projects, and would give ourselves a few headaches -- we would have to start with v1, for overall compatibility, which would mean we would have to migrate to v2 at some point in the future.

Instead, together with `huggingface_hub` (which is a key dependency of `transformers`), we will create shared basic functionality to validate classes that mostly contain data, using `dataclasses` (part of the standard library) and their `__post_init__` method. 

This means we will have most benefits (data validation) with no additional dependencies.

Stay tuned 🤗 

I opened https://github.com/huggingface/huggingface_hub/pull/2895 as a **suggestion** on how to move forward with data validation in the HF ecosystem. Open to feedback and subject to change based on community requirements 🤗