Title: How do custom __init__ functions work in pydantic with inheritance?
Tags: python,inheritance,init,pydantic

Question: I'm trying to use inheritance in pydantic with custom __init__ functions. I have parent (fish) and child (shark) classes that both require more in initialization than just setting fields (which in the MWE is represented by an additional print statement). So I need to override their inits.
I tried:
class fish(BaseModel):
    name: str
    def __init__(self, name):
        super().__init__(name=name)
        print("Fish initialization successful!")
        

class shark(fish):
    color: str
    def __init__(self, name, color):
        super().__init__(name=name)
        self.color=color
        print("Shark initialization successful!")
        
f = fish(name="nemo")
print(f)
s = shark(name="bruce", color="grey")

but that throws a validation error:
Fish initialization successful!
name='nemo'
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
Cell In[149], line 17
     15 f = fish(name="nemo")
     16 print(f)
---> 17 s = shark(name="bruce", color="grey")

Cell In[149], line 11, in shark.__init__(self, name, color)
     10 def __init__(self, name, color):
---> 11     super().__init__(name=name)
     12     self.color=color
     13     print("Shark initialization successful!")

Cell In[149], line 4, in fish.__init__(self, name)
      3 def __init__(self, name):
----> 4     super().__init__(name=name)
      5     print("Fish initialization successful!")

File ~/Desktop/treeline_wt/1588-yieldmodeling-integration/device-predictions/.venv/lib/python3.9/site-packages/pydantic/main.py:341, in pydantic.main.BaseModel.__init__()

ValidationError: 1 validation error for shark
color
  field required (type=value_error.missing)

The solution I got from a coworker that works is:
class fish(BaseModel):
    name: str
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        print("Fish initialization successful!")
        

class shark(fish):
    color: str
    
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        print("Shark initialization successful!")
        

# f = fish(name="nemo")
# print(f)
s = shark(name="bruce", color="grey")

which, on inspection, only works because the fish super().__init__ receives the color keyword, i.e. changing it to super().__init__(name=kwargs['name']) throws the same validation error. This is baffling to me, I don't understand why the fish class needs to know anything about the properties of its child classes. How do I understand this?

Answer: This has nothing to do with Fish needing to know anything about the fields defined on Shark. It has everything to do with BaseModel.__init__ knowing, which fields any given model has, and validating all keyword-arguments against those.
You need to keep in mind that a lot is happening "behind the scenes" with any model class during class creation, i.e. way before you initialize any specific instance of it. The metaclass is responsible for this.
Essentially, you need to think of the Shark definition process like this:

The Shark class namespace is read.
The annotations/attributes are collected (in this case color: str).
Fields are created from those.
The parent class' fields (in this case name from Fish) are added.
All the fields for Shark (plus validators and a bunch of other things) are fully constructed. It now has the fields name and color.

The BaseModel.__init__ method will always look at all the fields defined on the given model and validate the provided keyword arguments against those fields.
When you call super().__init__(name=name) from inside Shark.__init__, you are basically calling the Fish.__init__(self, name=name), i.e. you are passing the (uninitialized) Shark instance self as well as the name argument to Fish.__init__.
Then from inside Fish.__init__ you are again doing super().__init__(name=name), which means you are calling BaseModel.__init__(self, name=name) and again only passing that unfinished Shark instance and the keyword-argument name to it. (Remember self is still that Shark object.)
But BaseModel.__init__ will look at that Shark instance it got, see that the Shark class has two fields (name and color) defined for it and neither of them is optional/has a default value. It will see that you only provided the name keyword-argument, but failed to provide color. Therefore it will raise a corresponding ValidationError.
The fact that you then manually try to assign self.color = color does not matter because that line is never even reached.
That is why you must always pass all the field-related keyword-arguments "up the chain" to BaseModel.__init__. This is why the code in your second code snippet works without error.
This may seem unintuitive at first, but Pydantic models are not simple data classes. A lot is happening under the hood and this imposes some restrictions, which are unfortunately sometimes left undocumented (as in this case) and only become clear, once you actually dig into the source code.