Managed to make it work!

As Youtube Data API v3 doesn't accept yet tokens generated by JWT aka Service Account
I generated a refresh_token by a normal client OAuth2 flow and use it to execute calls