Package: json2python-models
json2python-models is aPythontool that can generate Python models classes
(pydantic, (sqlmodel,
dataclasses,attrs)
from JSON datasets.FeaturesFulltypingmodulesupportTypes merging- if some field contains data of different types this will be represented asUniontypeFields and modelsnamesgeneration (unicode support included)Similarmodels generalizationHandlingrecursive datastructures (i.e. family tree)Detectingstring serializable types(i.e. datetime or just stringify numbers)Detecting fields containing string constants (Literal['foo', 'bar'])Generation models aslist(flat models structure) ortree(nested models)Specifying when dictionaries should be processed asdicttype(by default every dict is considered as some model)CLIAPI with a lot of optionsTable of ContentsFeaturesTable of ContentsExampleInstallationUsageCLILow level APITestsTest examplesBuilt WithContributingLicenseExamplesPart of Path of Exile public items APIfrompydanticimportBaseModel,FieldfromtypingimportList,Optionalfromtyping_extensionsimportLiteralclassTab(BaseModel):id_:str=Field(...,alias="id")public:boolstash_type:Literal["CurrencyStash","NormalStash","PremiumStash"]=Field(...,alias="stashType")items:List['Item']account_name:Optional[str]=Field(None,alias="accountName")last_character_name:Optional[str]=Field(None,alias="lastCharacterName")stash:Optional[str]=Noneleague:Optional[Literal["Hardcore","Standard"]]=NoneF1 Season Results----- Show -----driver_standings.json
[
    {
        "season": "2019",
        "round": "3",
        "DriverStandings": [
            {
                "position": "1",
                "positionText": "1",
                "points": "68",
                "wins": "2",
                "Driver": {
                    "driverId": "hamilton",
                    "permanentNumber": "44",
                    "code": "HAM",
                    "url": "http://en.wikipedia.org/wiki/Lewis_Hamilton",
                    "givenName": "Lewis",
                    "familyName": "Hamilton",
                    "dateOfBirth": "1985-01-07",
                    "nationality": "British"
                },
                "Constructors": [
                    {
                        "constructorId": "mercedes",
                        "url": "http://en.wikipedia.org/wiki/Mercedes-Benz_in_Formula_One",
                        "name": "Mercedes",
                        "nationality": "German"
                    }
                ]
            },
            ...
        ]
    }
]json2models -f pydantic -m DriverStandings driver_standings.jsonr"""generated by json2python-models v0.2.0 at Mon May  4 17:46:30 2020command: /opt/projects/json2python-models/venv/bin/json2models -f pydantic -s flat -m DriverStandings driver_standings.json"""frompydanticimportBaseModel,FieldfromtypingimportListfromtyping_extensionsimportLiteralclassDriverStandings(BaseModel):season:intround_:int=Field(...,alias="round")DriverStandings:List['DriverStanding']classDriverStanding(BaseModel):position:intposition_text:int=Field(...,alias="positionText")points:intwins:intdriver:'Driver'=Field(...,alias="Driver")constructors:List['Constructor']=Field(...,alias="Constructors")classDriver(BaseModel):driver_id:str=Field(...,alias="driverId")permanent_number:int=Field(...,alias="permanentNumber")code:strurl:strgiven_name:str=Field(...,alias="givenName")family_name:str=Field(...,alias="familyName")date_of_birth:str=Field(...,alias="dateOfBirth")nationality:strclassConstructor(BaseModel):constructor_id:str=Field(...,alias="constructorId")url:strname:strnationality:Literal["Austrian","German","American","British","Italian","French"]Swagger----- Show -----swagger.jsonfrom any online API (I tested file generated by drf-yasg and another one for Spotify API)It requires a bit of tweaking:Some fields store routes/models specs as dictsThere are a lot of optional fields, so we reduce merging thresholdDisable string literalsjson2models -f dataclasses -m Swagger testing_tools/swagger.json \
    --dict-keys-fields securityDefinitions paths responses definitions properties \
    --merge percent_50 number --max-strings-literals 0r"""generated by json2python-models v0.2.0 at Mon May  4 18:08:09 2020command: /opt/projects/json2python-models/json_to_models/__main__.py -s flat -f dataclasses -m Swagger testing_tools/swagger.json --max-strings-literals 0 --dict-keys-fields securityDefinitions paths responses definitions properties --merge percent_50 number"""fromdataclassesimportdataclass,fieldfromjson_to_models.dynamic_typingimportFloatStringfromtypingimportAny,Dict,List,Optional,Union@dataclassclassSwagger:swagger:FloatStringinfo:'Info'host:strschemes:List[str]base_path:strconsumes:List[str]produces:List[str]security_definitions:Dict[str,'Parameter_SecurityDefinition']security:List['Security']paths:Dict[str,'Path']definitions:Dict[str,'Definition_Schema']@dataclassclassInfo:title:strdescription:strversion:str@dataclassclassSecurity:api_key:Optional[List[Any]]=field(default_factory=list)basic:Optional[List[Any]]=field(default_factory=list)@dataclassclassPath:parameters:List['Parameter_SecurityDefinition']post:Optional['Delete_Get_Patch_Post_Put']=Noneget:Optional['Delete_Get_Patch_Post_Put']=Noneput:Optional['Delete_Get_Patch_Post_Put']=Nonepatch:Optional['Delete_Get_Patch_Post_Put']=Nonedelete:Optional['Delete_Get_Patch_Post_Put']=None@dataclassclassProperty:type_:strformat_:Optional[str]=Nonexnullable:Optional[bool]=Noneitems:Optional['Item_Schema']=None@dataclassclassProperty_2E:type_:strtitle:Optional[str]=Noneread_only:Optional[bool]=Nonemax_length:Optional[int]=Nonemin_length:Optional[int]=Noneitems:Optional['Item']=Noneenum:Optional[List[str]]=field(default_factory=list)maximum:Optional[int]=Noneminimum:Optional[int]=Noneformat_:Optional[str]=None@dataclassclassItem:title:Optional[str]=Nonetype_:Optional[str]=Noneref:Optional[str]=Nonemax_length:Optional[int]=Nonemin_length:Optional[int]=None@dataclassclassParameter_SecurityDefinition:name:Optional[str]=Nonein_:Optional[str]=Nonerequired:Optional[bool]=Noneschema:Optional['Item_Schema']=Nonedescription:Optional[str]=Nonetype_:Optional[str]=None@dataclassclassDelete_Get_Patch_Post_Put:operation_id:strdescription:strparameters:List['Parameter_SecurityDefinition']responses:Dict[str,'Response']tags:List[str]@dataclassclassItem_Schema:ref:str@dataclassclassResponse:description:strschema:Optional[Union['Item_Schema','Definition_Schema']]=None@dataclassclassDefinition_Schema:type_:strrequired:Optional[List[str]]=field(default_factory=list)properties:Optional[Dict[str,Union['Property','Property_2E']]]=field(default_factory=dict)ref:Optional[str]=NoneGithub-actions config files----- Show -----Github-actions model based on files fromstarter-workflowsjson2models -m Actions "./starter-workflows/ci/*.yml" -s flat -f pydantic -i yaml --dkf env with jobsr"""generated by json2python-models v0.2.3 at Tue Jul 13 19:52:43 2021command: /opt/projects/json2python-models/venv/bin/json2models -m Actions ./starter-workflows/ci/*.yml -s flat -f pydantic -i yaml --dkf env with jobs"""frompydanticimportBaseModel,FieldfromtypingimportDict,List,Optional,Unionfromtyping_extensionsimportLiteralclassActions(BaseModel):on:Union['On',List[Literal["push"]]]jobs:Dict[str,'Job']name:Optional[str]=Noneenv:Optional[Dict[str,Union[int,str]]]={}classOn(BaseModel):push:Optional['Push']=Nonepull_request:Optional['PullRequest']=Nonerelease:Optional['Release']=Noneschedule:Optional[List['Schedule']]=[]workflow_dispatch:Optional[None]=NoneclassPush(BaseModel):branches:List[Literal["$default-branch"]]tags:Optional[List[Literal["v*.*.*"]]]=[]classPullRequest(BaseModel):branches:List[Literal["$default-branch"]]classRelease(BaseModel):types:List[Literal["created","published"]]classSchedule(BaseModel):cron:Literal["$cron-daily"]classJob(BaseModel):runson:Literal["${{ matrix.os }}","macOS-latest","macos-latest","ubuntu-18.04","ubuntu-latest","windows-latest"]=Field(...,alias="runs-on")steps:List['Step']name:Optional[str]=Noneenvironment:Optional[Literal["production"]]=Noneoutputs:Optional['Output']=Nonecontainer:Optional['Container']=Noneneeds:Optional[Literal["build"]]=Nonepermissions:Optional['Permission']=Nonestrategy:Optional['Strategy']=Nonedefaults:Optional['Default']=Noneenv:Optional[Dict[str,str]]={}classStep(BaseModel):uses:Optional[str]=Nonename:Optional[str]=Nonewith_:Optional[Dict[str,Union[bool,float,str]]]=Field({},alias="with")run:Optional[str]=Noneenv:Optional[Dict[str,str]]={}workingdirectory:Optional[str]=Field(None,alias="working-directory")id_:Optional[Literal["build-image","composer-cache","deploy-and-expose","image-build","login-ecr","meta","push-to-registry","task-def"]]=Field(None,alias="id")if_:Optional[str]=Field(None,alias="if")shell:Optional[Literal["Rscript{0}"]]=NoneclassOutput(BaseModel):route:str=Field(...,alias="ROUTE")selector:str=Field(...,alias="SELECTOR")classContainer(BaseModel):image:Literal["crystallang/crystal","erlang:22.0.7"]classPermission(BaseModel):contents:Literal["read"]packages:Literal["write"]classStrategy(BaseModel):matrix:Optional['Matrix']=Nonemaxparallel:Optional[int]=Field(None,alias="max-parallel")failfast:Optional[bool]=Field(None,alias="fail-fast")classMatrix(BaseModel):rversion:Optional[List[float]]=Field([],alias="r-version")pythonversion:Optional[List[float]]=Field([],alias="python-version")deno:Optional[List[Literal["canary","v1.x"]]]=[]os:Optional[List[Literal["macOS-latest","ubuntu-latest","windows-latest"]]]=[]rubyversion:Optional[List[float]]=Field([],alias="ruby-version")nodeversion:Optional[List[Literal["12.x","14.x","16.x"]]]=Field([],alias="node-version")configuration:Optional[List[Literal["Debug","Release"]]]=[]classDefault(BaseModel):run:'Run'classRun(BaseModel):shell:Literal["bash"]Example with preamble----- Show -----A simple example to demonstrate adding extra code before the class list.json2models-fpydantic--preamble"# set up defaultsUSERNAME = 'user'SERVER_IP = '127.0.0.1'"-mSwaggertesting_tools/swagger.jsonr"""generated by json2python-models v0.2.5 at Tue Aug 23 08:55:09 2022command: json2models -f pydantic --preamble # set up defaultsUSERNAME = 'user'SERVER_IP = '127.0.0.1'-m Swagger testing_tools/swagger.json -o output.py"""frompydanticimportBaseModel,FieldfromtypingimportAny,List,Literal,Optional,Union# set up defaultsUSERNAME='user'SERVER_IP='127.0.0.1'classSwagger(BaseModel):# etc.InstallationBeware: this project supports onlypython3.7and higher.To install it, usepip:pip install json2python-modelsOr you can build it from source:git clone https://github.com/bogdandm/json2python-models.git
cd json2python-models
python setup.py installUsageCLIFor regular usage CLI tool is the best option. After you install this package you can use it asjson2models <arguments>orpython -m json_to_models <arguments>. I.e.:json2models -m Car car_*.json -f attrs > car.pyArguments:-h,--help- Show help message and exit-m,--model- Model name and its JSON data as path or unix-like path pattern.*,**or?patterns symbols are supported.
JSON data could be an array of models or single model.
If this file contains dict with nested list than you can pass<JSON lookup>. Deep lookups are supported by dot-separated path.
If no lookup needed pass '-' as<JSON lookup>(default)Format:-m <Model name> [<JSON lookup>] <File path or pattern>Example:-m Car audi.json -m Car results reno.json-i,--input-format- Input file format (parser). Default is JSON parser. Yaml parser requires PyYaml or
ruamel.yaml to be installed. Ini parser uses
builtinconfigparser. To implement new one - add new method
tocli.FileLoaders(and create pull request :) )Format:-i {json, yaml, ini}Example:-i yamlDefault:-i json-o,--output- Output fileFormat:-o <FILE>Example:-o car_model.py-f,--framework- Model framework for which python code is generated.base(default) mean no framework so code will be generated without any decorators and additional meta-data.Format:-f {base, pydantic, sqlmodel, attrs, dataclasses, custom}Example:-f pydanticDefault:-f baseWarning: SQLModel generator does not support Relationships and Foreign keys, they have to be added manually-s,--structure- Models composition style.Format:-s {flat, nested}Example:-s nestedDefault:-s flat--preamble- Additional material to beFormat:--preamble "<formatted python code string to be added after module imports>"Example:--preamble"# set up defaultsUSERNAME = 'user'SERVER = '127.0.0.1'"Optional--datetime- Enable datetime/date/time strings parsing.Default: disabledWarning: This can lead to 6-7 times slowdown on large datasets. Be sure that you really need this option.--disable-unicode-conversion,--no-unidecode- Disable unicode conversion in field labels and class namesDefault: enabled--strings-converters- Enable generation of string types converters (i.e.IsoDatetimeStringorBooleanString).Default: disabled--max-strings-literals-  GenerateLiteral['foo', 'bar']when field have less than NUMBER string constants as values.Format:--max-strings-literals <NUMBER>Default: 10 (generator classes could override it)Example:--max-strings-literals 5- only 5 literals will be saved and used to code generationNote: There could not be more than15literals per field (for performance reasons)Note:attrscode generator do not use Literals and just generatestrfields instead--merge- Merge policy settings. Possible values are:Format:--merge MERGE_POLICY [MERGE_POLICY ...]Possible values(MERGE_POLICY):percent[_<percent>]- two models had a certain percentage of matched field names.
Custom value could be i.e.percent_95.number[_<number>]- two models had a certain number of matched field names.exact- two models should have exact same field names to merge.Example:--merge percent_95 number_20- merge if 95% of fields are matched or 20 of fields are matchedDefault:--merge percent_70 number_10--dict-keys-regex,--dkr- List of regular expressions (Python syntax).
If all keys of some dict are match one of the pattern then
this dict will be marked as dict field but not nested model.Format:--dkr RegEx [RegEx ...]Example:--dkr node_\d+ \d+_\d+_\d+Note:^and$(string borders) tokens will be added automatically, but you
have to escape other special characters manually.Optional--dict-keys-fields,--dkf- List of model fields names that will be marked as dict fieldsFormat:--dkf FIELD_NAME [FIELD_NAME ...]Example:--dkf "dict_data" "mapping"Optional--code-generator- Absolute import path toGenericModelCodeGeneratorsubclass.Format:--code-generator CODE_GENERATORExample:-f mypackage.mymodule.DjangoModelsGeneratorNote: Is ignored without-f custombut is required with it.--code-generator-kwargs- List of GenericModelCodeGenerator subclass arguments (for__init__method,
see docs of specific subclass).
Each argument should be in following format:argument_name=valueor"argument_name=value with space".
Boolean values should be passed in JS style:trueorfalseFormat:--code-generator-kwargs [NAME=VALUE [NAME=VALUE ...]]Example:--code-generator-kwargs kwarg1=true kwarg2=10 "kwarg3=It is string with spaces"Optional--disable-str-serializable-types- List of python types for which StringSerializable should be disabled.
Alternatively you could use the name of StringSerializable subclass itself (i.e. IntString).Format:--disable-str-serializable-types [TYPE [TYPE ...]]Example:--disable-str-serializable-types float int BooleanString IsoDatetimeStringOptionalLow level API-TestsTo run tests you should clone project and runsetup.pyscript:git clone https://github.com/bogdandm/json2python-models.git
cd json2python-models
python setup.py test -a '<pytest additional arguments>'Also, I would recommend you to installpytest-sugarfor pretty printing test resultsTest examplesYou can find out some examples of usage of this project attesting_tools/real_apis/...Each file contains functions to download data from some online API (references included at the top of file) andmainfunction that generates and prints code. Some examples may print debug data before actual code.
Downloaded data will be saved attesting_tools/real_apis/<name of example>/<dataset>.jsonBuilt Withpython-dateutil- Datetime parsinginflection- String transformationsUnidecode- Unicode to ASCII conversionJinja2- Code templatesordered-setis used in models merging algorithmTest tools:pytest- Test frameworkpytest-xdist- Parallel execution of test suitespytest-sugar- Test results pretty printingrequests- Test data downloadContributingFeel free to open pull requests with new features or bug fixes. Just follow few rules:Always use some code formatter (blackor PyCharm built-in)Keep code coverage above 95-98%All existing tests should be passed (including test examples fromtesting_tools/real_apis)UsetypingmoduleFixcodacyissues from your PRLicenseThis project is licensed under the MIT License - see theLICENSEfile for details