Package: coqpit
👩‍✈️ CoqpitSimple, light-weight and no dependency config handling through python data classes with to/from JSON serialization/deserialization.Currently it is being used by🐸TTS.❔ Why I need thisWhat I need from a ML configuration library...Fixing a general config schema in Python to guide users about expected values.Python is good but not universal. Sometimes you train a ML model and use it on a different platform. So, you
need your model configuration file importable by other programming languages.Simple dynamic value and type checking with default values.If you are a beginner in a ML project, it is hard to guess the right values for your ML experiment. Therefore it is important
to have some default values and know what range and type of input are expected for each field.Ability to decompose large configs.As you define more fields for the training dataset, data preprocessing, model parameters, etc., your config file tends
to get quite large but in most cases, they can be decomposed, enabling flexibility and readability.Inheritance and nested configurations.Simply helps to keep configurations consistent and easier to maintain.Ability to override values from the command line when necessary.For instance, you might need to define a path for your dataset, and this changes for almost every run. Then the user
should be able to override this value easily over the command line.It also allows easy hyper-parameter search without changing your original code. Basically, you can run different models
with different parameters just using command line arguments.Defining dynamic or conditional config values.Sometimes you need to define certain values depending on the other values. Using python helps to define the underlying
logic for such config values.No dependenciesYou don't want to install a ton of libraries for just configuration management. If you install one, then it
is better to be just native python.🚫 LimitationsUniontype dataclass fields cannot be parsed from console arguments due to the type ambiguity.JSONis the only supported serialization format, although the others can be easily integrated.Listtype with multiple item type annotations are not supported. (e.g.List[int, str]).dictfields are parsed from console arguments as JSON str without type checking. (e.g--val_dict '{"a":10, "b":100}').MISSINGfields cannot be avoided when parsing console arguments.🔍 Examples👉 Simple Coqpitimportosfromdataclassesimportasdict,dataclass,fieldfromtypingimportList,UnionfromcoqpitimportMISSING,Coqpit,check_argument@dataclassclassSimpleConfig(Coqpit):val_a:int=10val_b:int=Noneval_d:float=10.21val_c:str="Coqpit is great!"# mandatory field# raise an error when accessing the value if it is not changed. It is a way to defineval_k:int=MISSING# optional fieldval_dict:dict=field(default_factory=lambda:{"val_aa":10,"val_ss":"This is in a dict."})# list of listval_listoflist:List[List]=field(default_factory=lambda:[[1,2],[3,4]])val_listofunion:List[List[Union[str,int]]]=field(default_factory=lambda:[[1,3],[1,"Hi!"]])defcheck_values(self,):# you can define explicit constraints on the fields using `check_argument()`"""Check config fields"""c=asdict(self)check_argument("val_a",c,restricted=True,min_val=10,max_val=2056)check_argument("val_b",c,restricted=True,min_val=128,max_val=4058,allow_none=True)check_argument("val_c",c,restricted=True)if__name__=="__main__":file_path=os.path.dirname(os.path.abspath(__file__))config=SimpleConfig()# try MISSING class argumenttry:k=config.val_kexceptAttributeError:print(" val_k needs a different value before accessing it.")config.val_k=1000# try serialization and deserializationprint(config.serialize())print(config.to_json())config.save_json(os.path.join(file_path,"example_config.json"))config.load_json(os.path.join(file_path,"example_config.json"))print(config.pprint())# try `dict` interfaceprint(*config)print(dict(**config))# value assignment by mappingconfig["val_a"]=-999print(config["val_a"])assertconfig.val_a==-999👉 Serializationimportosfromdataclassesimportasdict,dataclass,fieldfromcoqpitimportCoqpit,check_argumentfromtypingimportList,Union@dataclassclassSimpleConfig(Coqpit):val_a:int=10val_b:int=Noneval_c:str="Coqpit is great!"defcheck_values(self,):'''Check config fields'''c=asdict(self)check_argument('val_a',c,restricted=True,min_val=10,max_val=2056)check_argument('val_b',c,restricted=True,min_val=128,max_val=4058,allow_none=True)check_argument('val_c',c,restricted=True)@dataclassclassNestedConfig(Coqpit):val_d:int=10val_e:int=Noneval_f:str="Coqpit is great!"sc_list:List[SimpleConfig]=Nonesc:SimpleConfig=SimpleConfig()union_var:Union[List[SimpleConfig],SimpleConfig]=field(default_factory=lambda:[SimpleConfig(),SimpleConfig()])defcheck_values(self,):'''Check config fields'''c=asdict(self)check_argument('val_d',c,restricted=True,min_val=10,max_val=2056)check_argument('val_e',c,restricted=True,min_val=128,max_val=4058,allow_none=True)check_argument('val_f',c,restricted=True)check_argument('sc_list',c,restricted=True,allow_none=True)check_argument('sc',c,restricted=True,allow_none=True)if__name__=='__main__':file_path=os.path.dirname(os.path.abspath(__file__))# init 🐸 dataclassconfig=NestedConfig()# save to a json fileconfig.save_json(os.path.join(file_path,'example_config.json'))# load a json fileconfig2=NestedConfig(val_d=None,val_e=500,val_f=None,sc_list=None,sc=None,union_var=None)# update the config with the json file.config2.load_json(os.path.join(file_path,'example_config.json'))# now they should be having the same values.assertconfig==config2# pretty print the dataclassprint(config.pprint())# export values to a dictconfig_dict=config.to_dict()# crate a new config with different values than the defaultsconfig2=NestedConfig(val_d=None,val_e=500,val_f=None,sc_list=None,sc=None,union_var=None)# update the config with the exported valuess from the previous config.config2.from_dict(config_dict)# now they should be having the same values.assertconfig==config2👉argparsehandling and parsing.importargparseimportosfromdataclassesimportasdict,dataclass,fieldfromtypingimportListfromcoqpitimportCoqpit,check_argumentimportsys@dataclassclassSimplerConfig(Coqpit):val_a:int=field(default=None,metadata={'help':'this is val_a'})@dataclassclassSimpleConfig(Coqpit):val_req:str# required fieldval_a:int=field(default=10,metadata={'help':'this is val_a of SimpleConfig'})val_b:int=field(default=None,metadata={'help':'this is val_b'})nested_config:SimplerConfig=SimplerConfig()mylist_with_default:List[SimplerConfig]=field(default_factory=lambda:[SimplerConfig(val_a=100),SimplerConfig(val_a=999)],metadata={'help':'list of SimplerConfig'})# mylist_without_default: List[SimplerConfig] = field(default=None, metadata={'help': 'list of SimplerConfig'})  # NOT SUPPORTED YET!defcheck_values(self,):'''Check config fields'''c=asdict(self)check_argument('val_a',c,restricted=True,min_val=10,max_val=2056)check_argument('val_b',c,restricted=True,min_val=128,max_val=4058,allow_none=True)check_argument('val_req',c,restricted=True)defmain():# reference config that we like to match with the one parsed from argparseconfig_ref=SimpleConfig(val_req='this is different',val_a=222,val_b=999,nested_config=SimplerConfig(val_a=333),mylist_with_default=[SimplerConfig(val_a=222),SimplerConfig(val_a=111)])# create new config object from CLI inputsparsed=SimpleConfig.init_from_argparse()parsed.pprint()# check the parsed config with the reference configassertparsed==config_refif__name__=='__main__':sys.argv.extend(['--coqpit.val_req','this is different'])sys.argv.extend(['--coqpit.val_a','222'])sys.argv.extend(['--coqpit.val_b','999'])sys.argv.extend(['--coqpit.nested_config.val_a','333'])sys.argv.extend(['--coqpit.mylist_with_default.0.val_a','222'])sys.argv.extend(['--coqpit.mylist_with_default.1.val_a','111'])main()🤸‍♀️ Merging coqpitsimportosfromdataclassesimportdataclassfromcoqpitimportCoqpit,check_argument@dataclassclassCoqpitA(Coqpit):val_a:int=10val_b:int=Noneval_d:float=10.21val_c:str="Coqpit is great!"@dataclassclassCoqpitB(Coqpit):val_d:int=25val_e:int=257val_f:float=-10.21val_g:str="Coqpit is really great!"if__name__=='__main__':file_path=os.path.dirname(os.path.abspath(__file__))coqpita=CoqpitA()coqpitb=CoqpitB()coqpitb.merge(coqpita)print(coqpitb.val_a)print(coqpitb.pprint())DevelopmentInstall the pre-commit hook to automatically check your commits for style and hinting issues:$python.pre-commit-2.12.1.pyzinstall