Package: ml-collections
ML CollectionsML Collections is a library of Python Collections designed for ML use cases.ConfigDictThe two classes calledConfigDictandFrozenConfigDictare "dict-like" data
structures with dot access to nested elements. Together, they are supposed to be
used as a main way of expressing configurations of experiments and models.This document describes example usage ofConfigDict,FrozenConfigDict,FieldReference.FeaturesDot-based access to fields.Locking mechanism to prevent spelling mistakes.Lazy computation.FrozenConfigDict() class which is immutable and hashable.Type safety."Did you mean" functionality.Human readable printing (with valid references and cycles), using valid YAML
format.Fields can be passed as keyword arguments using the**operator.There are two exceptions to the strong type-safety of the ConfigDict.intvalues can be passed in to fields of typefloat. In such a case, the value
is type-converted to afloatbefore being stored. Similarly, all string
types (including Unicode strings) can be stored in fields of typestrorunicode.Basic Usageimportml_collectionscfg=ml_collections.ConfigDict()cfg.float_field=12.6cfg.integer_field=123cfg.another_integer_field=234cfg.nested=ml_collections.ConfigDict()cfg.nested.string_field='tom'print(cfg.integer_field)# Prints 123.print(cfg['integer_field'])# Prints 123 as well.try:cfg.integer_field='tom'# Raises TypeError as this field is an integer.exceptTypeErrorase:print(e)cfg.float_field=12# Works: `Int` types can be assigned to `Float`.cfg.nested.string_field=u'bob'# `String` fields can store Unicode strings.print(cfg)FrozenConfigDictAFrozenConfigDictis an immutable, hashable type ofConfigDict:importml_collectionsinitial_dictionary={'int':1,'list':[1,2],'tuple':(1,2,3),'set':{1,2,3,4},'dict_tuple_list':{'tuple_list':([1,2],3)}}cfg=ml_collections.ConfigDict(initial_dictionary)frozen_dict=ml_collections.FrozenConfigDict(initial_dictionary)print(frozen_dict.tuple)# Prints tuple (1, 2, 3)print(frozen_dict.list)# Prints tuple (1, 2)print(frozen_dict.set)# Prints frozenset {1, 2, 3, 4}print(frozen_dict.dict_tuple_list.tuple_list[0])# Prints tuple (1, 2)frozen_cfg=ml_collections.FrozenConfigDict(cfg)print(frozen_cfg==frozen_dict)# Trueprint(hash(frozen_cfg)==hash(frozen_dict))# Truetry:frozen_dict.int=2# Raises TypeError as FrozenConfigDict is immutable.exceptAttributeErrorase:print(e)# Converting between `FrozenConfigDict` and `ConfigDict`:thawed_frozen_cfg=ml_collections.ConfigDict(frozen_dict)print(thawed_frozen_cfg==cfg)# Truefrozen_cfg_to_cfg=frozen_dict.as_configdict()print(frozen_cfg_to_cfg==cfg)# TrueFieldReferences and placeholdersAFieldReferenceis useful for having multiple fields use the same value. It
can also be used forlazy computation.You can useplaceholder()as a shortcut to create aFieldReference(field)
with aNonedefault value. This is useful if a program uses optional
configuration fields.importml_collectionsfromml_collections.config_dictimportconfig_dictplaceholder=ml_collections.FieldReference(0)cfg=ml_collections.ConfigDict()cfg.placeholder=placeholdercfg.optional=config_dict.placeholder(int)cfg.nested=ml_collections.ConfigDict()cfg.nested.placeholder=placeholdertry:cfg.optional='tom'# Raises Type error as this field is an integer.exceptTypeErrorase:print(e)cfg.optional=1555# Works fine.cfg.placeholder=1# Changes the value of both placeholder and# nested.placeholder fields.print(cfg)Note that the indirection provided byFieldReferences will be lost if accessed
through aConfigDict.importml_collectionsplaceholder=ml_collections.FieldReference(0)cfg.field1=placeholdercfg.field2=placeholder# This field will be tied to cfg.field1.cfg.field3=cfg.field1# This will just be an int field initialized to 0.Lazy computationUsing aFieldReferencein a standard operation (addition, subtraction,
multiplication, etc...) will return anotherFieldReferencethat points to the
original's value. You can useFieldReference.get()to execute the operations
and get the reference's computed value, andFieldReference.set()to change the
original reference's value.importml_collectionsref=ml_collections.FieldReference(1)print(ref.get())# Prints 1add_ten=ref.get()+10# ref.get() is an integer and so is add_tenadd_ten_lazy=ref+10# add_ten_lazy is a FieldReference - NOT an integerprint(add_ten)# Prints 11print(add_ten_lazy.get())# Prints 11 because ref's value is 1# Addition is lazily computed for FieldReferences so changing ref will change# the value that is used to compute add_ten.ref.set(5)print(add_ten)# Prints 11print(add_ten_lazy.get())# Prints 15 because ref's value is 5If aFieldReferencehasNoneas its original value, or any operation has an
argument ofNone, then the lazy computation will evaluate toNone.We can also use fields in aConfigDictin lazy computation. In this case a
field will only be lazily evaluated ifConfigDict.get_ref()is used to get it.importml_collectionsconfig=ml_collections.ConfigDict()config.reference_field=ml_collections.FieldReference(1)config.integer_field=2config.float_field=2.5# No lazy evaluatuations because we didn't use get_ref()config.no_lazy=config.integer_field*config.float_field# This will lazily evaluate ONLY config.integer_fieldconfig.lazy_integer=config.get_ref('integer_field')*config.float_field# This will lazily evaluate ONLY config.float_fieldconfig.lazy_float=config.integer_field*config.get_ref('float_field')# This will lazily evaluate BOTH config.integer_field and config.float_Fieldconfig.lazy_both=(config.get_ref('integer_field')*config.get_ref('float_field'))config.integer_field=3print(config.no_lazy)# Prints 5.0 - It uses integer_field's original valueprint(config.lazy_integer)# Prints 7.5config.float_field=3.5print(config.lazy_float)# Prints 7.0print(config.lazy_both)# Prints 10.5Changing lazily computed valuesLazily computed values in a ConfigDict can be overridden in the same way as
regular values. The reference to theFieldReferenceused for the lazy
computation will be lost and all computations downstream in the reference graph
will use the new value.importml_collectionsconfig=ml_collections.ConfigDict()config.reference=1config.reference_0=config.get_ref('reference')+10config.reference_1=config.get_ref('reference')+20config.reference_1_0=config.get_ref('reference_1')+100print(config.reference)# Prints 1.print(config.reference_0)# Prints 11.print(config.reference_1)# Prints 21.print(config.reference_1_0)# Prints 121.config.reference_1=30print(config.reference)# Prints 1 (unchanged).print(config.reference_0)# Prints 11 (unchanged).print(config.reference_1)# Prints 30.print(config.reference_1_0)# Prints 130.CyclesYou cannot create cycles using references. Fortunatelythe only wayto create a cycle is by
assigning a computed field to one thatis notthe result of computation. This
is forbidden:importml_collectionsfromml_collections.config_dictimportconfig_dictconfig=ml_collections.ConfigDict()config.integer_field=1config.bigger_integer_field=config.get_ref('integer_field')+10try:# Raises a MutabilityError because setting config.integer_field would# cause a cycle.config.integer_field=config.get_ref('bigger_integer_field')+2exceptconfig_dict.MutabilityErrorase:print(e)Advanced usageHere are some more advanced examples showing lazy computation with different
operators and data types.importml_collectionsconfig=ml_collections.ConfigDict()config.float_field=12.6config.integer_field=123config.list_field=[0,1,2]config.float_multiply_field=config.get_ref('float_field')*3print(config.float_multiply_field)# Prints 37.8config.float_field=10.0print(config.float_multiply_field)# Prints 30.0config.longer_list_field=config.get_ref('list_field')+[3,4,5]print(config.longer_list_field)# Prints [0, 1, 2, 3, 4, 5]config.list_field=[-1]print(config.longer_list_field)# Prints [-1, 3, 4, 5]# Both operands can be referencesconfig.ref_subtraction=(config.get_ref('float_field')-config.get_ref('integer_field'))print(config.ref_subtraction)# Prints -113.0config.integer_field=10print(config.ref_subtraction)# Prints 0.0Equality checkingYou can use==and.eq_as_configdict()to check equality amongConfigDictandFrozenConfigDictobjects.importml_collectionsdict_1={'list':[1,2]}dict_2={'list':(1,2)}cfg_1=ml_collections.ConfigDict(dict_1)frozen_cfg_1=ml_collections.FrozenConfigDict(dict_1)frozen_cfg_2=ml_collections.FrozenConfigDict(dict_2)# True because FrozenConfigDict converts lists to tuplesprint(frozen_cfg_1.items()==frozen_cfg_2.items())# False because == distinguishes the underlying differenceprint(frozen_cfg_1==frozen_cfg_2)# False because == distinguishes these typesprint(frozen_cfg_1==cfg_1)# But eq_as_configdict() treats both as ConfigDict, so these are True:print(frozen_cfg_1.eq_as_configdict(cfg_1))print(cfg_1.eq_as_configdict(frozen_cfg_1))Equality checking with lazy computationEquality checks see if the computed values are the same. Equality is satisfied
if two sets of computations are different as long as they result in the same
value.importml_collectionscfg_1=ml_collections.ConfigDict()cfg_1.a=1cfg_1.b=cfg_1.get_ref('a')+2cfg_2=ml_collections.ConfigDict()cfg_2.a=1cfg_2.b=cfg_2.get_ref('a')*3# True because all computed values are the sameprint(cfg_1==cfg_2)Locking and copyingHere is an example withlock()anddeepcopy():importcopyimportml_collectionscfg=ml_collections.ConfigDict()cfg.integer_field=123# Locking prohibits the addition and deletion of new fields but allows# modification of existing values.cfg.lock()try:cfg.integer_field=124# Raises AttributeError and suggests valid field.exceptAttributeErrorase:print(e)withcfg.unlocked():cfg.integer_field=1555# Works fine too.# Get a copy of the config dict.new_cfg=copy.deepcopy(cfg)new_cfg.integer_field=-123# Works fine.print(cfg)Dictionary attributes and initializationimportml_collectionsreferenced_dict={'inner_float':3.14}d={'referenced_dict_1':referenced_dict,'referenced_dict_2':referenced_dict,'list_containing_dict':[{'key':'value'}],}# We can initialize on a dictionarycfg=ml_collections.ConfigDict(d)# Reference structure is preservedprint(id(cfg.referenced_dict_1)==id(cfg.referenced_dict_2))# True# And the dict attributes have been converted to ConfigDictprint(type(cfg.referenced_dict_1))# ConfigDict# However, the initialization does not look inside of lists, so dicts inside# lists are not converted to ConfigDictprint(type(cfg.list_containing_dict[0]))# dictMore ExamplesFor more examples, take a look atml_collections/config_dict/examples/For examples and gotchas specifically about initializing a ConfigDict, seeml_collections/config_dict/examples/config_dict_initialization.py.Config FlagsThis library adds flag definitions toabsl.flagsto handle config files. It
does not wrapabsl.flagsso if using any standard flag definitions alongside
config file flags, users must also importabsl.flags.Currently, this module adds two new flag types, namelyDEFINE_config_filewhich accepts a path to a Python file that generates a configuration, andDEFINE_config_dictwhich accepts a configuration directly. Configurations are
dict-like structures (seeConfigDict) whose nested elements
can be overridden using special command-line flags. See the examples below
for more details.UsageUseml_collections.config_flagsalongsideabsl.flags. For
example:script.py:fromabslimportappfromabslimportflagsfromml_collections.config_flagsimportconfig_flagsFLAGS=flags.FLAGSconfig_flags.DEFINE_config_file('my_config')defmain(_):print(FLAGS.my_config)if__name__=='__main__':app.run(main)config.py:# Note that this is a valid Python script.# get_config() can return an arbitrary dict-like object. However, it is advised# to use ml_collections.ConfigDict.# See ml_collections/config_dict/examples/config_dict_basic.pyimportml_collectionsdefget_config():config=ml_collections.ConfigDict()config.field1=1config.field2='tom'config.nested=ml_collections.ConfigDict()config.nested.field=2.23config.tuple=(1,2,3)returnconfigNow, after running:pythonscript.py--my_config=config.py\--my_config.field1=8\--my_config.nested.field=2.1\--my_config.tuple='(1, 2, (1, 2))'we get:field1: 8
field2: tom
nested:
  field: 2.1
tuple: !!python/tuple
- 1
- 2
- !!python/tuple
  - 1
  - 2Usage ofDEFINE_config_dictis similar toDEFINE_config_file, the main
difference is the configuration is defined inscript.pyinstead of in a
separate file.script.py:fromabslimportappfromabslimportflagsimportml_collectionsfromml_collections.config_flagsimportconfig_flagsconfig=ml_collections.ConfigDict()config.field1=1config.field2='tom'config.nested=ml_collections.ConfigDict()config.nested.field=2.23config.tuple=(1,2,3)FLAGS=flags.FLAGSconfig_flags.DEFINE_config_dict('my_config',config)defmain(_):print(FLAGS.my_config)if__name__=='__main__':app.run()config_fileflags are compatible with the command-line flag syntax. All the
following options are supported for non-boolean values in configurations:-(-)config.field=value-(-)config.field valueOptions for boolean values are slightly different:-(-)config.boolean_field: set boolean value to True.-(-)noconfig.boolean_field: set boolean value to False.-(-)config.boolean_field=value:valueistrue,false,TrueorFalse.Note that-(-)config.boolean_field valueis not supported.Parameterising the get_config() functionIt's sometimes useful to be able to pass parameters intoget_config, and
change what is returned based on this configuration. One example is if you are
grid searching over parameters which have a different hierarchical structure -
the flag needs to be present in the resulting ConfigDict. It would be possible
to include the union of all possible leaf values in your ConfigDict,
but this produces a confusing config result as you have to remember which
parameters will actually have an effect and which won't.A better system is to pass some configuration, indicating which structure of
ConfigDict should be returned. An example is the following config file:importml_collectionsdefget_config(config_string):possible_structures={'linear':ml_collections.ConfigDict({'model_constructor':'snt.Linear','model_config':ml_collections.ConfigDict({'output_size':42,}),'lstm':ml_collections.ConfigDict({'model_constructor':'snt.LSTM','model_config':ml_collections.ConfigDict({'hidden_size':108,})})}returnpossible_structures[config_string]The value ofconfig_stringwill be anything that is to the right of the first
colon in the config file path, if one exists. If no colon exists, no value is
passed toget_config(producing a TypeError ifget_configexpects a value.)The above example can be run like:pythonscript.py----config=path_to_config.py:linear\--config.model_config.output_size=256or like:pythonscript.py----config=path_to_config.py:lstm\--config.model_config.hidden_size=512Additional featuresLoads any valid python script which definesget_config()function
returning any python object.Automatic locking of the loaded object, if the loaded object defines a
callable.lock()method.Supports command-line overriding of arbitrarily nested values in dict-like
objects (with key/attribute based getters/setters) of the following types:types.IntType(integer)types.FloatType(float)types.BooleanType(bool)types.StringType(string)types.TupleType(tuple)Overriding is type safe.Overriding ofTupleTypecan be done by passing in thetupleas a string
(see the example in theUsagesection).The overridingtupleobject can be of a different size and have different
types than the original. Nested tuples are also supported.AuthorsSergio Gómez Colmenarejo -sergomez@google.comWojciech Marian Czarnecki -lejlot@google.comNicholas WattersMohit Reddy -mohitreddy@google.com