Package: configurun
ConfigurunConfigurun is a cross-platform PySide6-based package that implements an application for managing, creating and (remotely) running python configurations.
The app automatically generates editable GUIs based on an existingargparse.Argumentparseror a python-@dataclass, making it easy to view and edit parameters in an orginized manner - even when the amount of parameters is large.Configurun was designed mainly with machine-learning tasks in mind, but can be used for any python script or framework that takes arguments as an input.The Configurun-app is especially useful for scripts/experiments that require a lot of arguments to be tweaked across many experiment-runs. It also makes the process of running experiments remotely much easier by enabling the user to edit, add and schedule tasks on any running Configurun server-instance reachable via a network connection, or locally.Configurun was created in tandem withpyside6-utils.Table of contentsConfigurunTable of contentsFeaturesConfiguration EditorRun QueueRemote-processingInstallationHow to run?Local AppClient AppServer-instanceOption-sourceCustom Options (@dataclass)Custom Options (ArgumentParser)Custom Options (Callable)Target FunctionConfigurationOption metadataSSH-tunnelingNo-hopHoppingFeaturesConfiguration EditorThe configuration editor is automatically generated using either (groups of)@dataclass-classor anArgumentParser-instance. Editors are specifically created for each option-property based on the provided types (andextra constraints).
Help-messages are displayed on hover, required arguments are highlighted when not filled in, etc.More complex typehints usingList/Union/Literalare supported and are automatically built into widgets that allow the user to add/remove items and switch input-types:We can also define our ownoption-source-methodto dynamically create new option-groups based on the current configuration. This can be useful if we want to group options together, and only show certain groups when an attribute of another group is set to a certain value. E.g: only showExtendedExampleModel-options if propertymodel_typeinMainOptionsis set to"ExtendedExampleModel".Configurations can be saved and loaded, a file-explorer view for the current workspace is made available.:Run QueueThe run-queue window manages the currently running items. This could either be locally when using alocal-app, or remotely, when using aclient-appand aserver-instanceon which the actual Run-Queue is running. The Run-Queue allows us to add/remove items, pause/resume items, change the queue-order of items, and start autoprocessing, which will automatically start the next item in the queue when the current item is finished. We can set the number of processors as well, to run multiple items in parallel.Configurations are passed to the user-providedtarget-functionin separate processes. The stdout/stderr of each of the items is captured and displayed as a selectable console-output-view in the command-line-output window:Remote-processingInstead of using thelocal-appto manage and run the configurations on your own machine, we can use theclient-appto connect to aserver-instanceon a remote machine. The client-app works analogous to the local-app and allows us to create and manage new configuration, but the Run-Queue runs on the connected remote Configurun-server:InstallationThis package can downloaded fromthis repository, or can be installed directly from PyPi by using pip:pipinstallconfigurunHow to run?Creating the app is done via theconfigurun.create-module. We can create 3 different types of apps:Local app- For running everything locally on your machineClient app- For running the configurations on a remote machine, connects to aserver-instanceServer instance- Command-line instance that listens to connections fromclient-apps. If login is succesful, accepts control of theRunQueuefrom theclient-app.On the client-side, theoptions_sourceshould be set - the template of the settings used to create theconfiguration-editor.On the server/running-machine, thetarget_functionshould be set - the function that actually runs the task/experiment (example).Local AppA local app is an all-in-one app that can be used to create and run configurations locally on your machine.
To run the example app, we can either callrun_example_app()fromconfigurun.examplesor run the following code to construct the app ourselves:### This example will run the app with an example configuration# Also see `configurun/examples/example_target_function.py`# Also see `configurun/examples/example_deduce_new_option_class_types.py`importosfromconfigurun.appimportrun_localfromconfigurun.examplesimportexample_target_function,example_deduce_new_option_classesif__name__=="__main__":#Makes sure bootstrapping process is done when running apprun_local(#Create and runs a local configurun app-instancetarget_function=example_target_function,#The function that will be called with the configurationoptions_source=example_deduce_new_option_classes,#Template for UI-optiosn: Callable/@datclass/ArgumentParserworkspace_path=os.path.join(#Settings, configs and the Run-Queue will be saved/loaded from/to hereos.getcwd(),"LocalExampleWorkspace"))In this example,example_target_functionruns a dummy task that logs to a file for 20 seconds. We canspecify our own target-functionto run our own scripts.We canspecify our own options sourceto create our own options-class for the configuration-editor, for example byusing an existingArgumentParser-object.Client AppWe can create a client-app and use it to login to runningserver-instances. We can then use the client-app analogous to thelocal-appto create new confiugrations and add/run/manage configurations on the remote machine.# Opens a client-side app that we can use to connect to and control# the server-instanceimportosfromconfigurun.appimportrun_clientfromconfigurun.examplesimportexample_deduce_new_option_classesif__name__=="__main__":run_client(options_source=example_deduce_new_option_classes,workspace_path=os.path.join(os.getcwd(),"ClientExampleWorkspace"),)Server-instanceThe server-instance is a command-line app that listens to connections fromclient-instance(s) to receive new configurations and commands to manage its RunQueue. The actual target-functions are ran on this machine.NOTE:afterauthentication,pickle/dillis used to transmit data, which indirectly enables arbitrary code execution on the server-side if the password is known. Please run the server on trusted network environments only. Run at your own risk!# Opens a server-instance which tries to connect with clients and allows# them to add configurations to the queue to be run on this machineimportosfromconfigurun.serverimportrun_serverfromconfigurun.examples.example_target_functionimportexample_target_functionif__name__=="__main__":# WARNING:# THIS ALLOWS OTHER MACHINES THAT RESIDE ON THE SAME NETWORK# TO EXECUTE ARBITRARY CODE ON THIS MACHINE IF THEY KNOW THE# PASSWORD. PLEASE RUN IN A TRUSTED NETWORK ENVIRONMENT ONLY# RUN AT YOUR OWN RISK!run_server(target_function=example_target_function,workspace_path=os.path.join(os.getcwd(),"ServerExampleWorkspace"),password="password",#Password to connect to the server, make sure to change this!port=5454#Port to connect to the server, defaults to 5454)Option-sourceWhen creating an app using thecreate-module, we can define a custom source, using theoptions_source=..., so we can construct the UI using our own options.
We can use the following types as an options-source:@dataclass-objectArgumentParser-objectCallableCustom Options (@dataclass)NOTE:Using@dataclass-fields results in more control over the final UI, for a more thorough example, please seethis sectionand/or the example implementations inconfigurun/examples/example_options/example_options.py.NOTE:When implementing custom option-classes, don't forget to add the@dataclass-decorator, and always inherit fromBaseOptionsUI-supported type-hints includestr,int,float,datatime,typing.Literal(and more) and combinations usingtyping.List,typing.Union/|. For example:importosimporttypingfromdataclassesimportdataclassfromconfigurun.configuration.base_optionsimportBaseOptionsfromconfigurun.appimportrun_localfromconfigurun.examplesimportexample_target_function@dataclass#Don't forget to add this(!) - otherwise the app will not recognize the fieldsclassMyCustomOptions(BaseOptions):#Always inherit from BaseOptions (required to run config)simple_int:int=1complex_property:typing.List[typing.Union[int,str]]|None=None# Union = |-notation# etc...if__name__=="__main__":run_local(target_function=example_target_function,options_source=MyCustomOptions,#Simple: each configuration consists of a single options-classworkspace_path=os.path.join(os.getcwd(),"ExampleDataclassOptions"))Custom Options (ArgumentParser)We can use aArgumentParser-object as an options source, this will internally convert the argument parser into a@dataclass-object, which is then used as an options-class. Whenever possible, arguments are also parsed to the UI (e.g.required=True,help="Will be displayed on hover"etc.).importargparseimportosfromconfigurun.appimportrun_localfromconfigurun.examplesimportexample_target_functionparser=argparse.ArgumentParser()parser.add_argument("--required_arg",type=str,required=True,help="Required argument help")#... add more arguments hereif__name__=="__main__":run_local(target_function=example_target_function,options_source=parser,#Parser is converted internally to a dataclass-class which is used as the options-classworkspace_path=os.path.join(os.getcwd(),"ExampleArgparseOptions"))Custom Options (Callable)We define anoption-class as a class that has the@decoratorand inherits from theBaseOptions-class.A configuration is a collection of option-instances, which are grouped toghether in aConfiguration()-wrapper, which enables us to access the attributes of all enclosed options-instances using theconfiguration[attribute]/configuration.<attribute>/option_class.get(attribute, default). For more information, seethis section.As an options-source, we can create a callable which takes the current Configuration-instance as an argument and returns 1 or more new options-classes (notinstances) which is called every time a setting is changed. If the types-change, the UI will be updated to reflect the new templates.
This can be useful if we want to group options together, and only show certain groups when an attribute of another group is set to a certain value. For example:#In this example, we will create a callable which returns new options-classes based on the# current configurationimportosimporttypingfromdataclassesimportdataclassfromconfigurun.appimportrun_localfromconfigurun.examplesimportexample_target_functionfromconfigurun.configurationimportBaseOptions,Configuration@dataclass#NOTE: Always use @dataclass for optionsclassAlwaysTheSame(BaseOptions):#NOTE: Always use BaseOptions as base class for optionsbase_int:int=1#...@dataclassclassCustomOptionsDefault(BaseOptions):simple_int:int=1#...@dataclassclassCustomOptionsUnderConditions(BaseOptions):simple_int:int=2some_more_options:str='Some string'#...defdeduce_new_option_classes(configuration:Configuration)\->typing.Dict[str,typing.Type[BaseOptions|None]]:#Always return a dict of option# classes the key of the dict is the name/group of the option-class# the value is the option-class (@dataclass & BaseOptions) itselfifconfiguration.optionsisNoneorlen(configuration.options)==0:pass#If initial configuration is being retrieved -> return default dictelifconfiguration.base_int==2andconfiguration.simple_int!=1:#Only return the CustomOptionsUnderConditions-class when base_int == 2 & simple_int != 1#NOTE: if we're not sure if attributes exist, we can use the `.get(key, default)` methodreturn{#UI will be built using the following option-classes, each key gets a tab/window:'always_the_same':AlwaysTheSame,'custom_options':CustomOptionsUnderConditions}return{#UI will be built using the following option-classes, each key gets a tab/window:'always_the_same':AlwaysTheSame,'custom_options':CustomOptionsDefault}#NOTE: we must ALWAYS return a dictionary with at least 1 option classif__name__=='__main__':run_local(target_function=example_target_function,options_source=deduce_new_option_classes,workspace_path=os.path.join(os.getcwd(),"ExampleCallableOptions"))Target FunctionThe target function is the function that does all the work. This is the function that is being called when an item starts "running" in the Run-Queue.
It takes a single argument: aConfiguration-instance.
The configuration-object contains all settings as set by the user when "add to queue" was pressed.This example uses the example-configuration fromthe Configuration section, we simply print the values of the configuration to the console:deftarget_function(configuration:Configuration):#Do something with the configurationprint(configuration.simple_int)print(configuration.some_other_int)#etc.If you have replaced anargparse.Argumentparser, this is the place where you insert the user-provided settings to the script that uses theArgumentParser-object. For example:# parsed_args = parser.parse_args() #will be done by user in UI# your_framework_that_used_parsed_args(parsed_args) #Will be called in target_functiondeftarget_function(configurtion:Configuration):# Since we can use the Configuration-instance as a dict# and as configuration.<attribute> we can just# pass it to the framework compatible with the ArgumentParser:your_framework_that_used_parsed_args(configuration)Of course you can also directly pass theyour_framework_that_used_parsed_args(...)-function as the target function when creating the app:if__name__=='__main__':local_app(target_function=your_framework_that_used_parsed_args,options_source=deduce_new_option_classes,workspace_path=os.path.join(os.getcwd(),"ExampleCallableOptions"))This example uses thededuce_new_option_classes-function fromcallable option source example.ConfigurationConfigurun works withconfiguration-objects. A configuration is a collection of option-instances (=@dataclass-instances that inherit fromBaseOptions), which are grouped toghether in aConfiguration-wrapper.
We can think of the option-instances as the different groups of options we want to edit and use in our run (e.g.GeneralOptions(),LogOptions(),ModelOptions(), etc.).
In the simplest case, we have 1 single option-instance which contains all the options, for example:AllOptions().TheConfiguration-wrapper enables us to access the attributes of all enclosed options-instances usingconfiguration[attribute]/configuration.<attribute>/option_class.get(attribute, default).An example of how to use aConfiguration-instance:fromdataclassesimportdataclassfromconfigurun.configurationimportConfigurationfromconfigurun.configurationimportBaseOptions@dataclassclassGeneralOptionsClass(BaseOptions):simple_int:int=1#etc.@dataclassclassOtherOptionClass(BaseOptions):some_other_int:int=2#etc.# Normally, the following would be done by the app using the UI# input and/or the user-provided option-sourceconfig=Configuration()config.options['general_options']=GeneralOptionsClass()config.options['other_options']=OtherOptionClass()#Accessing the options, all of the following are equivalent:print(config['simple_int'])print(config.simple_int)print(config.get('simple_int',-1)))#Would return -1 if key-error occurs#These are also equivalent:print(config['some_other_int'])print(config.some_other_int)print(config.get('some_other_int',-1)))# Note that we can use the config.<attr>-notation to our advantage# when we want to use autocomplete in our editor. For example:# def target_function(configuration: GeneralOptionsClass)# Would result in our editor of choice recognizing/autocompleting# the `configuration.simple_hint` way of accessing `simple_hint`Option metadataThe UI is mainly built around thefield()functionality of python-dataclass, which allows the display-model to make use of the default values, type hints and other information.
While typehints (e.g.int,str,typing.List[int], etc.) are enough to create the editors, we can also provide additional information in themetadata-attribute offield()to further constrain the editors and provide additional information to the user.For example:fromconfigurun.configurationimportbase_optionsfromdataclassesimportfield,dataclass#Used to constrain the editors: (can also be imported from sklearn)frompyside6_utils.classes.constraintsimportInterval,ConstrainedList@dataclassclassTestOptions(BaseOptions):test_int_list_property:typing.List[int]|None=field(default=None,#The default value used in the UImetadata=dict(#Contains additional information for the UIdisplay_name="Test property",#The display-namehelp="This is a test property that can also be none",#On-hover help-messagemrequired=True,#If required, the field is red if not filled inconstraints=[#Limit editors (min/max, options, etc.)#The following constrains the editor to have value > 1ConstrainedList([Interval(type=int,left=1,right=None,closed="both")]),None#Or value can be None]# etc...))For more examples, please see theexample-options.The following metadata-keys are supported:Metadata KeyTypeDescription"display_name"strName to display for this attribute in the view - defaults to the variable name itself"display_path"strPath to display this attribute - we can group/structure items when using a treeview - defaults to no parents"help"strHelp-message which will be shown when the user hovers over this item - empty by default"constraints"List[sklearn_param_validation constraints]Additional constraints on which the editor will be determined to apply to the field*, if none provided, use typehint of the field"required"boolWhether this field is required to be filled in - if true - a red background will appear if the value is not set"editable"boolWhether this field is editable - if false - the editor will be disabled*=Constraints are (almost entirely) sourced from thesklearn.utils._validationmodule and provides a way to constrain the dataclass fields such that the user can only enter valid values. They are also packed into this package underclasses.constraints. The following constraints are supported:ConstraintDescriptionEditor TypetypeThe type of the value should match the type of the constraintbased on typeOptions/ContainerThe value should be one of the options provided in the constraintQComboBoxStrOptionsThe value should be one of the str-options provided in the constraintQComboBoxIntervalThe value should be within the interval provided in the constraintQSpinBoxorQDoubleSpinBox(limited)NoneNoneis a valid value for this field, same astyping.OptionalAdds reset-button to editorRangeThe value should be within the range provided in the constraintQSpinBox(limited)ConstrainedList*(Custom - not part of Sklearn)Indicates a list of of values of a constrained typeBased on type of list*=For example,ConstrainedList([Interval(float, 0.0, 1.0), StrOptions(["string1", "string2"])])indicates a list of values that should either be a float between 0.0 and 1.0, or the string "string1" or "string2". The editor for this field would be constructed as aWidgetListto which the user can addWidgetSwitcher. TheWidgetSwitcherwould then have two widgets, one with aQSpinBoxand one with aQComboBoxas the editor.NOTE:the same editor would be parsed from aTyping.List[typing.Union[float, Typing.Literal["string1", "string2"]]]typehint, minus the bounded-interval constraint on the float:SSH-tunnelingThis section contains a quick tutorial on how to use SSH-tunnels to connect to a remote server-instance in case we're not running client and server on the same machine/network.No-hopA SSH-tunnel can be used to forwardlocalhost:5454on the client-instance-side to<remote>:5454on the server side so we connect to the server instance without permanently port forwarding any ports on the server-side.
If we can directly SSH into the target machine, we can connect the ports as follows (assuming we're using the default5454-port in both the server and client configuration):ssh-L5454:localhost:5454user@remote_hostOn the client-side, we can then connect to the server-instance by usinglocalhost:5454as the host and the password we set in the server-instance.HoppingIf we cannot directly access the remote-machine, we can use a 2-step SSH-tunnel to connect to the server-instance.
This example assumes we have:remote1: directly accessible from our machineremote2: only accessible fromremote1Using default Configurun-port5454on both the client and server-sideWe can then connect toremote2from our machine using the following command:ssh-J<username1>@<remote1>,<username2>@<remote2>-L5454:localhost:5454This example forwardslocalhost:5454->remote2:5454, so we can connect to the server-instance onremote2by usinglocalhost:5454as the host and the password we set in the server-instance.