Package: pyRDDLGym-symbolic
Symbolic Toolset for pyRDDLGymAuthor:Jihwan JeongIn this directory, we provide symbolic toolkits that you can use withpyRDDLGym. Specifically, we provide the following:Symbolic compilation of CPFs(conditional probability functions) in XADD (eXtended Algebraic Decision Diagram)Dynamic Bayes Net (DBN) visualization, allowing dependence analysisSymbolic Dynamic Programming (SDP):Symbolic value iterationSymbolic policy evaluationRDDL simulation with the symbolically compiled modelInstallation# Create and activate a new conda environment
conda create -n symbolic python     # Note: Python 3.12 won't work with Gurobi 10.
conda activate symbolic

# Install the xaddpy >= 0.2.5
pip install xaddpy

# Manually install pyRDDLGym >= 2.0.1
cd ~/path/to/pyRDDLGym
git checkout pyRDDLGym-v2-branch
pip install -e .

# (Optional) manually install rddlrepository-v2
# First, make sure you git cloned rddlrepository
# https://github.com/pyrddlgym-project/rddlrepository
cd ~/path/to/rddlrepository
git checkout v2-branch
pip install -e .

# Install Gurobipy (make sure you have a license)
python -m pip install gurobipy

# Finally, install pyRDDLGym_symbolic
cd ~/path/to/pyRDDLGym_symbolic
pip install -e .Installing pygraphvizIn addition, you'll need to installpygraphvizto visualize XADDs in the graph format.Step 1: Installing graphvizFor Ubuntu/Debian users, run the following command.sudoapt-getinstallgraphvizgraphviz-devFor Fedora and Red Hat systems, you can do as follows.sudodnfinstallgraphvizgraphviz-develFor Mac users, you can usebrewto installgraphviz.brewinstallgraphvizUnfortunately, we do not provide support for Windows systems, though you can refer to thepygraphviz documentationfor information.Step 2: Installing pygraphvizLinux systemspipinstallpygraphvizMacOS: please use conda installcondainstallpygraphvizNote that due to the default installation location bybrew, you need to provide some additional options forpipinstallation.XADD Compilation of CPFsXADD (eXtended Algebraic Decision Diagram) [Sanner at al., 2011] enables compact representation and operations with symbolic variables and functions. In fact, this data structure can be used to represent CPFs defined in a RDDL domain once it is grounded for a specific RDDL instance.We use thexaddpypackage that provides a Python implementation of XADD (originally implemented in Java). To install the package, simply run the following:pipinstallxaddpyXADD compilation of the Wildfire domainIn this article, we are going to walk you through how you can usexaddpyto compile a CPF of a grounded fluent into an XADD node.For example, let's look at theWildfireinstance of 3 x 3 locations.Once the CPFs are grounded for this instance, we can see that the values of the non-fluents will simplify the CPFs. For instance, the neighboring cells of the(x1, y1)location are(x1, y2),(x2, y1), and(x2, y2); hence,burning'(x1, y1)should only depend on the states of these neighbors --- plus(x1, y1)itself --- but none others.Once you compile the CPFs of this instance into an XADD, you can actually see this structure easily. In other words, XADD compilation reveals the DBN dependency structures of different variables, which we also explain below.To run the XADD compilation, we first need to import the domain and instance files. Then, we instantiate theRDDLModelXADDclass with the grounded CPFs given by theRDDLGrounderobject. TheRDDLModelXADDhas the method calledcompilewhich will compile the pyRDDLGymExpressionobjects toXADDnodes.You can find an example run script frompyRDDLGym_symbolic/examples/run_xadd_compilation.py.A nice way to interpret the resulting XADD may be to visualize it as a graph.
You can do this by calling thesave_graphmethod of theXADDobject.# This is the object that has compiled XADD nodesxadd_model=RDDLModelXADD(...)# RDDLModelXADD.context is the XADD context object that# handles all operations and stores all the nodesxadd_model.context.save_graph(xadd_model.cpfs["burning___x1__y1'"],file_name="Wildfire/burning___x1__y1")Here's theresult:If the figure is too small to comprehend, you can click the link above to check out the XADD graph. Notice that the leaf nodes contain either a Boolean value or a real value. This is the case when you passreparam=Falseto theRDDLModelXADDclass constructor. Otherwise, you'll see the Bernoulli variables in the CPFs reparameterized using uniform random variables. When we don't reparameterize, the leaf nodes show the Bernoulli probability values.How will the graph look like forout-of-fuel'(x1, y1)variable? Here's the result ofcontext.save_graph(model_xadd.cpfs["out-of-fuel_x1_y1'"], file_name="out_of_fuel_x1_y1"):Very neat!XADD compilation of a domain with mixed continuous / discrete variablesAlthough the Wildfire example nicely shows how XADD can be used to represent the CPFs of the domain, it only contains Boolean variables. In this part, we will show another example domain that has continuous fluents.The domain we want to look at is theUAV mixeddomain, whose definition is providedhere.If we follow the same procedure described above for the Wildfire domain with the domain name being replaced by'UAV/Mixed', then we can compile the domain/instance in XADD. The overall DBN (dynamic Bayes net) structure of this instance is shown below.Specifically, let's print out the CPF ofvel'(?a1), which is( [-1 + set_acc___a1 <= 0]
        ( [1 + set_acc___a1 <= 0]
                ( [-0.1 + vel___a1 <= 0]
                        ( [0] )
                        ( [-0.1 + vel___a1] )
                )  
                ( [set_acc___a1 + 10*vel___a1 <= 0]
                        ( [0] )
                        ( [0.1*set_acc___a1 + vel___a1] )
                )  
        )  
        ( [0.1 + vel___a1 <= 0]
                ( [0] )
                ( [0.1 + vel___a1] )
        )  
)When visualized withpygraphviz, we get the following:In this case, you can see that the decision nodes have linear inequlity expressions instead of a Boolean decision. As for the function values at the leaf nodes, they are also linear expressions.xaddpypackage can also handle arbitrary nonlinear decisions and function values using SymEngine/SymPy under the hood.Now, you can go ahead and use this functionality to analyze a given RDDL instance!Visualizing DBNs with XADDNext, we can now go ahead and draw DBN diagrams of various RDDL domain/instances. As a running example, we show how you can visualize aWildfireinstance as defined inpyRDDLGym_symbolic/examples/files/Wildfire.If you want to run an example code and follow the steps for better understanding, please take a look at therun_dbn_visualization.pyfile.Instantiate RDDL2Graph objectFirstly, you can instantiate aRDDL2Graphobject by specifying the domain, instance, and some other parameters.frompyRDDLGym_symbolic.core.visualizerimportRDDL2Graphdomain='Wildfire'domain_file=f'pyRDDLGym_symbolic/examples/files/{domain}/domain.rddl'instance_file=f'pyRDDLGym_symbolic/examples/files/{domain}/instance0.rddl'r2g=RDDL2Graph(domain=domain,domain_file=domain_file,instance_file=instance_file,directed=True,strict_grouping=True,)Then, you can visualize the corresponding DBN by callingr2g.save_dbn(file_name='Wildfire')which will save a file namedWildfire_inst_0.pdfto./tmp/Wildfire. Additionally, you can check theWildfire_inst_0.txtfile which records grounded fluent names and their parents in the DBN.The output of the function call looks likethis(opens a new tab showing the PDF file, which we omit to include here since it would take up quite some space).You can also specify a single fluent and/or a ground fluent that you are interested in for visualization. For example,r2g.save_dbn(file_name='Wildfire',fluent='burning',gfluent='x1__y1',file_type='png')will output the following graph:huh, Nice! You can see from this diagram that the next state transition of the burning state at (x1, y1) only depends on 6 grounded variables (i.e., whether neighboring cells are burning; whether this location is out of fuel; whether the put-out action has been taken).To give you a taste of another example, here's the DBN visualization of thePower Generation instance, in which intermediate variables are placed in the middle column:Symbolic Dynamic Programming  (SDP)Value Iteration (VI)With therun_vi.pyfile, you can run a value iteration solver.Here, we provide a detailed dissection of the run script.First, we compile a given RDDL domain/instance to XADD. This step follows the same procedure as in the examples above, so we skip it here.Constructing the MDP problem with the associated XADD modelmdp_parser=Parser()mdp=mdp_parser.parse(xadd_model,xadd_model.discount,concurrency=rddl_ast.instance.max_nondef_actions,is_linear=args.is_linear,include_noop=notargs.skip_noop,is_vi=True,)Then, in lines 46 - 54, we instantiate anMDPParserobject that has theparsemethod, which interprets the XADD RDDL model and construct some necessary attributes, like CPFs and such.Some important operations that happen within theparserare as follows:Bound analysis on continuous variables (lines 50-57 and lines 102-105):# Configure the bounds of continuous states.cont_s_vars=set()forsinmodel.state_fluents:ifmodel.variable_ranges[s]!='real':continuecont_s_vars.add(model.ns[s])cont_state_bounds=self.configure_bounds(mdp,model.invariants,cont_s_vars)mdp.cont_state_bounds=cont_state_bounds......# Configure the bounds of continuous actions.iflen(mdp.cont_a_vars)>0:cont_action_bounds=self.configure_bounds(mdp,model.preconditions,mdp.cont_a_vars)mdp.cont_action_bounds=cont_action_boundsHere, the parser has a method calledconfigure_boundsin which we perform the analysis on bounds of continuous variables. Specifically, the bound information has to be provided instate-invariantsandaction-preconditionsblocks of the original RDDL domain file. If no bounds are provided for a variable, we assume[-inf, inf]as its bounds.Once configured, this bound information is then updated to theXADDcontext object such that each continuous symbolic variable is associated with its upper and lower bounds.Handling of concurrent boolean actions (lines 75 - 90)# Add concurrent actions for Boolean actions.ifis_vi:# Need to consider all combinations of boolean actions.# Note: there's always an implicit no-op action with which# none of the boolean actions are set to True.total_bool_actions=tuple(_truncated_powerset(bool_actions,mdp.max_allowed_actions,include_noop=include_noop,))foractionsintotal_bool_actions:names=tuple(a.nameforainactions)symbols=tuple(a.symbolforainactions)action=BActions(names,symbols,model)mdp.add_action(action)This part is where we handle concurrent actions, specifically for Value Iteration. Here we have a few modeling assumptions. First, continuous actions will always be concurrent, so we only specifically handle concurrent Boolean actions. Second, we provide an option to either use or not use ano-opaction, which sets all Boolean action values toFalse.Now, let's say we have 2 Boolean actions:move___a1andpick___a1. When the concurrency is set to1and we allow thenoopaction, then we'll have the following Boolean actions:noop(i.e.,{move___a1: False, pick___a1: False}){move___a1: True, pick___a1: False}{move___a1: False, pick___a1: True}On the other hand, if the concurrency is set to2, then we will have the following concurrent Boolean actions:noop(i.e.,{move___a1: False, pick___a1: False}){move___a1: True, pick___a1: False}{move___a1: False, pick___a1: True}{move___a1: True, pick___a1: True}That is, the concurrency value specifies the maximum number of Boolean actions that can be taken at each time step, so we should consider all possible combinations, which is done by the_truncated_powersethelper function.We define a classBActionsthat can handle any of these concurrent actions. More importantly, the class implements arestrictmethod in which we restrict a given XADD with the associated action values.Constructing the full CPFs for Boolean next state and interm variables (line 114)By callingmdp.update(is_vi=is_vi), we update the CPFs of Boolean next state and interm variables to fully considerP(b'=0|...). This is a necessary step as in the RDDL file, we have only specified the probability of a Boolean variable beingTrue. Also, theupdatemethod links updated CPFs with each action.Solving the MDPFinally, we callvi_solver.solve()which will perform SDP to obtain the optimal symbolic value function.Notice that thesolvemethod is shared by theValueIterationandPolicyEvaluationsolvers; hence, it's defined inbase.py. The method will return the integer ID of the optimal value function at a set iteration number.Policy Evaluation (PE)With therun_pe.pyfile, you can run a policy evaluation solver.The script is exactly the same as run_vi.py until the XADD RDDL model compilation is done. Then, a slight difference of PE from VI is what we pass to theMDPParser.parsefunction.mdp=mdp_parser.parse(xadd_model,xadd_model.discount,concurrency=rddl_ast.instance.max_nondef_actions,is_linear=args.is_linear,is_vi=False,)In PE, we do not have to specify the maximum concurrency value to the parser as that should be implicitly determined by the given policy. Instead, we setis_vi=Falsesuch that we do not createBActionsobjects.Then, in lines 56 - 62, we instantiate aPolicyParserobject and parse the policy provided in a json format, specified by the argument--policy_fpath. An example policy json file looks like the following (p1.json):{"action-fluents":["a"],"a":"pyRDDLGym_symbolic/examples/files/RobotLinear_1D/policy/a.xadd"}A policy json file should have the following field:"action-fluents": a list of grounded action variable names.Then, it should be followed by "action-name": "file path" pairs for all actions specified in "action-fluents". This json file should specify the file path of each and every action fluent of a given problem; otherwise, an assertion error will occur from the parser.The value of one action variable points to the file path where the XADD of that action is defined. ThePolicyParserwill read in the XADD and perform some checks (e.g., type and dependency checks). Check out the comments in thepolicy_parser.pyfile for more detailed information.Assertion for concurrencyThePolicyParserclass implements an assertion that in the entire state space no more than the setconcurrencynumber of Boolean actions can be set toTrue. Check out the_assert_concurrencymethod in lines 153 - 175 ofpolicy_parser.py.Substitution of the policy into CPFs and reward functionA unique step in PE is where we substitute in the policy XADDs into the CPFs and the reward function. See lines 20 - 58 ofpe.py. Note how we handle the continuous and Boolean action variables differently.Once all the CPFs and reward function are restricted with the given policy XADDs, the remaining steps are identical to VI, except that we do not have to iterate over actions as they have all been already incorporated into CPFs.SimulationOptionally, we showcase how you can simulate an XADD-compiled RDDL instance inpyRDDLGym_symbolic/examples/run_simulation.py. Check outpyRDDLGym_symbolic/core/simulation.pyto see how things are implemented.Disclaimer: note that we won't actively support the simulation functionality as it has a limited use-case. Hence, there is a chance that an update in the main pyRDDLGym repository breaks the current simulation code we provide in this repository. This code is only provided as an example.CitationsIf you use the code provided in this repository, please use the following bibtex for citation:@InProceedings{pmlr-v162-jeong22a,
  title = 	 {An Exact Symbolic Reduction of Linear Smart {P}redict+{O}ptimize to Mixed Integer Linear Programming},
  author =       {Jeong, Jihwan and Jaggi, Parth and Butler, Andrew and Sanner, Scott},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {10053--10067},
  year = 	 {2022},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
}