Repository: ucb-bar/hammer (language: python)

Issue:
Refactor Hammer for PyPI Publishing
This is still a work in progress. The goals are:

- To be able to publish hammer to PyPI so tool and PDK plugins (and Chipyard) can depend on the published artifact
- Use poetry to set up a consistent development environment
- Get rid of all the PYTHONPATH hacks and the need to set HAMMER_VLSI
- Unify the testing scripts so a normal pytest invocation can be used
- Make CI scripts easy to write
- To the minimal extent possible, migrate to using pydantic models in lieu of jsonschema dynamic objects or NamedTuples

The things done so far are:

- Unify the hammer namespace by migrating all subprojects inside a top-level package called hammer
    - `hammer-vlsi/hammer_vlsi` -> `hammer.vlsi`
    - `hammer-vlsi/technology` -> `hammer.technology`
    - `hammer-vlsi/lvs` -> `hammer.lvs` (and so forth)
    - `hammer-tech` -> `hammer.tech`
    - `hammer-config` -> `hammer.config`
    - `hammer-shell` -> `hammer.shell`
- Dependencies are now specified in a pyproject.toml using the latest upstream versions, all submodules removed, no sourceme or other magic required to run tests
- Tests moved into a top-level folder (some tests have been migrated already, I'm working on the others)

To test this out:

1. Checkout the 'packaging' branch
1. Install poetry (https://python-poetry.org/docs/master/#installing-with-the-official-installer)
1. `poetry install` to fetch dependencies and set up a virtualenv
1. `poetry shell` to pop into the virtualenv, run `hammer-vlsi` as usual to run the CLI top-level
1. `pytest tests` to run the tests
1. `poetry build` to build a wheel
Support relative directory meta application inside hierarchical constraints
We add the special key for tracking relative directories to each dictionary that contains a `vlsi.inputs.hierarchical.constraints`.
This allows any future use of this dictionary to be able to resolve `prependlocal` meta actions or any future path relative meta actions.
We avoid adding this path variable if it already exists in the case that this procedure has already occured. Overwriting the path in that case would cause us to update it and then potentially disconnect the relative path from its original directory.
This happens in the normal usage with syn-to-par step creating a new hammer_config file in a different level of the directory hierarchy.
Hierarchical  constraints not propagated correctly in certain circumstances
This is still hearsay at this point, but I keep having issues with the bumps API in my hierarchical design.
I will try to come up with a small example.
Hierarchical  constraints not propagated correctly in certain circumstances
This is still hearsay at this point, but I keep having issues with the bumps API in my hierarchical design.
I will try to come up with a small example.


Discussion / hints:
Hey Vighnesh long time no see and nice to hear from you again! 1) Will this keep the `hammer-vlsi` CLI executable (which is quite useful)? 2) Will it be required to use `poetry` or pypi or will it still be possible to depend on hammer by source?
Hey Edward! Thanks looking over this.

1) Yes, `hammer-vlsi` is CLI executable when `hammer` is installed as a package. These 'installable' scripts are specified in `pyproject.toml`

```toml
[tool.poetry.scripts]
hammer-generate-mdf = "hammer.shell.hammer_generate_mdf:main"
get-config = "hammer.shell.get_config:main"
hammer-vlsi = "hammer.shell.hammer_vlsi:main"
```

2) You can always depend on hammer by source just by cloning it somewhere and adding `hammer/hammer` to your `PYTHONPATH`. Of course, the nicer way to do this is to install hammer as an 'editable local package' using `pip install -e` and that will allow you to use hammer in any virtualenv (or conda env) you want while being able to edit its source files and seeing the changes reflect immediately.
Big time support this, great work taking it on. 

A note on one of your comments: 

`Of course, the nicer way to do this is to install hammer as an 'editable local package' using pip install -e and that will allow you to use hammer in any virtualenv (or conda env) you want while being able to edit its source files and seeing the changes reflect immediately. `

The “editable installs” don’t quite place nice with Poetry / pyproject.toml, at least just yet. Or at least they didn’t as of a few months ago, when we decided to “revert” a bunch of projects (Hdl21, everything Vlsir-related) from pyproject.toml to setup.py. I can look up the problem if helpful, but I recall it was not work-around-able. 

I too would prefer the TOML config; the editable installs, which we use heavily, are the only thing holding it back. 

Good news is any of the features you want, including the ”scripts” / installed entry points, will work with either TOML or setup.py.  
I thought that newer versions of pip are able to handle delegating builds to poetry properly and still support locally editable packages that are only specified via pyproject.toml: https://github.com/python-poetry/poetry/issues/34#issuecomment-1054626460

https://github.com/python-poetry/poetry/issues/34#issuecomment-1193365526

> poetry-core also supports [PEP 660](https://peps.python.org/pep-0660/) in the stable and beta versions, which allows for a Poetry-managed project to be installed in editable mode by any PEP 517 frontend (e.g. Poetry itself, pip, etc) that also supports PEP 660.

But I haven't tried this out yet myself; I'll try it out with the tool plugin packages once I get this PR done
That would be great. I certainly had the impression they *intended* for it to work at the time. 

Looking back, it appears we abandoned Poetry in January: 
https://github.com/dan-fritchman/Hdl21/commit/afe44015c98e61cca84358b5a507efd8d5a2be22

And the commentary on the Poetry issue has all been since. Hopefully they’ve added this support.

@dpgrubb13 @harrisonliew This is ready for "review". I know that reviewing these changes line-by-line is going to be difficult, so it is better if you can set up the dev environment yourself and verify it works.

The documentation is here (https://github.com/ucb-bar/hammer/blob/packaging/doc/Hammer-Basics/Hammer-Setup.md#developer-setup). It should walk you through the process of setting up the dev environment, running tests, typechecking, testing multiple Python versions, and building the docs.

I've published this branch to TestPyPI (https://test.pypi.org/project/hammer-vlsi/) just to demonstrate the flow works.

While people are verifying that the dev flow works fine, I will use the published hammer-vlsi artifact to begin porting over the tool plugin repos. @dpgrubb13 will set up a strawman hammer-example repo which will use the published artifacts and implement a simple rtl2gds flow with asap7 and/or sky130 and the cadence toolchain.
I can confirm the dev flow works for me.
@harrisonliew 

1. Yes we should make sure the tests don't pollute the source directory. I need to hunt down where every emission comes from and make sure to redirect it by setting --obj_dir or running hammer inside a scratch test directory.
2. No, having py.typed in `hammer/hammer` is the right thing to do (https://peps.python.org/pep-0561/#packaging-type-information). I'm not sure why py.typed is not in the top-level ruamel.yaml package, but the hack works at least.
3. Yes that dependency is removed, and there is no need for json schema parsing and dynamic object construction anymore since TechJSON is just a Pydantic dataclass now (probably should rename TechJSON later when we do configuration as code). Yes, this means tech plugins need their json patched to convert spaces to underscores in keys. I've already done this for asap7 and it is pretty quick and easy so I anticipate it should be the same for other tech jsons.
4. The published gdstk includes the C++ library and the Python bindings together. The C++ .so needs to be built from source and the instructions in the asap7 README still apply (on BWRC machines we will have to compile gdstk manually with the documented hacks). I can clean up the asap7 readme with specific instructions.
Unit test?
> Unit test?

I'm inclined not to gate this PR based on a unit test because we don't have any unit tests for hierarchical mode besides this width/height matching: https://github.com/ucb-bar/hammer/blob/95428fb44f3d8eda395b496dbdd4e04689e08acb/src/hammer-vlsi/test.py#L775. This is beyond the scope of this PR (opening an issue instead).
I found the following:

* Bumps constraints are not propagated to `par` in the `syn_to_par` when they are placed under `vlsi.inputs.hierarchical.constraints: [{<module>: [{vlsi.inputs.bumps..}]}]`. I believe this has something to do with the raw dicts all as independent lists underneath the module key? The workaround I have for the moment is to put bumps at the top level and skip the bumps step in leaf modules.

* `_meta` doesn't work for hierarchical module constraints. For example:
I have in `foo.yml`:
```vlsi.inputs.hierarchical:
  constraints:
  - a: [x1, x2]
  - b: [y]
```
and in `bar.yml`:
```vlsi.inputs.hierarchical:
  constraints:
  - a: [z]
```
In the `get_hierarchical_settings` method, it combines these hierarchical constraints together using `reduce(add_dicts, list_of_hier_constraints, {})`. Unfortunately, this generates:
```vlsi.inputs.hierarchical:
  constraints:
  - a: [z]
  - b: [y]
```
because `a: [z]` overwrote `a: [x1, x2]`.

* `vlsi.inputs.supplies` is present for leaf modules in the `config_db_temp.json` for the module, but is immediately followed by the default supplies (VDD, VSS).
@harrisonliew can you work on this?
I found the following:

* Bumps constraints are not propagated to `par` in the `syn_to_par` when they are placed under `vlsi.inputs.hierarchical.constraints: [{<module>: [{vlsi.inputs.bumps..}]}]`. I believe this has something to do with the raw dicts all as independent lists underneath the module key? The workaround I have for the moment is to put bumps at the top level and skip the bumps step in leaf modules.

* `_meta` doesn't work for hierarchical module constraints. For example:
I have in `foo.yml`:
```vlsi.inputs.hierarchical:
  constraints:
  - a: [x1, x2]
  - b: [y]
```
and in `bar.yml`:
```vlsi.inputs.hierarchical:
  constraints:
  - a: [z]
```
In the `get_hierarchical_settings` method, it combines these hierarchical constraints together using `reduce(add_dicts, list_of_hier_constraints, {})`. Unfortunately, this generates:
```vlsi.inputs.hierarchical:
  constraints:
  - a: [z]
  - b: [y]
```
because `a: [z]` overwrote `a: [x1, x2]`.

* `vlsi.inputs.supplies` is present for leaf modules in the `config_db_temp.json` for the module, but is immediately followed by the default supplies (VDD, VSS).
@harrisonliew can you work on this?

Fix patch:
diff --git a/doc/Hammer-Use/Hierarchical.rst b/doc/Hammer-Use/Hierarchical.rst
index ae0263964..d81fe10df 100644
--- a/doc/Hammer-Use/Hierarchical.rst
+++ b/doc/Hammer-Use/Hierarchical.rst
@@ -11,7 +11,7 @@ Hierarchical Hammer Config
 The hierarchal flow is controlled in the ``vlsi.inputs.hierarchal`` namespace. To specify hierarchical mode, you must specify the following keys. In this example, we have our top module set as ``ChipTop``, with a submodule ``ModuleA`` and another submdule ``ModuleAA`` below that (these are names of Verilog modules).
 
 .. code-block:: yaml
-    
+
     vlsi.inputs.hierarchical:
       mode: hierarchical
       top_module: ChipTop
@@ -23,42 +23,31 @@ The hierarchal flow is controlled in the ``vlsi.inputs.hierarchal`` namespace. T
         - ModuleAA
       constraints:
       - ChipTop:
-        - vlsi.core...
-        - vlsi.inputs...
+          vlsi.core...
+          vlsi.inputs...
       - ModuleA:
-        - vlsi.core...
-        - vlsi.inputs...
+          vlsi.core...
+          vlsi.inputs...
       - ModuleAA:
-        - vlsi.core...
-        - vlsi.inputs...
+          vlsi.core...
+          vlsi.inputs...
 
-Note how the configuration specific to each module in ``vlsi.inputs.hierarchical.constraints`` are list items, whereas in a flat flow, they would be at the root level.
+Note how the configuration specific to each module in ``vlsi.inputs.hierarchical.constraints`` appears like they would in a flat flow. This means you can apply meta directives to them as normal, as long as the references are in the same parent dictionary (i.e., key = module name).
+If you have constraints for the same module in multiple files, you can use ``vlsi.inputs.hierarchical.constraints_meta: append`` and the constraints will be combined properly.
 
-Placement constraints for each module, however, are not specified here. Instead, they should be specified in ``vlsi.inputs.hierarchical.manual_placement_constraints``. The parameters such as ``x``, ``y``, ``width``, ``height``, etc. are omitted from each constraint for clarity. In the bottom-up hierarchal flow, instances of submodules are of ``type: hardmacro`` because they are hardened from below.
+.. note::
+    In a bottom-up hierarchical flow, submodule instances must have ``type: hardmacro`` in ``vlsi.inputs.placement_constraints`` because they are hardened from below.
 
-.. code-block:: yaml
+Special considerations for legacy support:
 
-    vlsi.inputs.hierarchical:
-      manual_placement_constraints_meta: append
-      manual_placement_constraints:
-      - ChipTop:
-        - path: "ChipTop"
-          type: toplevel
-        - path: "ChipTop/path/to/instance/of/ModuleA"
-          type: hardmacro
-      - ModuleA:
-        - path: "ModuleA"
-          type: toplevel
-        - path: "ModuleA/path/to/instance/of/ModuleAA"
-          type: hardmacro
-      - ModuleAA:
-        - path: "moduleAA"
-          type: toplevel
+* If each module's ``constraints`` is a list of dicts with a single key/value pair, meta actions are not supported and the entire project configuration must be specified in a single file.
+
+* If placement constraints are specified with ``vlsi.inputs.hierarchical.manual_placement_constraints``, all of a given module's placement constraints must be specified in a single file.
 
 Flow Management and Actions
 ---------------------------
 
-Based on the structure in ``vlsi.inputs.hierarchical.manual_modules``, Hammer constructs a hierarchical flow graph of dependencies. In this particular example, synthesis and place-and-route of ``ModuleAA`` will happen first. Synthesis of ``ModuleA`` will then depend on the place-and-route output of ``ModuleAA``, and so forth. 
+Based on the structure in ``vlsi.inputs.hierarchical.manual_modules``, Hammer constructs a hierarchical flow graph of dependencies. In this particular example, synthesis and place-and-route of ``ModuleAA`` will happen first. Synthesis of ``ModuleA`` will then depend on the place-and-route output of ``ModuleAA``, and so forth.
 
 These are enumerated in the auto-generated Makefile, ``hammer.d``, which is placed in the directory pointed to by the ``--obj_dir`` command line flag when the ``buildfile`` action is run. This action must be run BEFORE executing your flow. If you adjust the hierarchy, you must re-run this action.
 
@@ -93,15 +82,7 @@ In a bottom-up hierarchical flow, is is important to remember that submodules do
 Special Notes & Limitations
 ---------------------------
 
-#. Hammer IR keys propagate up through the hierarchical tree. For example, if ``vlsi.inputs.clocks`` was specified in the constraints for ``ModuleAA`` but not for ``ModuleA``, ``ModuleA`` will inherit ``ModuleAA``'s constraints. Take special care of where your constraints come from, especially for a parent module with more than one submodule. To avoid confusion, it is recommended to specify the same set of keys for every module.
-
-#. Hammer IR keys specified at the root level (i.e. outside of ``vlsi.inputs.hierarchical.constraints``) do not override the corresponding submodule constraints. However, if you add a Hammer IR file using ``-p`` on the command line (after the file containing ``vlsi.inputs.hierarchical.constraints``), those keys are global and override submodule constraints unless a meta action is specified. To avoid confusion, it is recommended to specify all constraints with ``vlsi.inputs.hierarchical.constraints``.
-
-#. Due to the structure of ``vlsi.inputs.hierarchical.constraints`` as a list structure, currently, there are the following limitations:
-
-    * You must include all of the constraints in a single file. The config parser is unable to combine constraints from differnt files because most meta actions do not work on list items (advanced users will need to use ``deepsubst``). This will make it harder for collaboration, and unfortunately, changes to module constraints at a higher level of hierarchy after submodules are hardened will trigger the Make dependencies, so you will need to modify the generated Makefile or use redo-targets.
-
-    * Other issues have been observed, such as the bump API failing (see `this issue <https://github.com/ucb-bar/hammer/issues/401>`_ at the top module level. This is caused by similar mechanisms as above. The workaround is to ensure that bumps are specified at the root level for only the top module and the bumps step is removed from submodule par actions.
+#. Hammer IR keys specified at the root level (i.e. outside of ``vlsi.inputs.hierarchical.constraints``) are overridden by the corresponding submodule constraints. Generally, to avoid confusion, it is recommended to specify all constraints module-by-module with ``vlsi.inputs.hierarchical.constraints``.
 
 #. Most Hammer APIs are not yet intelligent enough to constrain across hierarchical boundaries. For example:
 
diff --git a/hammer/config/config_src.py b/hammer/config/config_src.py
index 80493940d..4d92a6b06 100644
--- a/hammer/config/config_src.py
+++ b/hammer/config/config_src.py
@@ -790,9 +790,9 @@ def runtime(self) -> List[dict]:
         return [self._runtime]
 
     @staticmethod
-    def internal_keys() -> Set[str]:
+    def internal_keys() -> List[str]:
         """Internal keys that shouldn't show up in any final config."""
-        return {_CONFIG_PATH_KEY, _NEXT_FREE_INDEX_KEY}
+        return [_CONFIG_PATH_KEY, _NEXT_FREE_INDEX_KEY]
 
     def get_config(self) -> dict:
         """
diff --git a/hammer/config/defaults.yml b/hammer/config/defaults.yml
index f0c86151f..3cad320e8 100644
--- a/hammer/config/defaults.yml
+++ b/hammer/config/defaults.yml
@@ -60,13 +60,13 @@ vlsi.technology:
   bump_block_cut_layer: null # Top cut/via layer for blockage under bumps. (Optional[str])
   # Only used if using vlsi.inputs.bumps
   # TODO: remove this after stackup supports vias ucb-bar/hammer#354
-  
+
   tap_cell_interval: 10.0 # Spacing between columns of tap cells (float)
   # Must be overridden by technology plugin defaults or else you will have DRC/latch-up errors.
 
   tap_cell_offset: 0.0 # Offset of the first column of tape cells from the edge of the floorplan (float)
   # Should be overridden by technology plugin defaults.
-  
+
   routing_layers: null # If specified, set/override the [bottom, top] layers used for routing. (Optional[Tuple[int, int]])
   # Both must match the index number (not name) of layers used in the stackup.
 
@@ -103,7 +103,7 @@ vlsi.technology:
 
   timing_lib_pref: "NLDM"
   # Select a timing lib preference, available options include:
-  # NLDM, ECSM, and CCS (lower or upper case acceptable). 
+  # NLDM, ECSM, and CCS (lower or upper case acceptable).
   # If no preference is specified, then the following preference order is followed:
   # NLDM -> ECSM -> CCS
 
@@ -145,7 +145,7 @@ vlsi.inputs:
     manual_modules: [] # Manual hierarchical definitions used only if hierarchical_definition_source is set to manual mode.
     # Should be a list along the lines of [{"module1": "module1_sub1", "module1_sub2", "module2": "module2_sub"}].
 
-    manual_placement_constraints: [] # Manual hierarchical placement constraints used only if hierarchical_definition_source is set to manual mode.
+    manual_placement_constraints: [] # (Deprecated) Manual hierarchical placement constraints used only if hierarchical_definition_source is set to manual mode.
     # Should be a list along the lines of [{"module1": <list of PlacementConstraint structs>}].
 
     constraints: [] # Manual hierarchical constraints. Overrides generic constraints on a per module basis.
@@ -212,7 +212,7 @@ vlsi.inputs:
   custom_sdc_constraints: [] # List of custom sdc constraints to use. (List[str])
   # These are appended after all other generated constraints (clock, pin, delay, load, etc.).
   # IMPORTANT: Since SDC is unitless, you must manually verify that your time & cap units match the tech library's units.
-  
+
   custom_sdc_files: [] # List of custom sdc files to append. (List[str])
   # These are appended after all other generated constraints and custom_sdc_constraints.
 
@@ -328,7 +328,7 @@ vlsi.inputs:
     # type: float
     pitch: 0.0
     global_x_offset: 0.0 # offset the bump map in the x-axis (float)
-    global_y_offset: 0.0 # offset the bump map in the y-axis (float)    
+    global_y_offset: 0.0 # offset the bump map in the y-axis (float)
     cell: "" # cell (str) - Name of the default bump cell
     assignments: [] # assignments - List of BumpAssignment structs. You must specify one of name or no_connect.
     # If both are specified the bump will be left unconnected
@@ -898,7 +898,7 @@ power.inputs:
   #       "write_profile" - profiles all power types on all categories for all the sub-hierarchies for a given design instance (*.fsdb)
   #       "profile" - run plot_profile + dump_profile
   #       "all" - generate all of the above report formats
-  
+
   # examples:
   #   report_configs: [{waveform_path: "/path/to/fsdb", module: "chiptop", levels:3, start_time: "0ns", end_time: "1ns", toggle_signal:"/ChipTop/clock", num_toggles:1, frame_count:1000, report_name: "my_fsdb_report"}]
   #   report_configs: [{waveform_path: "/path/to/fsdb", inst: "ChipTop/system/tile_prci_domain/tile_reset_domain_tile", interval_size: "1ns", output_formats: ["plot_profile"]}]
diff --git a/hammer/config/defaults_types.yml b/hammer/config/defaults_types.yml
index a2ad9c90f..e1ea96554 100644
--- a/hammer/config/defaults_types.yml
+++ b/hammer/config/defaults_types.yml
@@ -58,7 +58,7 @@ vlsi.technology:
 
   # Offset of the first column of tape cells from the edge of the floorplan
   # type: float
-  tap_cell_offset: float 
+  tap_cell_offset: float
 
   # Set the [bottom, top] layer used for routing. (Optional[Tuple[int, int]])
   routing_layers: Optional[list[int]]
@@ -105,7 +105,7 @@ vlsi.inputs:
     manual_placement_constraints: list[dict[str, list]]
 
     # Manual hierarchical constraints. Overrides generic constraints on a per module basis.
-    constraints: list[dict[str, list]]
+    constraints: list[dict[str, Any]]
 
   # ILMs for hierarchical mode.
   ilms: list[dict[str, str]]
@@ -160,9 +160,9 @@ vlsi.inputs:
     y: int
     # pitch (float) - pitch of bumps in microns
     pitch: float
-    # global_x_offset (float) - offset the bump map in the x-axis 
+    # global_x_offset (float) - offset the bump map in the x-axis
     global_x_offset: float
-    # global_y_offset (float) - offset the bump map in the y-axis 
+    # global_y_offset (float) - offset the bump map in the y-axis
     global_y_offset: float
     # cell (str) - Name of the default bump cell
     cell: str
@@ -283,7 +283,7 @@ par:
   # Spacing around blocks and hardmacros in microns
   blockage_spacing: float
 
-  # Ratio between the power blockage + place halo relative to route blockage. Route blockage should be smaller. 
+  # Ratio between the power blockage + place halo relative to route blockage. Route blockage should be smaller.
   power_to_route_blockage_ratio: float
 
   # Top metal layer that is pulled back around blocks and hardmacros
diff --git a/hammer/par/innovus/__init__.py b/hammer/par/innovus/__init__.py
index 0c1b95593..da50ef507 100644
--- a/hammer/par/innovus/__init__.py
+++ b/hammer/par/innovus/__init__.py
@@ -108,7 +108,7 @@ def fill_outputs(self) -> bool:
 
     @property
     def output_gds_filename(self) -> str:
-        return os.path.join(self.run_dir, "{top}.gds".format(top=self.top_module))
+        return os.path.join(self.run_dir, "{top}.gds.gz".format(top=self.top_module))
 
     @property
     def output_netlist_filename(self) -> str:
diff --git a/hammer/vlsi/cli_driver.py b/hammer/vlsi/cli_driver.py
index 61430267d..574914862 100644
--- a/hammer/vlsi/cli_driver.py
+++ b/hammer/vlsi/cli_driver.py
@@ -456,31 +456,6 @@ def create_par_action(self, custom_hooks: List[HammerToolHookAction],
         return self.create_action("par", hooks if len(hooks) > 0 else None,
                                   pre_action_func, post_load_func, post_run_func)
 
-    @staticmethod
-    def get_full_config(driver: HammerDriver, output: dict) -> dict:
-        """
-        Get the full configuration by combining the project config from the
-        driver with the given output dict (i.e. it contains only
-        "synthesis.output.blah") that we want to combine with the project
-        config.
-        :param driver: HammerDriver that has the full project config.
-        :param output: Output dict containing specific settings we want to add
-                       to the full project config.
-        :return: Full project config combined with the output dict
-        """
-        if "vlsi.builtins.is_complete" in output:
-            if bool(output["vlsi.builtins.is_complete"]):
-                raise ValueError("Output-only config claims it is complete")
-        else:
-            raise ValueError("Output-only config does not appear to be output only")
-
-        output_full = deepdict(driver.project_config)
-        output_full.update(deepdict(output))
-        # Merged configs are always complete
-        if "vlsi.builtins.is_complete" in output_full:
-            del output_full["vlsi.builtins.is_complete"]
-        return output_full
-
     def create_drc_action(self, custom_hooks: List[HammerToolHookAction],
                           pre_action_func: Optional[Callable[[HammerDriver], None]] = None,
                           post_load_func: Optional[Callable[[HammerDriver], None]] = None,
@@ -597,13 +572,13 @@ def action(driver: HammerDriver, append_error_func: Callable[[str], None]) -> Op
                 if not success:
                     driver.log.error("Synthesis tool did not succeed")
                     return None
+                post_run_func_checked(driver)
                 dump_config_to_json_file(os.path.join(driver.syn_tool.run_dir, "syn-output.json"), output)
                 dump_config_to_json_file(os.path.join(driver.syn_tool.run_dir, "syn-output-full.json"),
                                          self.get_full_config(driver, output))
                 if driver.dump_history:
                     dump_config_to_yaml_file(os.path.join(driver.syn_tool.run_dir, "syn-output-history.yml"),
                                             add_key_history(self.get_full_config(driver, output), key_history))
-                post_run_func_checked(driver)
             elif action_type == "par":
                 if not driver.load_par_tool(get_or_else(self.par_rundir, "")):
                     return None
@@ -617,13 +592,13 @@ def action(driver: HammerDriver, append_error_func: Callable[[str], None]) -> Op
                 if not success:
                     driver.log.error("Place-and-route tool did not succeed")
                     return None
+                post_run_func_checked(driver)
                 dump_config_to_json_file(os.path.join(driver.par_tool.run_dir, "par-output.json"), output)
                 dump_config_to_json_file(os.path.join(driver.par_tool.run_dir, "par-output-full.json"),
                                          self.get_full_config(driver, output))
                 if driver.dump_history:
                     dump_config_to_yaml_file(os.path.join(driver.par_tool.run_dir, "par-output-history.yml"),
                                             add_key_history(self.get_full_config(driver, output), key_history))
-                post_run_func_checked(driver)
             elif action_type == "drc":
                 if not driver.load_drc_tool(get_or_else(self.drc_rundir, "")):
                     return None
@@ -637,13 +612,13 @@ def action(driver: HammerDriver, append_error_func: Callable[[str], None]) -> Op
                 if not success:
                     driver.log.error("DRC tool did not succeed")
                     return None
+                post_run_func_checked(driver)
                 dump_config_to_json_file(os.path.join(driver.drc_tool.run_dir, "drc-output.json"), output)
                 dump_config_to_json_file(os.path.join(driver.drc_tool.run_dir, "drc-output-full.json"),
                                          self.get_full_config(driver, output))
                 if driver.dump_history:
                     dump_config_to_yaml_file(os.path.join(driver.drc_tool.run_dir, "drc-output-history.yml"),
                                             add_key_history(self.get_full_config(driver, output), key_history))
-                post_run_func_checked(driver)
             elif action_type == "lvs":
                 if not driver.load_lvs_tool(get_or_else(self.lvs_rundir, "")):
                     return None
@@ -657,13 +632,13 @@ def action(driver: HammerDriver, append_error_func: Callable[[str], None]) -> Op
                 if not success:
                     driver.log.error("LVS tool did not succeed")
                     return None
+                post_run_func_checked(driver)
                 dump_config_to_json_file(os.path.join(driver.lvs_tool.run_dir, "lvs-output.json"), output)
                 dump_config_to_json_file(os.path.join(driver.lvs_tool.run_dir, "lvs-output-full.json"),
                                          self.get_full_config(driver, output))
                 if driver.dump_history:
                     dump_config_to_yaml_file(os.path.join(driver.lvs_tool.run_dir, "lvs-output-history.yml"),
                                             add_key_history(self.get_full_config(driver, output), key_history))
-                post_run_func_checked(driver)
             elif action_type == "sram_generator":
                 if not driver.load_sram_generator_tool(get_or_else(self.sram_generator_rundir, "")):
                     return None
@@ -691,13 +666,13 @@ def action(driver: HammerDriver, append_error_func: Callable[[str], None]) -> Op
                 if not success:
                     driver.log.error("Sim tool did not succeed")
                     return None
+                post_run_func_checked(driver)
                 dump_config_to_json_file(os.path.join(driver.sim_tool.run_dir, "sim-output.json"), output)
                 dump_config_to_json_file(os.path.join(driver.sim_tool.run_dir, "sim-output-full.json"),
                                          self.get_full_config(driver, output))
                 if driver.dump_history:
                     dump_config_to_yaml_file(os.path.join(driver.sim_tool.run_dir, "sim-output-history.yml"),
                                             add_key_history(self.get_full_config(driver, output), key_history))
-                post_run_func_checked(driver)
             elif action_type == "power":
                 if not driver.load_power_tool(get_or_else(self.power_rundir, "")):
                     return None
@@ -711,13 +686,13 @@ def action(driver: HammerDriver, append_error_func: Callable[[str], None]) -> Op
                 if not success:
                     driver.log.error("Power tool did not succeed")
                     return None
+                post_run_func_checked(driver)
                 dump_config_to_json_file(os.path.join(driver.power_tool.run_dir, "power-output.json"), output)
                 dump_config_to_json_file(os.path.join(driver.power_tool.run_dir, "power-output-full.json"),
                                          self.get_full_config(driver, output))
                 if driver.dump_history:
                     dump_config_to_yaml_file(os.path.join(driver.power_tool.run_dir, "power-output-history.yml"),
                                             add_key_history(self.get_full_config(driver, output), key_history))
-                post_run_func_checked(driver)
             elif action_type == "formal":
                 if not driver.load_formal_tool(get_or_else(self.formal_rundir, "")):
                     return None
@@ -731,13 +706,13 @@ def action(driver: HammerDriver, append_error_func: Callable[[str], None]) -> Op
                 if not success:
                     driver.log.error("Formal tool did not succeed")
                     return None
+                post_run_func_checked(driver)
                 dump_config_to_json_file(os.path.join(driver.formal_tool.run_dir, "formal-output.json"), output)
                 dump_config_to_json_file(os.path.join(driver.formal_tool.run_dir, "formal-output-full.json"),
                                          self.get_full_config(driver, output))
                 if driver.dump_history:
                     dump_config_to_yaml_file(os.path.join(driver.formal_tool.run_dir, "formal-output-history.yml"),
                                             add_key_history(self.get_full_config(driver, output), key_history))
-                post_run_func_checked(driver)
             elif action_type == "timing":
                 if not driver.load_timing_tool(get_or_else(self.timing_rundir, "")):
                     return None
@@ -751,10 +726,10 @@ def action(driver: HammerDriver, append_error_func: Callable[[str], None]) -> Op
                 if not success:
                     driver.log.error("Timing tool did not succeed")
                     return None
+                post_run_func_checked(driver)
                 dump_config_to_json_file(os.path.join(driver.timing_tool.run_dir, "timing-output.json"), output)
                 dump_config_to_json_file(os.path.join(driver.timing_tool.run_dir, "timing-output-full.json"),
                                          self.get_full_config(driver, output))
-                post_run_func_checked(driver)
             elif action_type == "pcb":
                 if not driver.load_pcb_tool(get_or_else(self.pcb_rundir, "")):
                     return None
@@ -768,13 +743,13 @@ def action(driver: HammerDriver, append_error_func: Callable[[str], None]) -> Op
                 if not success:
                     driver.log.error("PCB deliverable tool did not succeed")
                     return None
+                post_run_func_checked(driver)
                 dump_config_to_json_file(os.path.join(driver.pcb_tool.run_dir, "pcb-output.json"), output)
                 dump_config_to_json_file(os.path.join(driver.pcb_tool.run_dir, "pcb-output-full.json"),
                                          self.get_full_config(driver, output))
                 if driver.dump_history:
                     dump_config_to_yaml_file(os.path.join(driver.pcb_tool.run_dir, "pcb-output-history.yml"),
                                             add_key_history(self.get_full_config(driver, output), key_history))
-                post_run_func_checked(driver)
             else:
                 raise ValueError("Invalid action_type = " + str(action_type))
             # TODO: detect errors
@@ -1263,6 +1238,31 @@ def valid_actions(self) -> List[str]:
         """Get the list of valid actions for the command-line driver."""
         return list(self.action_map().keys())
 
+    @staticmethod
+    def get_full_config(driver: HammerDriver, output: dict) -> dict:
+        """
+        Get the full configuration by combining the project config from the
+        driver with the given output dict (i.e. it contains only
+        "synthesis.output.blah") that we want to combine with the project
+        config.
+        :param driver: HammerDriver that has the full project config.
+        :param output: Output dict containing specific settings we want to add
+                       to the full project config.
+        :return: Full project config combined with the output dict
+        """
+        if "vlsi.builtins.is_complete" in output:
+            if bool(output["vlsi.builtins.is_complete"]):
+                raise ValueError("Output-only config claims it is complete")
+        else:
+            raise ValueError("Output-only config does not appear to be output only")
+
+        output_full = deepdict(driver.project_config)
+        output_full.update(deepdict(output))
+        # Merged configs are always complete
+        if "vlsi.builtins.is_complete" in output_full:
+            del output_full["vlsi.builtins.is_complete"]
+        return output_full
+
     def args_to_driver(self, args: dict,
                        default_options: Optional[HammerDriverOptions] = None) -> \
             Tuple[HammerDriver, List[str]]:
@@ -1424,6 +1424,8 @@ def syn_pre_func(d: HammerDriver) -> None:
                         module=module))  # TODO(edwardw): fix this ugly os.path.join; it doesn't belong here.
                     # TODO(edwardw): remove ugly hack to store stuff in parent context
                     base_project_config[0] = deeplist(driver.project_configs)
+                    print(base_project_config[0])
+                    print(config)
                     d.update_project_configs(deeplist(base_project_config[0]) + [config])
 
                 def par_pre_func(d: HammerDriver) -> None:
@@ -1483,7 +1485,7 @@ def post_run(d: HammerDriver, rundir: str) -> None:
                     with open(os.path.join(rundir, "module_config.json"), "w") as f:
                         new_output_json = json.dumps(config, cls=HammerJSONEncoder, indent=4)
                         f.write(new_output_json)
-
+                    # Restore the original project config to remove hierarchical settings.
                     d.update_project_configs(deeplist(base_project_config[0]))
 
                 def syn_post_run(d: HammerDriver) -> None:
diff --git a/hammer/vlsi/driver.py b/hammer/vlsi/driver.py
index 8b2862bdd..eedc9af9d 100644
--- a/hammer/vlsi/driver.py
+++ b/hammer/vlsi/driver.py
@@ -65,7 +65,7 @@ def __init__(self, options: HammerDriverOptions, extra_project_config: dict = {}
         file_logger = HammerVLSIFileLogger(options.log_file)
         HammerVLSILogging.add_callback(file_logger.callback)
         self.log = HammerVLSILogging.context()  # type: HammerVLSILoggingContext
-        
+
         # Create a new hammer database.
         self.database = hammer_config.HammerDatabase()  # type: hammer_config.HammerDatabase
 
@@ -1670,8 +1670,8 @@ def _hierarchical_helper(self) -> Tuple[List[Tuple[str, dict]], Dict[str, Tuple[
         hier_source_key = "vlsi.inputs.hierarchical.config_source"
         hier_source = str(self.database.get_setting(hier_source_key))
         hier_modules = {}  # type: Dict[str, List[str]]
+        hier_constraints = {}  # type: Dict[str, Any]
         hier_placement_constraints = {}  # type: Dict[str, List[PlacementConstraint]]
-        hier_constraints = {}  # type: Dict[str, List[Dict]]
 
         # This is retrieving the list of hard macro sizes to be used when creating PlacementConstraint tuples later
         list_of_hard_macros = self.database.get_setting("vlsi.technology.extra_macro_sizes")  # type: List[Dict]
@@ -1686,56 +1686,83 @@ def _hierarchical_helper(self) -> Tuple[List[Tuple[str, dict]], Dict[str, Tuple[
             if len(list_of_hier_modules) == 0:
                 raise ValueError("No hierarchical modules defined manually in manual hierarchical mode")
             hier_modules = reduce(add_dicts, list_of_hier_modules)
+            all_modules = list(hier_modules.keys()) + reduce_list_str(add_lists, hier_modules.values())
+            in_place_unique(all_modules)
 
-            list_of_placement_constraints = self.database.get_setting(
-                "vlsi.inputs.hierarchical.manual_placement_constraints")  # type: List[Dict]
-            assert isinstance(list_of_placement_constraints, list)
-            combined_raw_placement_dict = reduce(add_dicts, list_of_placement_constraints, {})  # type: Dict[str, List[Dict[str, Any]]]
-
-            # This helper function filters only the dict containing the toplevel placement constraint, if any, from the provided list of dicts.
-            # If the list does not contain a toplevel constraint, it returns None.
-            def get_toplevel(d: List[Dict[str, Any]]) -> Optional[Dict[str, Any]]:
-                results = list(filter(lambda x: x["type"] == "toplevel", d))
-                if len(results) == 0:
-                    return None
-                else:
-                    return results[-1]
-
-            # Use the above helper method to filter down the combined raw placement dict into a dict:
-            # - keys are hierarchical module name
-            # - values are dicts containing toplevel constraints or None
-            toplevels_opt = {k: get_toplevel(v) for k, v in combined_raw_placement_dict.items()}  # type: Dict[str, Optional[Dict[str, Any]]]
-            # This filters out all of the Nones to get only hierarchical modules with toplevel placement constraints
-            toplevels = {k: v for k, v in toplevels_opt.items() if v is not None}  # type: Dict[str, Dict[str, Any]]
-            # This converts each dict entry into a MacroSize tuple, which should now represent all hierarchical modules
-            hier_macros = [MacroSize(library="", name=x[0], width=x[1]["width"], height=x[1]["height"]) for x in toplevels.items()]
-            masters = hard_macros + hier_macros
-
-            hier_placement_constraints = {key: list(map(partial(PlacementConstraint.from_masters_and_dict, masters), lst))
-                                          for key, lst in combined_raw_placement_dict.items()}
             # Iterate over project configs to find which ones contain hierarchical constraints
             # For each file that does append its path to the special key in the extracted
             # hierarchical constraints section.
-            """
             hier_constraint_source = ""  # type: str
+            [config_path_key, _] = self.database.internal_keys()
             for project_conf in self.project_configs:
-                if "vlsi.inputs.hierarchical.constraints" in project_conf:
-                    hier_constraint_source = project_conf[hammer_config._CONFIG_PATH_KEY]
+                try:
+                    hier_constraint_source = project_conf[config_path_key]
                     pc = project_conf["vlsi.inputs.hierarchical.constraints"]  # type: List[Dict]
                     # Add CONFIG_PATH_KEY to actual project configs for each project config's hierarchical constraint
                     # keys then update project configs at the end
                     for md in pc:
-                        md  # one entry dict with a list
-                        for m in md.keys():
-                            if hammer_config._CONFIG_PATH_KEY in md[m][-1]:
+                        md  # one entry dict with a list or dict
+                        for mc in md.values():
+                            if config_path_key in mc:
                                 pass
+                            elif isinstance(mc, list):  # legacy (see below)
+                                mc.append({config_path_key: hier_constraint_source})
+                            elif isinstance(mc, dict):  # new (see below)
+                                mc[config_path_key] = hier_constraint_source
                             else:
-                                md[m].append({hammer_config._CONFIG_PATH_KEY: hier_constraint_source})
-            """
+                                raise TypeError("Invalid type for hierarchical constraints")
+                except:
+                    # No _config_path in the config (from tests)
+                    # Or no vlsi.inputs.hierarchical.constraints in project_conf
+                    continue
             self.update_project_configs(self.project_configs)
             list_of_hier_constraints = self.database.get_setting(
                     "vlsi.inputs.hierarchical.constraints") # type: List[Dict]
-            hier_constraints = reduce(add_dicts, list_of_hier_constraints, {})
+            if len(list_of_hier_constraints) > 0:
+                # Check if the hierarchical constraints are a list of dicts or a dict
+                # list of dicts is for legacy compatibility. Meta directives don't work.
+                if isinstance(next(iter(list_of_hier_constraints[0].values())), list):
+                    self.log.warning("In vlsi.inputs.hierarchical.constraints, specifying constraints for each module " +
+                        "as a list of single key/value pair dicts is deprecated but still functional. " +
+                        "Recommend using a full dict (like in a flat flow) instead.")
+                    hier_constraints = reduce(add_dicts, list_of_hier_constraints, {})
+                else:
+                    # run combine_configs per module to resolve meta directives
+                    for module in all_modules:
+                        mod_configs = list(map(lambda x: x.get(module, {}), list_of_hier_constraints))
+                        hier_constraints[module] = hammer_config.combine_configs(mod_configs)
+
+            # Process manual placement constraints
+            list_of_placement_constraints = self.database.get_setting(
+                "vlsi.inputs.hierarchical.manual_placement_constraints")  # type: List[Dict]
+            assert isinstance(list_of_placement_constraints, list)
+            # Legacy method: derived from module's vlsi.inputs.placement_constraints
+            if len(list_of_placement_constraints) > 0:
+                self.log.warning("Use of vlsi.inputs.hierarchical.manual_placement_constraints is legacy. " +
+                    "Place them constraints inside vlsi.inputs.placement_constraints for each module instead.")
+                combined_raw_placement_dict = reduce(add_dicts, list_of_placement_constraints, {})  # type: Dict[str, List[Dict[str, Any]]]
+
+                # This helper function filters only the dict containing the toplevel placement constraint, if any, from the provided list of dicts.
+                # If the list does not contain a toplevel constraint, it returns None.
+                def get_toplevel(d: List[Dict[str, Any]]) -> Optional[Dict[str, Any]]:
+                    results = list(filter(lambda x: x["type"] == "toplevel", d))
+                    if len(results) == 0:
+                        return None
+                    else:
+                        return results[-1]
+
+                # Use the above helper method to filter down the combined raw placement dict into a dict:
+                # - keys are hierarchical module name
+                # - values are dicts containing toplevel constraints or None
+                toplevels_opt = {k: get_toplevel(v) for k, v in combined_raw_placement_dict.items()}  # type: Dict[str, Optional[Dict[str, Any]]]
+                # This filters out all of the Nones to get only hierarchical modules with toplevel placement constraints
+                toplevels = {k: v for k, v in toplevels_opt.items() if v is not None}  # type: Dict[str, Dict[str, Any]]
+                # This converts each dict entry into a MacroSize tuple, which should now represent all hierarchical modules
+                hier_macros = [MacroSize(library="", name=x[0], width=x[1]["width"], height=x[1]["height"]) for x in toplevels.items()]
+                masters = hard_macros + hier_macros
+
+                hier_placement_constraints = {key: list(map(partial(PlacementConstraint.from_masters_and_dict, masters), lst))
+                                              for key, lst in combined_raw_placement_dict.items()}
         elif hier_source == "from_placement":
             raise NotImplementedError("Generation from placement not implemented yet")
         else:
@@ -1795,11 +1822,19 @@ def visit_module(mod: str) -> None:
 
             constraint_dict = {
                 "vlsi.inputs.hierarchical.mode": str(mode),
-                "synthesis.inputs.top_module": module,
-                "vlsi.inputs.placement_constraints": list(
-                    map(PlacementConstraint.to_dict, hier_placement_constraints.get(module, [])))
+                "synthesis.inputs.top_module": module
             }
-            constraint_dict = reduce(add_dicts, hier_constraints.get(module, []), constraint_dict)
+            # manual_placement_constraints specified (legacy)
+            if len(hier_placement_constraints) > 0:
+                constraint_dict = add_dicts(constraint_dict, {
+                    "vlsi.inputs.placement_constraints": list(
+                        map(PlacementConstraint.to_dict, hier_placement_constraints.get(module, [])))
+                })
+            if len(hier_constraints) > 0:
+                if isinstance(next(iter(hier_constraints.values())), dict):  # new
+                    constraint_dict = add_dicts(hier_constraints.get(module, {}), constraint_dict)
+                else:  # legacy
+                    constraint_dict = reduce(add_dicts, hier_constraints.get(module, []), constraint_dict)
             output.append((module, constraint_dict))
 
         return (output, dependency_graph)


Test patch:
diff --git a/e2e/configs-design/pass/mock_hier.yml b/e2e/configs-design/pass/mock_hier.yml
index 40ef02692..96e381842 100644
--- a/e2e/configs-design/pass/mock_hier.yml
+++ b/e2e/configs-design/pass/mock_hier.yml
@@ -21,13 +21,36 @@ vlsi.inputs.hierarchical:
         - SubModD
     - SubModB:
         - SubModE
-  manual_placement_constraints:
-    - ChipTop: []
-    - SubModA: []
-    - SubModB: []
-    - SubModC: []
-    - SubModD: []
-    - SubModE: []
+  constraints:
+    - ChipTop:
+        vlsi.inputs.placement_constraints:
+        - path: ChipTop
+          type: toplevel
+          x: 0
+          y: 0
+          width: 1000
+          height: 1000
+          margins:
+            left: 0
+            right: 0
+            top: 0
+            bottom: 0
+    - SubModC:
+        vlsi.inputs.placement_constraints:
+        - path: SubModC
+          type: toplevel
+          x: 0
+          y: 0
+          width: 100
+          height: 100
+          margins:
+            left: 0
+            right: 0
+            top: 0
+            bottom: 0
+        vlsi.inputs.power_spec_type: manual
+        vlsi.inputs.power_spec_contents: fake_command
+        par.generate_power_straps_options.by_tracks.strap_layers: [m3, m4, m5]
 
 vlsi.core.synthesis_tool: "hammer.synthesis.mocksynth"
 synthesis.mocksynth.temp_folder: "obj_dir"
diff --git a/e2e/configs-design/pass/mock_hier_addl.yml b/e2e/configs-design/pass/mock_hier_addl.yml
new file mode 100644
index 000000000..1a38c462f
--- /dev/null
+++ b/e2e/configs-design/pass/mock_hier_addl.yml
@@ -0,0 +1,6 @@
+vlsi.inputs.hierarchical.constraints_meta: append
+vlsi.inputs.hierarchical.constraints:
+  - SubModC:
+      vlsi.inputs.power_spec_type: upf
+      par.generate_power_straps_options.by_tracks.strap_layers: [m6]
+      par.generate_power_straps_options.by_tracks.strap_layers_meta: append
\ No newline at end of file
diff --git a/e2e/configs-design/pass/mock_hier_legacy.yml b/e2e/configs-design/pass/mock_hier_legacy.yml
new file mode 100644
index 000000000..914fc9b7e
--- /dev/null
+++ b/e2e/configs-design/pass/mock_hier_legacy.yml
@@ -0,0 +1,53 @@
+# Generate Make include to aid in flow
+vlsi.core.build_system: make
+
+vlsi.inputs.power_spec_type: "cpf"
+vlsi.inputs.power_spec_mode: "auto"
+
+synthesis.inputs:
+  top_module: "pass"
+  input_files: ["src/pass.v"]
+
+vlsi.inputs.hierarchical:
+  mode: hierarchical
+  top_module: ChipTop
+  config_source: manual
+  manual_modules:
+    - ChipTop:
+        - SubModA
+        - SubModB
+    - SubModA:
+        - SubModC
+        - SubModD
+    - SubModB:
+        - SubModE
+  manual_placement_constraints:
+    - ChipTop: []
+    - SubModA: []
+    - SubModB: []
+    - SubModC:
+      - path: SubModC
+        type: toplevel
+        x: 0
+        y: 0
+        width: 100
+        height: 100
+        margins:
+          left: 0
+          right: 0
+          top: 0
+          bottom: 0
+    - SubModD: []
+    - SubModE: []
+  constraints:
+    - SubModC:
+        vlsi.inputs.power_spec_type: manual
+        vlsi.inputs.power_spec_contents: fake_command
+        par.generate_power_straps_options.by_tracks.strap_layers: [m3, m4, m5]
+
+vlsi.core.synthesis_tool: "hammer.synthesis.mocksynth"
+synthesis.mocksynth.temp_folder: "obj_dir"
+vlsi.core.par_tool: "hammer.par.mockpar"
+vlsi.core.sim_tool: "hammer.sim.mocksim"
+vlsi.core.drc_tool: "hammer.drc.mockdrc"
+vlsi.core.lvs_tool: "hammer.lvs.mocklvs"