Repository: mit-ll-responsible-ai/hydra-zen
Pull request #229 (2022-02-21 15:39:34)

Require patch info in version comparisons
This PR improves our internal version-checking utilities for making Hydra/omegaconf compatibility checks. 


Patch:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
deleted file mode 100644
index 5bfa8271..00000000
--- a/CONTRIBUTING.md
+++ /dev/null
@@ -1,117 +0,0 @@
-Thanks for your interest in contributing to hydra-zen! Please read 
-through the following resources before you begin working on any contributions to this 
-code base.
-
-
-- [Installing hydra-zen for development](#installing-hydra-zen-for-development)
-- [Pre-Commit Hooks (Required)](#pre-commit-hooks-required)
-  - [What does this do?](#what-does-this-do)
-- [Running Our Tests Manually](#running-our-tests-manually)
-- [Running Tests Using `tox`](#running-tests-using-tox)
-  - [Measuring Code Coverage](#measuring-code-coverage)
-- [Validating Type Correctness](#validating-type-correctness)
-  
-
-## Installing hydra-zen for development
-
-Install the toolkit along with its test dependencies; checkout the repo, navigate to its top level and run
-
-```shell
-pip install -e .
-```
-
-the `-e` option ensures that any changes that you make to the project's source code will be reflected in your local install – you need not reinstall the package in order for your modifications to take effect.
-
-If your contributions involve changes to our support for NumPy, PyTorch, PyTorch-Lightning, JAX, pydantic, or beartype then you will need to install those dependencies as well, in order for our tests-against-third-parties to run locally in your environment.
-
-
-## Pre-Commit Hooks (Required)
-
-We provide contributors with pre-commit hooks, which will apply auto-formatters and 
-linters to your code before your commit takes effect. You must install these in order to contribute to the repo.
-
-First install pre-commit in your Python environment. Run:
-
-```console
-pip install pre-commit
-```
-
-Then, in the top-level of the `hydra_zen` repo, run:
-
-```console
-pre-commit install
-pre-commit run
-```
-
-Great! You can read more about pre-commit hooks in general here: https://pre-commit.com/
-
-### What does this do?
-
-Our pre-commit hooks run the following auto-formatters on all commits:
-- [black](https://black.readthedocs.io/en/stable/)
-- [isort](https://pycqa.github.io/isort/)
-
-It also runs [flake8](https://github.com/PyCQA/flake8) to enforce PEP8 standards.
-
-## Running Our Tests Manually
-
-Install the latest version of pytest and hypothesis:
-
-```console
-pip install pytest hypothesis
-```
-
-Navigate to the top-level of `hydra-zen` and run:
-
-```console
-pytest tests/
-```
-
-## Running Tests Using `tox`
-
-`tox` is a tool that will create and manage a new Python environment where it can then run hydra-zen's
-automated tests against various Python versions and dependencies.
-
-Install `tox`:
-
-```console
-pip install tox
-```
-
-(if you like to use `conda` environments, you might also install `tox-conda`).
-
-List the various tox-jobs that are defined for hydra-zen:
-
-```console
-tox -a
-```
-
-Then, run the job of choice using:
-
-```console
-tox -e [job-name]
-```
-
-
-### Measuring Code Coverage
-
-Our CI requires that our tests achieve 100% code coverage. The easiest way to measure
-code-coverage is by using `tox`:
-
-```console
-tox -e coverage
-```
-
-This will produce a coverage report that indicates any lines of code that were note covered by tests.
-
-## Validating Type Correctness
-
-Our CI runs the `pyright` type-checker in basic mode against hydra-zen's entire code base and against specific test files; this ensures that our type-annotations are complete and accurate.
-
-If you use VSCode with Pylance, then make sure that `Type Checking Mode` is set to `basic` for your hydra-zen workspace. Your IDE will then mark any problematic code.Other IDEs can leverage the pyright language server to a similar effect. 
-
-While this is helpful for getting immediate feedback about your code, it is no substitute for running `pyright` from the commandline. To do so, [install pyright](https://github.com/microsoft/pyright#command-line) and, from the top-level hydra-zen directory, run:
-
-```console
-pyright --lib tests/annotations/ src/
-```
diff --git a/README.md b/README.md
index 3ec29c3d..e44bbb36 100644
--- a/README.md
+++ b/README.md
@@ -50,18 +50,13 @@ launching Hydra jobs. Please refer to [our documentation](https://mit-ll-respons
 ## Installation
 `hydra-zen` is lightweight: its only dependencies are `hydra-core` and `typing-extensions`.
 
-```console
+```shell script
 pip install hydra-zen
 ```
 
 ## Contributing
 
-Before opening a PR to this repo, consider posting an [issue](https://github.com/mit-ll-responsible-ai/hydra-zen/issues) or a [discussion topic](https://github.com/mit-ll-responsible-ai/hydra-zen/discussions) to share your ideas with us. We will work with you to ensure your feature is well-scoped and that your hard work goes to good use.
-
-(See an obvious bug or typo? Go ahead and just open a PR :) )
-
-For further details refer to [these docs](https://github.com/mit-ll-responsible-ai/hydra-zen/blob/main/CONTRIBUTING.md).
-
+(Still working on these docs...)
 
 ## Join the Discussion
 
diff --git a/ROADMAP.md b/ROADMAP.md
new file mode 100644
index 00000000..d8058d20
--- /dev/null
+++ b/ROADMAP.md
@@ -0,0 +1,62 @@
+# A Road Map of hydra-zen's Development
+
+This document details features and capabilities that we intend to add to hydra-zen in both the near-term and 
+in the long-term.
+
+## Road Map to `v0.4.0`
+
+Hydra's promise of reproducibility -- that it will leave behind a YAML-serialized configuration of any job -- 
+is a killer feature, especially for those who use Hydra to run experiments, simulations, etc. However, it is 
+important that these configs can be validated/maintained, so that compatibility issues with old 
+configs/results can be identified in an automated way. E.g., consider the case where a new release of a lab's 
+own research code is incompatible with some aspects of an old experiment. Ideally, one could incorporate 
+config-validation in their CI so that this issue is identified by nightly checks. Presently, one would likely 
+need to manually re-run the experiment itself in order to catch this problem Thus hydra-zen `v0.4.0` is all 
+about validating configs: be them YAML-based configs, manually defined dataclasses, or configs produced by 
+hydra-zen's config-creation functions. 
+
+### Strict validation performed by `hydra_zen.builds`, `hydra_zen.make_config`, et al.
+
+(Added in [#163](https://github.com/mit-ll-responsible-ai/hydra-zen/pull/163))
+
+hydra-zen's config-creation functions should provide strict validation that:
+   1. All configured values are compatible with Hydra's supported primitive types.
+   2. Fields specified in targeted configs are compatible with the target's signature
+
+As of `v0.4.0rc1` hydra-zen provides runtime support for both of these. Additionally, hydra-zen's internal 
+annotations also enables static tooling to flag unsupported config-value types.
+
+Because these functions perform validation upon constructing the configs themselves, incorporating 
+config-validation into ones CI / automated test suite is trivial: simply write a test that imports the 
+resulting configs. This is generally inexpensive and does not actually require the configs to be instantiated.
+Furthermore, any runtime validation errors will be localized to a particular config and thus will be easy to debug.
+
+### Built-in support for additional primitive types
+
+(Added in [#163](https://github.com/mit-ll-responsible-ai/hydra-zen/pull/163))
+
+hydra-zen can elegantly provide support for a variety of common types that are not natively supported by 
+Hydra. Thus values of types like `complex` and `pathlib.Path` can be specified directly within configs, and 
+hydra-zen will automatically create targeted configs that "build" those values; the resulting structured 
+configs carry no dependency on hydra-zen.
+
+### Validation of general configs
+
+hydra-zen should provide a `hydra_zen.validate` function that is capable of processing any yaml-string or 
+structured config and validate that:
+   1. All configured values are compatible with Hydra's supported primitive types.
+   2. Fields specified in targeted configs are compatible with the target's signature.
+   3. The annotations specified in structured configs are compatible with the subset of annotations that are natively supported by Hydra.
+   4. **MAYBE** perform runtime type-checking of values against annotated types. In the case that a targeted config is specified as a value, we could "look ahead" at the type that will be produced via recursive instantiation. If this is something we want to enable, we would strongly prefer that we either leverage a third-party type-checker to handle this (e.g. pydantic or beartype), or that we utilize omegaconf/Hydra's internal type-checking utilities.
+
+This will involve being able to recurse through nested configs.
+
+One use case in mind here is that a user runs a Hydra job and wants to ensure that the job can be reproduced 
+in the future against updated dependencies. Thus the user would thus add to their test suite a test that 
+feeds the job's resulting YAML-based config to `hydra_zen.validate`. It should also be convenient for users to
+validate their configs prior to actually launching their Hydra jobs.
+
+`hydra_zen.validate` would likely expose various configurable options for the stages of validation -- and 
+their "strictness levels" -- that will be performed. There is plenty of opportunity for additional tooling 
+here, such as a pytest plugin, but we should stay focused on the essentials here. We would be open to outside 
+contributions that provide this sort of additional polish.
diff --git a/src/hydra_zen/_compatibility.py b/src/hydra_zen/_compatibility.py
index 4c1bfef4..8f6798f5 100644
--- a/src/hydra_zen/_compatibility.py
+++ b/src/hydra_zen/_compatibility.py
@@ -16,14 +16,21 @@ NoneType = type(None)
 class Version(NamedTuple):
     major: int
     minor: int
-    patch: Optional[int] = None
+    patch: int
 
 
 def _get_version(ver_str: str) -> Version:
     # Not for general use. Tested only for Hydra and OmegaConf
     # version string styles
-    major, minor = (int(v) for v in ver_str.split(".")[:2])
-    return Version(major=major, minor=minor)
+
+    splits = ver_str.split(".")[:3]
+    if not len(splits) == 3:  # pragma: no cover
+        raise ValueError(f"Version string {ver_str} couldn't be parsed")
+
+    major, minor = (int(v) for v in splits[:2])
+    patch_str, *_ = splits[-1].split("rc")
+
+    return Version(major=major, minor=minor, patch=int(patch_str))
 
 
 OMEGACONF_VERSION: Final = _get_version(omegaconf.__version__)
@@ -37,7 +44,7 @@ HYDRA_VERSION: Final = _get_version(hydra.__version__)
 #
 # Uncomment dynamic setting once OmegaConf merges fix:
 # https://github.com/omry/omegaconf/pull/832
-PATCH_OMEGACONF_830: Final = True  # OMEGACONF_VERSION <= Version(2, 1)
+PATCH_OMEGACONF_830: Final = True  # OMEGACONF_VERSION < Version(2, 1, 1)
 
 # Hydra's instantiate API now supports partial-instantiation, indicated
 # by a `_partial_ = True` attribute.
@@ -45,7 +52,7 @@ PATCH_OMEGACONF_830: Final = True  # OMEGACONF_VERSION <= Version(2, 1)
 #
 # Uncomment dynamice setting of `HYDRA_SUPPORTS_PARTIAL` once we can
 # begin testing against nightly builds of Hydra
-HYDRA_SUPPORTS_PARTIAL: Final = False  # Version(1, 1) < HYDRA_VERSION
+HYDRA_SUPPORTS_PARTIAL: Final = False  # Version(1, 1, 1) < HYDRA_VERSION
 
 # Indicates primitive types permitted in type-hints of structured configs
 HYDRA_SUPPORTED_PRIMITIVE_TYPES: Final = {int, float, bool, str, Enum}


Test patch:
diff --git a/tests/annotations/declarations.py b/tests/annotations/declarations.py
index e02c14d1..9957db37 100644
--- a/tests/annotations/declarations.py
+++ b/tests/annotations/declarations.py
@@ -18,7 +18,7 @@ from dataclasses import dataclass
 from enum import Enum
 from functools import partial
 from pathlib import Path
-from typing import Any, Callable, List, Tuple, Type, TypeVar, Union
+from typing import Callable, List, Tuple, Type, TypeVar
 
 from omegaconf import MISSING, DictConfig, ListConfig
 
@@ -352,18 +352,6 @@ def supported_primitives():
     # make_config(a={ADataclass: 1})
 
 
-def check_zen_field():
-    ZenField(int)
-    ZenField(Callable[..., Any])
-    ZenField(List[int])
-
-    ZenField(1.0)  # type: ignore
-
-    # this is a known limitation
-    # TODO: open pyright issue
-    ZenField(Union[int, str])  # type: ignore
-
-
 def check_base_annotations():
     P1 = make_config(x=1)
     P2 = builds(int)
diff --git a/tests/test_config_value_validation.py b/tests/test_config_value_validation.py
index 622540bb..34e929f7 100644
--- a/tests/test_config_value_validation.py
+++ b/tests/test_config_value_validation.py
@@ -149,10 +149,6 @@ def test_that_configs_passed_by_zen_validation_are_serializable(
     config_construction_fn, value
 ):
     # `value` is literally any value that Hypothesis knows how to describe
-    if isinstance(value, str):
-        # avoid interpolation issues
-        assume(not value.startswith("${"))
-
     try:
         Conf = config_construction_fn(value)
     except (HydraZenUnsupportedPrimitiveError, ModuleNotFoundError):
diff --git a/tests/test_utils.py b/tests/test_utils.py
index e9974b5d..6a037582 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -280,25 +280,30 @@ def test_strings_that_fail_to_interpolate_are_not_interpolated_strings(any_text)
     assert out == 1 or not is_interpolated_string(any_text)
 
 
-@pytest.mark.parametrize(
-    "version_string, expected",
-    [
-        ("1.2.0", Version(1, 2)),
-        ("2.1.0dev24", Version(2, 1)),
-    ],
+@given(
+    major=st.integers(0, 100),
+    minor=st.integers(0, 100),
+    patch=st.integers(0, 100),
+    # tests Hydra-style and OmegaConf-style dev/rc-style
+    # release strings. See:
+    # https://pypi.org/project/hydra-core/#history
+    # https://pypi.org/project/omegaconf/#history
+    patch_suffix=st.just("")
+    | st.integers(0, 100).map(lambda x: f"rc{x}")
+    | st.integers(0, 100).map(lambda x: f".dev{x}"),
 )
-def test_get_version(version_string, expected):
+def test_get_version(major: int, minor: int, patch: int, patch_suffix: str):
+    version_string = f"{major}.{minor}.{patch}{patch_suffix}"
+    expected = Version(major, minor, patch)
     assert _get_version(version_string) == expected
 
 
-@pytest.mark.parametrize(
-    "smaller, larger",
-    [
-        (Version(1, 0), Version(2, 0)),
-        (Version(1, 0), Version(1, 1)),
-        (Version(1, 0, 0), Version(1, 0, 1)),
-        (Version(1, 0, 1), Version(1, 1, 0)),
-    ],
-)
-def test_version_comparison(smaller, larger):
-    assert smaller < larger
+def test_version_comparisons():
+    assert Version(1, 0, 0) == Version(1, 0, 0)
+    assert Version(1, 0, 0) <= Version(1, 0, 0)
+    assert Version(1, 0, 0) != Version(2, 0, 0)
+    assert Version(1, 1, 1) < Version(2, 0, 0)
+    assert Version(1, 0, 0) < Version(2, 0, 0)
+    assert Version(1, 0, 0) < Version(1, 1, 0)
+    assert Version(1, 0, 0) < Version(1, 0, 1)
+    assert Version(1, 0, 1) < Version(1, 1, 0)