Repository: mit-ll-responsible-ai/hydra-zen

Issue:
Design Question: Omit default hydra-fields from `builds`?
Currently, `builds` populates all hydra-relevant fields in the dataclass even if those fields are the default values that hydra would infer if they were missing.

## Potential Change / Pros of this

Our current behavior

```python
>>> print(to_yaml(builds(dict)))
_target_: builtins.dict
_recursive_: true
_convert_: none
```
We could omit `_recursive_` and `_convert_`  when these values are the defaults (and thus implicitly rely on hydra's default behavior), and thus end up with a cleaner yaml. 

```python
>>> print(to_yaml(builds(dict)))
_target_: builtins.dict
```


This matters more at-scale.

```
ExpConfig = builds(
    gradient_descent,
    optim=builds(SGD, lr=0.3, momentum=0.0, hydra_partial=True),
    landscape_fn=just(parabaloid),
    starting_xy=(-1.5, 0.5),
    num_steps=20,
)
print(to_yaml(ExpConfig))
_target_: __main__.gradient_descent
_recursive_: true
_convert_: none
optim:
  _target_: hydra_zen.funcs.partial
  _partial_target_:
    _target_: hydra_zen.funcs.get_obj
    path: torch.optim.sgd.SGD
  _recursive_: true
  _convert_: none
  lr: 0.3
  momentum: 0.0
landscape_fn:
  _target_: hydra_zen.funcs.get_obj
  path: __main__.parabaloid
starting_xy:
- -1.5
- 0.5
num_steps: 20
```

could become..

```
_target_: __main__.gradient_descent
optim:
  _target_: hydra_zen.funcs.partial
  _partial_target_:
    _target_: hydra_zen.funcs.get_obj
    path: torch.optim.sgd.SGD
  lr: 0.3
  momentum: 0.0
landscape_fn:
  _target_: hydra_zen.funcs.get_obj
  path: __main__.parabaloid
starting_xy:
- -1.5
- 0.5
num_steps: 20
```
which is tidier.


We would potentially implement the signature with `None` as the defaults `builds(..., hydra_recursive=None, hydra_convert=None)`. Here, fields are only omitted if they are `None`; any user-specified value, even if it is the default, would be included. This enables users to force these fields to appear in the dataclass (although it would be somewhat cumbersome to do so..)

## Cons

- The yamls are less explicit. Users need to know what hydra's default behaviors are. Although I do wonder how much users actually _need_ to know about / adjust these  at the yaml level
- The aforementioned discussion of using `None` as  the default values makes our documentation and behavior more complicated. We have to make `None` ultimately default to whatever the default hydra behavior is... which presumably is only available via "word of mouth" (i.e. docs). And we have to explain to the user why `hydra_recursive=None` exhibits the same behavior of `hydra_recursive=True` - i.e. answer the questions "well why isn't the default just `True` then?"
- Omitting these fields make inheritance via `builds_bases` a bit more complicated; we have to fill in these fields, even if they are the default values, if any of the parents have these fields parents. Otherwise we would mistakenly inherit that parent's fields.


Ultimately what this all comes down to: how much do we care about making yamls more succinct?


Discussion:
I'll think about this, the one question I have is how this impacts "reproducible" configurations.  

Also, I think I'm in the camp that we don't have to worry about making these configs succinct.  For me, if I ever look at the raw yaml outputs I'm doing it to debug, so verbose is good.
@jgbos  as you pointed out, currently `builds` can't inherit hydra-settings from a parent class because we set the defaults. This, I think, is sufficient motivation to move forward with this

Resolving patch:
diff --git a/src/hydra_zen/structured_configs/_implementations.py b/src/hydra_zen/structured_configs/_implementations.py
--- a/src/hydra_zen/structured_configs/_implementations.py
+++ b/src/hydra_zen/structured_configs/_implementations.py
@@ -115,8 +115,8 @@ def hydrated_dataclass(
     *pos_args: Any,
     populate_full_signature: bool = False,
     hydra_partial: bool = False,
-    hydra_recursive: bool = True,
-    hydra_convert: Literal["none", "partial", "all"] = "none",
+    hydra_recursive: Optional[bool] = None,
+    hydra_convert: Optional[Literal["none", "partial", "all"]] = None,
     frozen: bool = False,
 ) -> Callable[[Type[_T]], Type[_T]]:
     """A decorator that uses `hydra_zen.builds` to create a dataclass with the appropriate
@@ -140,14 +140,16 @@ def hydrated_dataclass(
         Values specified in **kwargs_for_target take precedent over the corresponding
         default values from the signature.
 
-    hydra_partial : bool, optional (default=False)
+    hydra_partial : Optional[bool] (default=False)
         If True, then hydra-instantiation produces `functools.partial(target, **kwargs)`
 
     hydra_recursive : bool, optional (default=True)
         If True, then upon hydra will recursively instantiate all other
         hydra-config objects nested within this dataclass [2]_.
 
-    hydra_convert: Literal["none", "partial", "all"], optional (default="none")
+        If ``None``, the ``_recursive_`` attribute is not set on the resulting dataclass.
+
+    hydra_convert: Optional[Literal["none", "partial", "all"]] (default="none")
         Determines how hydra handles the non-primitive objects passed to `target` [3]_.
 
         - `"none"`: Passed objects are DictConfig and ListConfig, default
@@ -156,6 +158,8 @@ def hydrated_dataclass(
         - `"all"`: Passed objects are dicts, lists and primitives without
           a trace of OmegaConf containers
 
+        If ``None``, the ``_convert_`` attribute is not set on the resulting dataclass.
+
     frozen : bool, optional (default=False)
         If `True`, the resulting dataclass will create frozen (i.e. immutable) instances.
         I.e. setting/deleting an attribute of an instance will raise `FrozenInstanceError`
@@ -335,8 +339,8 @@ def builds(
     *pos_args: Any,
     populate_full_signature: bool = False,
     hydra_partial: Literal[False] = False,
-    hydra_recursive: bool = True,
-    hydra_convert: Literal["none", "partial", "all"] = "none",
+    hydra_recursive: Optional[bool] = None,
+    hydra_convert: Optional[Literal["none", "partial", "all"]] = None,
     dataclass_name: Optional[str] = None,
     builds_bases: Tuple[Any, ...] = (),
     frozen: bool = False,
@@ -352,8 +356,8 @@ def builds(
     *pos_args: Any,
     populate_full_signature: bool = False,
     hydra_partial: Literal[True],
-    hydra_recursive: bool = True,
-    hydra_convert: Literal["none", "partial", "all"] = "none",
+    hydra_recursive: Optional[bool] = None,
+    hydra_convert: Optional[Literal["none", "partial", "all"]] = None,
     dataclass_name: Optional[str] = None,
     builds_bases: Tuple[Any, ...] = (),
     frozen: bool = False,
@@ -369,8 +373,8 @@ def builds(
     *pos_args: Any,
     populate_full_signature: bool = False,
     hydra_partial: bool,
-    hydra_recursive: bool = True,
-    hydra_convert: Literal["none", "partial", "all"] = "none",
+    hydra_recursive: Optional[bool] = None,
+    hydra_convert: Optional[Literal["none", "partial", "all"]] = None,
     dataclass_name: Optional[str] = None,
     builds_bases: Tuple[Any, ...] = (),
     frozen: bool = False,
@@ -386,8 +390,8 @@ def builds(
     *pos_args: Any,
     populate_full_signature: bool = False,
     hydra_partial: bool = False,
-    hydra_recursive: bool = True,
-    hydra_convert: Literal["none", "partial", "all"] = "none",
+    hydra_recursive: Optional[bool] = None,
+    hydra_convert: Optional[Literal["none", "partial", "all"]] = None,
     frozen: bool = False,
     dataclass_name: Optional[str] = None,
     builds_bases: Tuple[Any, ...] = (),
@@ -434,11 +438,13 @@ def builds(
         user or that have default values specified in the target's signature. I.e. it is
         presumed that un-specified parameters are to be excluded from the partial configuration.
 
-    hydra_recursive : bool, optional (default=True)
-        If True, then upon hydra will recursively instantiate all other
+    hydra_recursive : Optional[bool], optional (default=True)
+        If ``True``, then upon hydra will recursively instantiate all other
         hydra-config objects nested within this dataclass [2]_.
 
-    hydra_convert: Literal["none", "partial", "all"], optional (default="none")
+        If ``None``, the ``_recursive_`` attribute is not set on the resulting dataclass.
+
+    hydra_convert: Optional[Literal["none", "partial", "all"]], optional (default="none")
         Determines how hydra handles the non-primitive objects passed to `target` [3]_.
 
         - `"none"`: Passed objects are DictConfig and ListConfig, default
@@ -447,6 +453,8 @@ def builds(
         - `"all"`: Passed objects are dicts, lists and primitives without
           a trace of OmegaConf containers
 
+        If ``None``, the ``_convert_`` attribute is not set on the resulting dataclass.
+
     frozen : bool, optional (default=False)
         If `True`, the resulting dataclass will create frozen (i.e. immutable) instances.
         I.e. setting/deleting an attribute of an instance will raise `FrozenInstanceError`
@@ -546,7 +554,7 @@ def builds(
             f"`populate_full_signature` must be a boolean type, got: {populate_full_signature}"
         )
 
-    if not isinstance(hydra_recursive, bool):
+    if hydra_recursive is not None and not isinstance(hydra_recursive, bool):
         raise TypeError(
             f"`hydra_recursive` must be a boolean type, got {hydra_recursive}"
         )
@@ -554,14 +562,7 @@ def builds(
     if not isinstance(hydra_partial, bool):
         raise TypeError(f"`hydra_partial` must be a boolean type, got: {hydra_partial}")
 
-    if not isinstance(hydra_convert, str):
-        raise TypeError(
-            f"`hydra_convert` must be 'none', 'partial', or 'all', got: {hydra_convert}"
-        )
-
-    hydra_convert = hydra_convert.lower()  # normalize casing
-
-    if hydra_convert not in {"none", "partial", "all"}:
+    if hydra_convert is not None and hydra_convert not in {"none", "partial", "all"}:
         raise ValueError(
             f"`hydra_convert` must be 'none', 'partial', or 'all', got: {hydra_convert}"
         )
@@ -598,16 +599,17 @@ def builds(
             )
         ]
 
-    # `base_fields` stores the list of fields that will be present in our dataclass
-    #
-    # TODO: should we always write these? Or is it more legible to only write them
-    #       if they were explicitly specified by the user?
-    #       - Presently we always need to write these, otherwise inheritance
-    #         becomes an issue (as it is with _partial_target_
-    base_fields: List[Tuple[str, type, Field_Entry]] = target_field + [
-        (_RECURSIVE_FIELD_NAME, bool, field(default=hydra_recursive, init=False)),
-        (_CONVERT_FIELD_NAME, str, field(default=hydra_convert, init=False)),
-    ]
+    base_fields: List[Tuple[str, type, Field_Entry]] = target_field
+
+    if hydra_recursive is not None:
+        base_fields.append(
+            (_RECURSIVE_FIELD_NAME, bool, field(default=hydra_recursive, init=False))
+        )
+
+    if hydra_convert is not None:
+        base_fields.append(
+            (_CONVERT_FIELD_NAME, str, field(default=hydra_convert, init=False))
+        )
 
     if pos_args:
         base_fields.append(
diff --git a/src/hydra_zen/typing.py b/src/hydra_zen/typing.py
--- a/src/hydra_zen/typing.py
+++ b/src/hydra_zen/typing.py
@@ -7,9 +7,6 @@
 from typing_extensions import Literal, Protocol, runtime_checkable
 
 __all__ = [
-    "Importable",
-    "DataClass",
-    "Instantiable",
     "Just",
     "Builds",
     "PartialBuilds",
@@ -47,22 +44,17 @@ class DataClass(Protocol):
     __dataclass_params__: _DataclassParams
 
 
-class Instantiable(DataClass, Protocol[_T]):  # pragma: no cover
+@runtime_checkable
+class Builds(DataClass, Protocol[_T]):  # pragma: no cover
     _target_: str
 
 
 @runtime_checkable
-class Just(Instantiable, Protocol[_T]):
+class Just(Builds, Protocol[_T]):
     path: str  # interpolated string for importing obj
     _target_: str = "hydra_utils.funcs.get_obj"
 
 
-@runtime_checkable
-class Builds(Instantiable, Protocol[_T]):  # pragma: no cover
-    _convert_: Literal["none", "partial", "all"]
-    _recursive_: bool
-
-
 @runtime_checkable
 class PartialBuilds(Builds, Protocol[_T]):
     _partial_target_: str


Test patch:
diff --git a/tests/test_dataclass_semantics.py b/tests/test_dataclass_semantics.py
--- a/tests/test_dataclass_semantics.py
+++ b/tests/test_dataclass_semantics.py
@@ -139,3 +139,42 @@ def test_mutable_defaults_generated_from_factory(mutable):
 
     out_inst = instantiate(instance2)["x"]
     assert out_inst == mutable
+
+
+class NotSet:
+    pass
+
+
+@pytest.mark.parametrize("recursive", [True, False, NotSet])
+@pytest.mark.parametrize("convert", ["none", "partial", "all", NotSet])
+@pytest.mark.parametrize("via_builds", [True, False])
+def test_hydra_settings_can_be_inherited(recursive, convert, via_builds):
+    kwargs = {}
+    if recursive is not NotSet:
+        kwargs["hydra_recursive"] = recursive
+
+    if convert is not NotSet:
+        kwargs["hydra_convert"] = convert
+
+    if via_builds:
+        Base = builds(dict, **kwargs)
+        Child = builds(dict, builds_bases=(Base,))
+    else:
+
+        @hydrated_dataclass(target=dict, **kwargs)
+        class Base:
+            pass
+
+        @hydrated_dataclass(target=dict)
+        class Child(Base):
+            pass
+
+    if recursive is not NotSet:
+        assert Child._recursive_ is Base._recursive_
+    else:
+        assert not hasattr(Child, "_recursive_")
+
+    if convert is not NotSet:
+        assert Child._convert_ is Base._convert_
+    else:
+        assert not hasattr(Child, "_convert_")
diff --git a/tests/test_hydra_behaviors.py b/tests/test_hydra_behaviors.py
--- a/tests/test_hydra_behaviors.py
+++ b/tests/test_hydra_behaviors.py
@@ -126,16 +126,18 @@ def f_for_recursive(**kwargs):
 
 
 @pytest.mark.parametrize("via_hydrated_dataclass", [False, True])
-@pytest.mark.parametrize("recursive", [False, True])
+@pytest.mark.parametrize("recursive", [False, True, NotSet])
 def test_recursive(recursive: bool, via_hydrated_dataclass: bool):
     target_path = get_obj_path(f_for_recursive)
 
+    kwargs = dict(hydra_recursive=recursive) if recursive is not NotSet else {}
+
     if not via_hydrated_dataclass:
         out = instantiate(
             builds(
                 f_for_recursive,
                 x=builds(f_for_recursive, y=1),
-                hydra_recursive=recursive,
+                **kwargs,
             )
         )
 
@@ -145,23 +147,16 @@ def test_recursive(recursive: bool, via_hydrated_dataclass: bool):
         class B:
             y: int = 1
 
-        @hydrated_dataclass(f_for_recursive, hydra_recursive=recursive)
+        @hydrated_dataclass(f_for_recursive, **kwargs)
         class A:
             x: Any = B
 
         out = instantiate(A)
 
-    if recursive:
+    if recursive is NotSet or recursive:
         assert out == {"x": {"y": 1}}
     else:
-        assert out == {
-            "x": {
-                "_target_": target_path,
-                "_recursive_": True,
-                "_convert_": "none",
-                "y": 1,
-            }
-        }
+        assert out == {"x": {"_target_": target_path, "y": 1}}
 
 
 def f(x: Tuple[int]):
diff --git a/tests/test_hydrated_dataclass.py b/tests/test_hydrated_dataclass.py
--- a/tests/test_hydrated_dataclass.py
+++ b/tests/test_hydrated_dataclass.py
@@ -5,7 +5,7 @@
 
 import pytest
 
-from hydra_zen import hydrated_dataclass, instantiate
+from hydra_zen import builds, hydrated_dataclass, instantiate
 
 
 def f1(x, y):
diff --git a/tests/test_validation.py b/tests/test_validation.py
--- a/tests/test_validation.py
+++ b/tests/test_validation.py
@@ -223,9 +223,9 @@ def test_builds_raises_on_non_callable_target(not_callable, partial, full_sig):
     "param_name, value",
     [
         ("populate_full_signature", None),
-        ("hydra_recursive", None),
-        ("hydra_partial", None),
-        ("hydra_convert", None),
+        ("hydra_recursive", 1),
+        ("hydra_partial", 1),
+        ("hydra_convert", 1),
         ("hydra_convert", "wrong value"),
         ("dataclass_name", 1),
         ("builds_bases", (Class,)),