Repository: facebookresearch/hydra

Issue:
[Feature Request] Support _target_ to be a callable object itself in recursive construction
# 🚀 Feature Request

Now `_target_` has to be the full name of the callable object, supporting `_target_` to be a callable object itself in recursive construction would be very helpful.

## Motivation

In some situation there are some advantages using the callable directly:
* the name is often too long, such as `library.module_name.submodule_name.class`
* user writing a new class/function in code, and want to swap this class/function with the existing one in config.

## Pitch

In https://github.com/facebookresearch/hydra/blob/6ac5a6270530ff4acf76248a01e11c3cc4278847/examples/instantiate/object_recursive/my_app.py, be able to use the following instead:

```
class MyNewDriver()
...

@hydra.main(config_name="config")
def my_app(cfg: DictConfig) -> None:
    cfg = OmegaConf.to_container(cfg)
    cfg['car']['driver']['_target_'] = MyNewDriver
    car: Car = instantiate(cfg['car'])
    car.drive()
```


Discussion:
One thing that would work well with this is the ability to override the `_target_` similarly to the ability to override `_recursive_` in the #986. 

```python
car: Car = instantiate(cfg.car, driver={"_target_": MyNewDriver})
# or the equivalent:
car: Car = instantiate(cfg.car, **{"driver": {"_target_": MyNewDriver}})
```
I will look into supporting both features, although the callable name can be obtained via __name__:

```python
car: Car = instantiate(cfg.car, driver={"_target_": MyNewDriver.__name__})
```
One this that I am not sure about here is in what scenario this is preferred over config composition.
In your example, it's likely that MyNewDriver will require different parameters than Driver, and now your code will need to manually override multiple things.
The approach Hydra is taking to this problem is to use config groups and select the right option which will contain the correct config from the start - eliminating the need to manipulate it at program run time.

Do you have an example where building the config at runtime is somehow preferred to allowing Hydra to compose it?
keep in mind that for tests etc you can use the [Compose API](https://hydra.cc/docs/experimental/compose_api).
Upon an attempt to implement this I realized this is in conflict with an existing feature:
if a type is a dataclass:

```python
@dataclass
class Foo:
  x : float = 10
```
and you assign it to an OmegaConf container
```python
cfg = OmegaConf.create()
cfg.foo = Foo
```

The result is equivalent to assigning an object of that type:
```python
cfg = OmegaConf.create()
cfg.foo = Foo()
```

This is only true for dataclasses. however as they become more pervasive I am concerned that supporting this feature will cause confusion to people.
requiring a string solves that problem.

Obtaining a string you can use from a class or callable is straight forward and you don't need to type the whole thing:

```python
def get_full_name(type_or_callable : Any) -> str:
  return f"{type_or_callable.__module__}.{type_or_callable.__name__}"
```

I think supporting this feature will - in the long term - cause more problems than it's solving.

Resolving patch:
diff --git a/hydra/_internal/utils.py b/hydra/_internal/utils.py
--- a/hydra/_internal/utils.py
+++ b/hydra/_internal/utils.py
@@ -490,16 +490,6 @@ def get_column_widths(matrix: List[List[str]]) -> List[int]:
     return widths
 
 
-def _instantiate_or_call(
-    clazz: Any,
-    config: DictConfig,
-    *args: Any,
-    **kwargs: Any,
-) -> Any:
-    final_kwargs = _get_kwargs(config, **kwargs)
-    return clazz(*args, **final_kwargs)
-
-
 def _locate(path: str) -> Union[type, Callable[..., Any]]:
     """
     Locate an object by name or dotted path, importing as necessary.
@@ -575,6 +565,55 @@ def _is_rec(d: Any) -> Optional[bool]:
     return True
 
 
+def _convert_target_to_string(t: Any) -> Any:
+    if isinstance(t, type):
+        return f"{t.__module__}.{t.__name__}"
+    elif callable(t):
+        return f"{t.__module__}.{t.__qualname__}"
+    else:
+        return t
+
+
+def _convert_container_targets_to_strings(d: Any) -> None:
+    if isinstance(d, dict):
+        if "_target_" in d:
+            d["_target_"] = _convert_target_to_string(d["_target_"])
+        for k, v in d.items():
+            _convert_container_targets_to_strings(v)
+    elif isinstance(d, list):
+        for e in d:
+            if isinstance(e, (list, dict)):
+                _convert_container_targets_to_strings(e)
+
+
+def _get_target_type(config: Any, kwargs: Any) -> Union[type, Callable[..., Any]]:
+    kwargs_target = None
+    config_target = None
+    if "_target_" in kwargs:
+        kwargs_target = kwargs.pop("_target_")
+
+    if "_target_" in config:
+        config_target = config.pop("_target_")
+
+    target = None
+    if kwargs_target is not None:
+        target = kwargs_target
+    elif config_target is not None:
+        target = config_target
+
+    if target is None:
+        raise InstantiationException("Unable to determine target")
+
+    if isinstance(target, str):
+        return _locate(target)
+    elif isinstance(target, type):
+        return target
+    elif callable(target):
+        return target  # type: ignore
+    else:
+        raise InstantiationException(f"Unsupported target type : {type(target)}")
+
+
 def _get_kwargs(
     config: Union[DictConfig, ListConfig],
     **kwargs: Any,
@@ -596,8 +635,7 @@ def _get_kwargs(
     config.merge_with(overrides)
 
     for k, v in config.items():
-        if k not in ("_target_", "_recursive_"):
-            final_kwargs[k] = v
+        final_kwargs[k] = v
 
     if recursive:
         for k, v in final_kwargs.items():
@@ -631,7 +669,7 @@ def _get_kwargs(
     return final_kwargs
 
 
-def _get_cls_name(config: DictConfig, pop: bool = True) -> str:
+def _get_cls_name(config: Any, pop: bool = True) -> str:
     if "_target_" not in config:
         raise InstantiationException("Input config does not have a `_target_` field")
 
diff --git a/hydra/core/plugins.py b/hydra/core/plugins.py
--- a/hydra/core/plugins.py
+++ b/hydra/core/plugins.py
@@ -21,6 +21,7 @@
 from hydra.plugins.search_path_plugin import SearchPathPlugin
 from hydra.plugins.sweeper import Sweeper
 from hydra.types import TaskFunction
+from hydra.utils import instantiate
 
 
 @dataclass
@@ -86,7 +87,7 @@ def _instantiate(self, config: DictConfig) -> Plugin:
             if classname not in self.class_name_to_class.keys():
                 raise RuntimeError(f"Unknown plugin class : '{classname}'")
             clazz = self.class_name_to_class[classname]
-            plugin = internal_utils._instantiate_or_call(clazz=clazz, config=config)
+            plugin = instantiate(config=config, _target_=clazz)
             assert isinstance(plugin, Plugin)
 
         except ImportError as e:
diff --git a/hydra/utils.py b/hydra/utils.py
--- a/hydra/utils.py
+++ b/hydra/utils.py
@@ -7,7 +7,13 @@
 from omegaconf import OmegaConf
 from omegaconf._utils import is_structured_config
 
-from hydra._internal.utils import _get_cls_name, _instantiate_or_call, _locate
+from hydra._internal.utils import (
+    _convert_container_targets_to_strings,
+    _convert_target_to_string,
+    _get_kwargs,
+    _get_target_type,
+    _locate,
+)
 from hydra.core.hydra_config import HydraConfig
 from hydra.errors import HydraException, InstantiationException
 from hydra.types import TargetConf
@@ -49,28 +55,33 @@ def instantiate(config: Any, *args: Any, **kwargs: Any) -> Any:
     ):
         raise HydraException(f"Unsupported config type : {type(config).__name__}")
 
+    if isinstance(config, dict):
+        configc = config.copy()
+        _convert_container_targets_to_strings(configc)
+        config = configc
+
+    kwargsc = kwargs.copy()
+    _convert_container_targets_to_strings(kwargsc)
+    kwargs = kwargsc
+
     # make a copy to ensure we do not change the provided object
     config_copy = OmegaConf.structured(config, flags={"allow_objects": True})
     if OmegaConf.is_config(config):
         config_copy._set_parent(config._get_parent())
     config = config_copy
+    assert OmegaConf.is_config(config)
+    OmegaConf.set_readonly(config, False)
+    OmegaConf.set_struct(config, False)
 
-    cls = "<unknown>"
+    target = _get_target_type(config, kwargs)
     try:
-        assert OmegaConf.is_config(config)
-        OmegaConf.set_readonly(config, False)
-        OmegaConf.set_struct(config, False)
         config._set_flag("allow_objects", True)
-        cls = _get_cls_name(config)
-        type_or_callable = _locate(cls)
-        return _instantiate_or_call(
-            type_or_callable,
-            config,
-            *args,
-            **kwargs,
-        )
+        final_kwargs = _get_kwargs(config, **kwargs)
+        return target(*args, **final_kwargs)
     except Exception as e:
-        raise type(e)(f"Error instantiating/calling '{cls}' : {e}")
+        raise type(e)(
+            f"Error instantiating/calling '{_convert_target_to_string(target)}' : {e}"
+        )
 
 
 # Alias for instantiate


Test patch:
diff --git a/tests/__init__.py b/tests/__init__.py
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -38,6 +38,14 @@ def static_method(z: int) -> int:
         return z
 
 
+@dataclass
+class BClass:
+    a: Any
+    b: Any
+    c: Any = "c"
+    d: Any = "d"
+
+
 @dataclass
 class UntypedPassthroughConf:
     _target_: str = "tests.UntypedPassthroughClass"
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -9,6 +9,7 @@
 from omegaconf import DictConfig, OmegaConf
 
 from hydra import utils
+from hydra._internal.utils import _convert_container_targets_to_strings
 from hydra.conf import HydraConf, RuntimeConf
 from hydra.core.hydra_config import HydraConfig
 from hydra.errors import InstantiationException
@@ -20,6 +21,7 @@
     AnotherClass,
     ASubclass,
     BadAdamConf,
+    BClass,
     CenterCrop,
     CenterCropConf,
     Compose,
@@ -35,6 +37,7 @@
     TreeConf,
     UntypedPassthroughClass,
     UntypedPassthroughConf,
+    module_function,
 )
 
 
@@ -665,3 +668,119 @@ def test_recursive_override(
 ) -> None:
     obj = utils.instantiate(cfg, **passthrough)
     assert obj == expected
+
+
+@pytest.mark.parametrize(  # type: ignore
+    "input_conf, passthrough, expected",
+    [
+        pytest.param(
+            {"_target_": "tests.AClass", "a": 10, "b": 20, "c": 30, "d": 40},
+            {"_target_": "tests.BClass"},
+            BClass(10, 20, 30, 40),
+            id="str:override_same_args",
+        ),
+        pytest.param(
+            {"_target_": "tests.AClass", "a": 10, "b": 20, "c": 30, "d": 40},
+            {"_target_": BClass},
+            BClass(10, 20, 30, 40),
+            id="type:override_same_args",
+        ),
+        pytest.param(
+            {"_target_": "tests.AClass", "a": 10, "b": 20},
+            {"_target_": "tests.BClass"},
+            BClass(10, 20, "c", "d"),
+            id="str:override_other_args",
+        ),
+        pytest.param(
+            {"_target_": "tests.AClass", "a": 10, "b": 20},
+            {"_target_": BClass},
+            BClass(10, 20, "c", "d"),
+            id="type:override_other_args",
+        ),
+        pytest.param(
+            {
+                "_target_": "tests.AClass",
+                "a": 10,
+                "b": 20,
+                "c": {
+                    "_target_": "tests.AClass",
+                    "a": "aa",
+                    "b": "bb",
+                    "c": "cc",
+                },
+            },
+            {
+                "_target_": "tests.BClass",
+                "c": {
+                    "_target_": "tests.BClass",
+                },
+            },
+            BClass(10, 20, BClass(a="aa", b="bb", c="cc"), "d"),
+            id="str:recursive_override",
+        ),
+        pytest.param(
+            {
+                "_target_": "tests.AClass",
+                "a": 10,
+                "b": 20,
+                "c": {
+                    "_target_": "tests.AClass",
+                    "a": "aa",
+                    "b": "bb",
+                    "c": "cc",
+                },
+            },
+            {"_target_": BClass, "c": {"_target_": BClass}},
+            BClass(10, 20, BClass(a="aa", b="bb", c="cc"), "d"),
+            id="type:recursive_override",
+        ),
+    ],
+)
+def test_override_target(input_conf: Any, passthrough: Any, expected: Any) -> None:
+    conf_copy = copy.deepcopy(input_conf)
+    obj = utils.instantiate(input_conf, **passthrough)
+    assert obj == expected
+    # make sure config is not modified by instantiate
+    assert input_conf == conf_copy
+
+
+@pytest.mark.parametrize(  # type: ignore
+    "input_, expected",
+    [
+        pytest.param({"a": 10}, {"a": 10}),
+        pytest.param([1, 2, 3], [1, 2, 3]),
+        pytest.param({"_target_": "abc", "a": 10}, {"_target_": "abc", "a": 10}),
+        pytest.param(
+            {"_target_": AClass, "a": 10}, {"_target_": "tests.AClass", "a": 10}
+        ),
+        pytest.param(
+            {"_target_": AClass.static_method, "a": 10},
+            {"_target_": "tests.AClass.static_method", "a": 10},
+        ),
+        pytest.param(
+            {"_target_": ASubclass.class_method, "a": 10},
+            {"_target_": "tests.ASubclass.class_method", "a": 10},
+        ),
+        pytest.param(
+            {"_target_": module_function, "a": 10},
+            {"_target_": "tests.module_function", "a": 10},
+        ),
+        pytest.param(
+            {
+                "_target_": AClass,
+                "a": {"_target_": AClass, "b": 10},
+            },
+            {
+                "_target_": "tests.AClass",
+                "a": {"_target_": "tests.AClass", "b": 10},
+            },
+        ),
+        pytest.param(
+            [1, {"_target_": AClass, "a": 10}],
+            [1, {"_target_": "tests.AClass", "a": 10}],
+        ),
+    ],
+)
+def test_convert_target_to_string(input_: Any, expected: Any) -> None:
+    _convert_container_targets_to_strings(input_)
+    assert input_ == expected