Repository: facebookresearch/hydra

Issue:
[Bug] Error when using config with UTF characters
# 🐛 Bug
## Description

An exception is thrown when using Unicode chars in the config.


## Workaround
In `hydra/core/utils.py`:
```
def _save_config(cfg: DictConfig, filename: str, output_dir: Path) -> None:
    output_dir.mkdir(parents=True, exist_ok=True)
    yaml = OmegaConf.to_yaml(cfg).encode("utf8")
    with open(str(output_dir / filename), "wb") as file:
        file.write(yaml)
```

## Checklist
- [x] I checked on the latest version of Hydra
- [x] I created a minimal repro (See [this](https://stackoverflow.com/help/minimal-reproducible-example) for tips).

## To reproduce
Use a `config.yaml` with unicode chars, and make Hydra store it in `outputs/`.

## Minimal example
Use the tutorial code:
```
from omegaconf import DictConfig, OmegaConf
import hydra

@hydra.main(config_name='config')
def my_app(cfg: DictConfig) -> None:
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    my_app()
```

Along with a config that has Unicode chars:
`config.yaml`:
```
a: א
```



** Stack trace/error message **
```
  File "D:\proj\video-pipeline\venv38\lib\site-packages\hydra\main.py", line 32, in decorated_main
    _run_hydra(
  File "D:\proj\video-pipeline\venv38\lib\site-packages\hydra\_internal\utils.py", line 346, in _run_hydra
    run_and_report(
  File "D:\proj\video-pipeline\venv38\lib\site-packages\hydra\_internal\utils.py", line 201, in run_and_report
    raise ex
  File "D:\proj\video-pipeline\venv38\lib\site-packages\hydra\_internal\utils.py", line 198, in run_and_report
    return func()
  File "D:\proj\video-pipeline\venv38\lib\site-packages\hydra\_internal\utils.py", line 347, in <lambda>
    lambda: hydra.run(
  File "D:\proj\video-pipeline\venv38\lib\site-packages\hydra\_internal\hydra.py", line 107, in run
    return run_job(
  File "D:\proj\video-pipeline\venv38\lib\site-packages\hydra\core\utils.py", line 122, in run_job
    _save_config(task_cfg, "config.yaml", hydra_output)
  File "D:\proj\video-pipeline\venv38\lib\site-packages\hydra\core\utils.py", line 70, in _save_config
    file.write(OmegaConf.to_yaml(cfg))
  File "C:\Users\micha\AppData\Local\Programs\Python\Python38\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 267-271: character maps to <undefined>

```

## Expected Behavior
<!-- A clear and concise description of what you expected to happen. -->

## System information
- **Hydra Version** :  1.0.6
- **Python version** : 3.8.10
- **Virtual environment type and version** : pip
- **Operating system** : Windows 10

## Additional context
Add any other context about the problem here.




Discussion:
 hi @michaelitvin 
thanks for the reports. I do not have a windows machine and haven't been able to repro this on my MAC. 

What exact version of Hydra 1.0 are you on? we did fix a unicode in config file related bug in a later Hydra 1.0 release.  Could you try to upgrade to the latest Hydra 1.0 and see if the issue goes away?

```
pip install hydra-core --upgrade
```
I'm using the latest from pip - 1.0.6.
The exception does make sense, as when the config is written to file, an encoding isn't specified. So a default is used, and on Windows that's cp1252.
> I'm using the latest from pip - 1.0.6.
> The exception does make sense, as when the config is written to file, an encoding isn't specified. So a default is used, and on Windows that's cp1252.

I see! I will look into this. Thanks for reporting.


Resolving patch:
diff --git a/hydra/core/utils.py b/hydra/core/utils.py
--- a/hydra/core/utils.py
+++ b/hydra/core/utils.py
@@ -73,7 +73,7 @@ def configure_log(
 
 def _save_config(cfg: DictConfig, filename: str, output_dir: Path) -> None:
     output_dir.mkdir(parents=True, exist_ok=True)
-    with open(str(output_dir / filename), "w") as file:
+    with open(str(output_dir / filename), "w", encoding="utf-8") as file:
         file.write(OmegaConf.to_yaml(cfg))
 
 


Test patch:
diff --git a/tests/test_apps/app_with_unicode_in_config/__init__.py b/tests/test_apps/app_with_unicode_in_config/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_unicode_in_config/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
diff --git a/tests/test_apps/app_with_unicode_in_config/config.yaml b/tests/test_apps/app_with_unicode_in_config/config.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_unicode_in_config/config.yaml
@@ -0,0 +1 @@
+config: 数据库
diff --git a/tests/test_apps/app_with_unicode_in_config/my_app.py b/tests/test_apps/app_with_unicode_in_config/my_app.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_with_unicode_in_config/my_app.py
@@ -0,0 +1,13 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
+from omegaconf import DictConfig, OmegaConf
+
+import hydra
+
+
+@hydra.main(config_path=".", config_name="config")
+def my_app(cfg: DictConfig) -> None:
+    print(OmegaConf.to_yaml(cfg))
+
+
+if __name__ == "__main__":
+    my_app()
diff --git a/tests/test_hydra.py b/tests/test_hydra.py
--- a/tests/test_hydra.py
+++ b/tests/test_hydra.py
@@ -1368,6 +1368,15 @@ def test_hydra_main_without_config_path(tmpdir: Path) -> None:
     )
 
 
+def test_app_with_unicode_config(tmpdir: Path) -> None:
+    cmd = [
+        "tests/test_apps/app_with_unicode_in_config/my_app.py",
+        f"hydra.run.dir={tmpdir}",
+    ]
+    out, _ = run_python_script(cmd)
+    assert out == "config: 数据库"
+
+
 @mark.parametrize(
     "overrides,expected",
     [