Repository: iterative/dvc

Issue:
UTF-8 codec error while using dvc pull (surrogates not allowed)
# Bug Report

## Description
While using dvc pull, the task errors out with:
`ERROR: unexpected error - 'utf-8' codec can't encode characters in position 103-110: surrogates not allowed
`

### Reproduce
1. Try to push files with special characters in it. Ex: sydney-australia-â€“-january-â€“.txt
2. Set dvc cache type to reflink.
3. Try running dvc pull on the files that were pushed.

### Environment information


**Output of `dvc version`:**

```console
$ dvc version
DVC version: 1.10.2 (brew)
---------------------------------
Platform: Python 3.9.0 on macOS-10.15.7-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Caches: local
Remotes: s3
Repo: dvc, git
```

### Fix
By default on MacOS, the cache type used by DVC is reflink. By changing it to either symlink, hardlink or copy we can avoid the codec error. You can change the cache type this by using `dvc config cache.type <type>`

More on the cache types: https://dvc.org/doc/user-guide/large-dataset-optimization


```
2020-12-11 16:32:04,319 DEBUG: 'aug_set/annotations/sydney-australia-â€“-january-â€“-red-bull-energy-drink-mini-cooper-publicity-car-can-red-bull-drink-behind-used-123644712-0.json' doesn't exist.
2020-12-11 16:32:04,321 DEBUG: fetched: [(92173,)]
4:33
2020-12-11 16:32:04,325 ERROR: unexpected error - 'utf-8' codec can't encode characters in position 103-110: surrogates not allowed
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/main.py", line 90, in main
    ret = cmd.run()
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/command/data_sync.py", line 26, in run
    stats = self.repo.pull(
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/repo/__init__.py", line 60, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/repo/pull.py", line 36, in pull
    stats = self.checkout(
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/repo/__init__.py", line 60, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/repo/checkout.py", line 96, in checkout
    result = stage.checkout(
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/stage/decorators.py", line 36, in rwlocked
    return call()
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/stage/__init__.py", line 502, in checkout
    key, outs = self._checkout(out, **kwargs)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/stage/__init__.py", line 510, in _checkout
    result = out.checkout(**kwargs)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/output/base.py", line 356, in checkout
    return self.cache.checkout(
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/cache/base.py", line 40, in use_state
    return call()
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/cache/base.py", line 555, in checkout
    return self._checkout(
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/cache/base.py", line 578, in _checkout
    return self._checkout_dir(
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/cache/base.py", line 472, in _checkout_dir
    self.link(entry_cache_info, entry_info)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/cache/base.py", line 141, in link
    self._link(from_info, to_info, self.cache_types)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/cache/base.py", line 148, in _link
    self._try_links(from_info, to_info, link_types)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/remote/slow_link_detection.py", line 38, in wrapper
    result = f(remote, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/cache/base.py", line 166, in _try_links
    self._do_link(from_info, to_info, link_method)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/cache/base.py", line 182, in _do_link
    link_method(from_info, to_info)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/tree/local.py", line 240, in reflink
    System.reflink(from_info, tmp_info)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/system.py", line 108, in reflink
    ret = System._reflink_darwin(source, link_name)
  File "/usr/local/Cellar/dvc/1.10.2/libexec/lib/python3.9/site-packages/dvc/system.py", line 75, in _reflink_darwin
    ctypes.c_char_p(dst.encode("utf-8")),
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 103-110: surrogates not allowed
------------------------------------------------------------
2020-12-11 16:32:04,710 DEBUG: Version info for developers:
DVC version: 1.10.2 (brew)
---------------------------------
Platform: Python 3.9.0 on macOS-10.15.7-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Caches: local
Remotes: s3
Repo: dvc, git
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2020-12-11 16:32:04,712 DEBUG: Analytics is disabled.
```


Discussion:
@vss96 Thanks for great summary!

Our original discussion: https://discord.com/channels/485586884165107732/485596304961962003/786905389722501120

It seems like trying to encode the `dst` causes fail.

Resolving patch:
diff --git a/dvc/system.py b/dvc/system.py
--- a/dvc/system.py
+++ b/dvc/system.py
@@ -71,8 +71,8 @@ def _reflink_darwin(src, dst):
         clonefile.restype = ctypes.c_int
 
         return clonefile(
-            ctypes.c_char_p(src.encode("utf-8")),
-            ctypes.c_char_p(dst.encode("utf-8")),
+            ctypes.c_char_p(os.fsencode(src)),
+            ctypes.c_char_p(os.fsencode(dst)),
             ctypes.c_int(0),
         )
 


Test patch: