Repository: iterative/dvc

Issue:
Not supported reflink in APFS
```
$ dvc -V
0.23.2+bad2ef.mod
```

DVC creates hardlinks instead of reflinks in APFS.

File system:
```
$ diskutil info / | grep -i system
   File System Personality:   APFS
```

See last two lines:
```
$ dvc add Tags.xml -v
Debug: PRAGMA user_version;
Debug: fetched: [(3,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 3;
Debug: Skipping copying for '/Users/dmitry/src/modules-example/tmp/test/Tags.xml', since it is not a symlink or a hardlink.
Adding 'Tags.xml' to '.gitignore'.
Saving 'Tags.xml' to cache '.dvc/cache'.
Debug: Path /Users/dmitry/src/modules-example/tmp/test/Tags.xml inode 12888021464
Debug: SELECT * from state WHERE inode=12888021464
Debug: fetched: []
Debug: INSERT INTO state(inode, mtime, size, md5, timestamp) VALUES (12888021464, "1547854167848187904", "0", "6d861a1605e60b2f3a977a5a2a4419a2", "1547854178609126912")
Debug: File '/Users/dmitry/src/modules-example/tmp/test/.dvc/cache/6d/861a1605e60b2f3a977a5a2a4419a2', md5 '6d861a1605e60b2f3a977a5a2a4419a2', actual 'None'
Debug: Cache type 'reflink' is not supported: reflink is not supported
Debug: Created 'hardlink': /Users/dmitry/src/modules-example/tmp/test/.dvc/cache/6d/861a1605e60b2f3a977a5a2a4419a2 -> /Users/dmitry/src/modules-example/tmp/test/Tags.xml
```


Discussion:
Unable to confirm this. Reflinks are working for me on both bad2ef and the current master. I see you are working on a modified repo, could you try it out without your changes? Also note that you are working on an older version that has a bunch of bugs that were fixed already in the master, so you might want to upgrade as well.
Updated:
```
$ dvc -V
0.23.2+d76860.mod
```

The result is even worse:
```
$ dvc add Tags2.xml -v
Debug: PRAGMA user_version;
Debug: fetched: [(3,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 3;
Debug: Skipping copying for '/Users/dmitry/src/modules-example/tmp/test/Tags2.xml', since it is not a symlink or a hardlink.
Debug: checking if '{'scheme': 'local', 'path': '/Users/dmitry/src/modules-example/tmp/test/Tags2.xml'}'('{}') has changed.
Debug: checksum for '{'scheme': 'local', 'path': '/Users/dmitry/src/modules-example/tmp/test/Tags2.xml'}' is missing.
Adding 'Tags2.xml' to '.gitignore'.
Saving 'Tags2.xml' to cache '.dvc/cache'.
Debug: Path /Users/dmitry/src/modules-example/tmp/test/Tags2.xml inode 12888044947
Debug: SELECT * from state WHERE inode=12888044947
Debug: fetched: []
Debug: INSERT INTO state(inode, mtime, size, md5, timestamp) VALUES (12888044947, "1547898562303480320", "4225330", "6d861a1605e60b2f3a977a5a2a4419a2", "1547898694894590976")
Debug: Path /Users/dmitry/src/modules-example/tmp/test/.dvc/cache/6d/861a1605e60b2f3a977a5a2a4419a2 inode 12888021464
Debug: SELECT * from state WHERE inode=12888021464
Debug: fetched: [(12888021464, '1547854167848187904', '0', '6d861a1605e60b2f3a977a5a2a4419a2', '1547867685273969920')]
Debug: Inode '12888021464', mtime '1547854167848187904', actual mtime '1547854167848187904', size '0', actual size '4225330'.
Debug: UPDATE state SET mtime = "1547854167848187904", size = "4225330", md5 = "6d861a1605e60b2f3a977a5a2a4419a2", timestamp = "1547898694913789952" WHERE inode = 12888021464
Debug: File '/Users/dmitry/src/modules-example/tmp/test/.dvc/cache/6d/861a1605e60b2f3a977a5a2a4419a2', md5 '6d861a1605e60b2f3a977a5a2a4419a2', actual '6d861a1605e60b2f3a977a5a2a4419a2'
Debug: Removing 'Tags2.xml'
Debug: Cache type 'reflink' is not supported: reflink is not supported
Debug: Created 'hardlink': /Users/dmitry/src/modules-example/tmp/test/.dvc/cache/6d/861a1605e60b2f3a977a5a2a4419a2 -> /Users/dmitry/src/modules-example/tmp/test/Tags2.xml
Debug: Path /Users/dmitry/src/modules-example/tmp/test/Tags2.xml inode 12888021464
Debug: REPLACE INTO link_state(path, inode, mtime) VALUES ("Tags2.xml", 12888021464, "1547854167848187904")
Saving information to 'Tags2.xml.dvc'.
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(3,)]
Debug: UPDATE state_info SET count = 4 WHERE rowid = 1

To track the changes with git run:

	git add .gitignore Tags2.xml.dvc
Debug: Analytics is enabled.
Debug: Trying to spawn '['/usr/local/opt/python/bin/python3.6', '-m', 'dvc', 'daemon', '-q', 'analytics', '/var/folders/gp/f6lcbh3n5mggb0rm0k17yc3w0000gn/T/tmp43fepznp']'
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/dmitry/src/dvc/dvc/__main__.py", line 8, in <module>
    main(sys.argv[1:])
  File "/Users/dmitry/src/dvc/dvc/main.py", line 43, in main
    Analytics().send_cmd(cmd, args, ret)
  File "/Users/dmitry/src/dvc/dvc/analytics.py", line 241, in send_cmd
    daemon(['analytics', analytics.dump()])
  File "/Users/dmitry/src/dvc/dvc/daemon.py", line 80, in daemon
    _spawn_posix(cmd)
TypeError: 'staticmethod' object is not callable
```

Still hadrlinks.
Isn't it:
```TypeError: 'staticmethod' object is not callable```?
Yep, it is. Thanks @pared for extracting https://github.com/iterative/dvc/pull/1517 as a separate PR :) I'll merge it ASAP.

@dmpetrov Hm, interesting. Let me create another script for you to try out...
@dmpetrov #1517 is merged, please pull latest changes.

And could you please try running this script from the root of your dvc repo and show me the output(if any)?

```
$ cat test.py

import os

from dvc.system import System

FOO = 'foo'
BAR = 'bar'

if os.path.exists(FOO):
    os.unlink(FOO)

if os.path.exists(BAR):
    os.unlink(BAR)

with open(FOO, 'w+') as fobj:
    fobj.write(FOO)

ret = System._reflink_darwin(FOO, BAR)
assert ret == 0
assert os.path.isfile(FOO)
assert os.path.isfile(BAR)

$ python test.py
```
```
ret = System._reflink_darwin(FOO, BAR)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dmitry/src/dvc/dvc/system.py", line 59, in _reflink_darwin
    clib = ctypes.CDLL('libc.dylib')
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(libc.dylib, 6): image not found
```

It looks like an environment issue. So, it looks like a not a bug. Let's close if no other concerns.
However, the issue shows that having FS with reflink support does not guaranty reflinks.
@dmpetrov Hm, interesting. And could you please show me `otool -L $(which python)` output?
I'm running with python3:
```
$ otool -L /usr/local/bin/python3
/usr/local/bin/python3:
	/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/Python (compatibility version 3.6.0, current version 3.6.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
```
@dmpetrov and `ls -la /usr/lib/libSystem.B.dylib` please?
```
$ ls -la /usr/lib/libSystem.B.dylib
-rwxr-xr-x  1 root  wheel    59K Oct 24 03:47 /usr/lib/libSystem.B.dylib*
```
@dmpetrov I see that you are running dvc right from the source. Could show me your `which dvc` output as well as `which python`, please?
Closing as an env issue.
Reopenning. This is an issue with SIP on mac, we might be able to workaround it by using a direct path to libc. https://discordapp.com/channels/485586884165107732/485596304961962003/557499161671368706 

Resolving patch:
diff --git a/dvc/system.py b/dvc/system.py
--- a/dvc/system.py
+++ b/dvc/system.py
@@ -3,6 +3,7 @@
 from dvc.utils.compat import str, open
 
 import os
+import errno
 
 
 class System(object):
@@ -61,8 +62,22 @@ def symlink(source, link_name):
     @staticmethod
     def _reflink_darwin(src, dst):
         import ctypes
+        import dvc.logger as logger
+
+        LIBC = "libc.dylib"
+        LIBC_FALLBACK = "/usr/lib/libSystem.dylib"
+        try:
+            clib = ctypes.CDLL(LIBC)
+        except OSError as exc:
+            logger.debug(
+                "unable to access '{}' (errno '{}'). "
+                "Falling back to '{}'.".format(LIBC, exc.errno, LIBC_FALLBACK)
+            )
+            if exc.errno != errno.ENOENT:
+                raise
+            # NOTE: trying to bypass System Integrity Protection (SIP)
+            clib = ctypes.CDLL(LIBC_FALLBACK)
 
-        clib = ctypes.CDLL("libc.dylib")
         if not hasattr(clib, "clonefile"):
             return -1
 


Test patch: