Repository: dandi/dandi-cli

Issue:
play nicely with datalad datasets: do not download what is already present
It seems that download of files from S3 directly using `datalad get` results in files just to have current date/time as mtime.  git-annex version locally is 8.20210223-1~ndall+1 .  And I thought that in general git-annex does preserve mtime as known to remote and we uploaded that key awhile back.  

ATM `dandi-cli` first relies on mtime to decide to either redownload/overwrite the file since default mode is `--existing=refresh` (well - may be not that good of an idea). So I think we should

- ~~check if it is git-annex or our (using URLs from S3) issue which results in files with current date time~~
- [ ] make default `--existing=error` or add `--existing=ask` (should be made to play nicely with pyout). Related: non-interactive confirmations #620 
- [ ] become smart about git-annex: ideally ask datalad about key (could be unlocked, or adjusted branch if on windows) and deduce checksum from key.  If used backend one (we use sha256 ATM) present as digest in asset metadata - we could state "identity" the same and avoid 'refresh' altogether

<details>
<summary>full demo:</summary> 

```shell
(git)lena:/tmp[master]git-annex
$> datalad install -g ///dandi/dandisets/000027
[INFO   ] Scanning for unlocked files (this may take some time)                                                                                                                                                      
[INFO   ] access to 1 dataset sibling dandi-dandisets-dropbox not auto-enabled, enable with:                                                                                                                         
| 		datalad siblings -d "/tmp/000027" enable -s dandi-dandisets-dropbox 
install(ok): /tmp/000027 (dataset)
get(ok): /tmp/000027/sub-RAT123/sub-RAT123.nwb (file) [from web...]                                                                                                                                                  
action summary:
  get (ok: 1)
  install (ok: 1)
(dev3) (dandi-py3.9-troubleshoot) 1 79290.....................................:Fri 07 May 2021 05:54:00 PM EDT:.
(git)lena:/tmp[master]git-annex
$> stat -L 000027/sub-RAT123/sub-RAT123.nwb 
  File: 000027/sub-RAT123/sub-RAT123.nwb
  Size: 18792     	Blocks: 40         IO Block: 4096   regular file
Device: 10305h/66309d	Inode: 16400896    Links: 1
Access: (0400/-r--------)  Uid: (47521/     yoh)   Gid: (47522/     yoh)
Access: 2021-05-07 17:54:00.792805280 -0400
Modify: 2021-05-07 17:54:00.788805286 -0400
Change: 2021-05-07 17:54:00.792805280 -0400
 Birth: 2021-05-07 17:54:00.744805350 -0400
(dev3) (dandi-py3.9-troubleshoot) 1 79291.....................................:Fri 07 May 2021 05:54:10 PM EDT:.
(git)lena:/tmp[master]git-annex
$> ls -l -L 000027/sub-RAT123/sub-RAT123.nwb
-r-------- 1 yoh yoh 18792 May  7 17:54 000027/sub-RAT123/sub-RAT123.nwb
(dev3) (dandi-py3.9-troubleshoot) 1 79292.....................................:Fri 07 May 2021 05:54:19 PM EDT:.
(git)lena:/tmp[master]git-annex
$> dandi download --sync dandi://dandi/000027
PATH                      SIZE     DONE    DONE% CHECKSUM STATUS          MESSAGE         
dandiset.yaml                                             skipped         already exists  
sub-RAT123/sub-RAT123.nwb 18.8 kB  18.8 kB  100%    ok    done                            
Summary:                  18.8 kB  18.8 kB                1 skipped       1 already exists
                                   100.00%                1 done                          
2021-05-07 17:54:49,463 [    INFO] Logs saved in /home/yoh/.cache/dandi-cli/log/20210507215446Z-3813791.log
(dev3) (dandi-py3.9-troubleshoot) 1 79293.....................................:Fri 07 May 2021 05:54:49 PM EDT:.
(git)lena:/tmp[master]git-annex
$> dandi download --sync dandi://dandi/000027
PATH                      SIZE     DONE    DONE% CHECKSUM STATUS    MESSAGE             
dandiset.yaml                                             skipped   already exists      
sub-RAT123/sub-RAT123.nwb                                 skipped   same time and size  
Summary:                  0 Bytes  0 Bytes                2 skipped 1 already exists    
                          +18.8 kB 0.00%                            1 same time and size
2021-05-07 17:54:54,369 [    INFO] Logs saved in /home/yoh/.cache/dandi-cli/log/20210507215452Z-3813835.log
(dev3) (dandi-py3.9-troubleshoot) 1 79293.....................................:Fri 07 May 2021 05:54:54 PM EDT:.
(git)lena:/tmp[master]git-annex
$> ls -l -L 000027/sub-RAT123/sub-RAT123.nwb 
-rw------- 1 yoh yoh 18792 Oct 21  2020 000027/sub-RAT123/sub-RAT123.nwb

```
</details>

Comments:

Patch:
diff --git a/.autorc b/.autorc
index 0c6c3889..9869dabc 100644
--- a/.autorc
+++ b/.autorc
@@ -1,7 +1,7 @@
 {
     "onlyPublishWithReleaseLabel": true,
     "baseBranch": "master",
-    "author": "auto <auto@nil>",
+    "author": "DANDI Bot <team@dandiarchive.org>",
     "noVersionPrefix": true,
     "plugins": [
         "git-tag",
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3d468b7e..e2accd65 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,32 @@
+# 0.18.0 (Wed May 26 2021)
+
+#### 🚀 Enhancement
+
+- Move schema code to dandischema [#643](https://github.com/dandi/dandi-cli/pull/643) ([@jwodder](https://github.com/jwodder))
+- Add "shell-completion" command [#640](https://github.com/dandi/dandi-cli/pull/640) ([@jwodder](https://github.com/jwodder))
+- REF: updated model requirements [#623](https://github.com/dandi/dandi-cli/pull/623) ([@satra](https://github.com/satra) [@yarikoptic](https://github.com/yarikoptic))
+
+#### 🐛 Bug Fix
+
+- Pin dandischema to compatible 0.minor version [#650](https://github.com/dandi/dandi-cli/pull/650) ([@yarikoptic](https://github.com/yarikoptic))
+- Warn on ignored symlinks to directories [#647](https://github.com/dandi/dandi-cli/pull/647) ([@jwodder](https://github.com/jwodder))
+- Delete name2title() [#645](https://github.com/dandi/dandi-cli/pull/645) ([@jwodder](https://github.com/jwodder))
+- adding to_datacite method [#596](https://github.com/dandi/dandi-cli/pull/596) ([@djarecka](https://github.com/djarecka) [@yarikoptic](https://github.com/yarikoptic))
+- Datacite tmp [#595](https://github.com/dandi/dandi-cli/pull/595) ([@djarecka](https://github.com/djarecka))
+
+#### 🏠 Internal
+
+- Include CHANGELOG.md and tox.ini in sdists [#648](https://github.com/dandi/dandi-cli/pull/648) ([@jwodder](https://github.com/jwodder))
+
+#### Authors: 4
+
+- Dorota Jarecka ([@djarecka](https://github.com/djarecka))
+- John T. Wodder II ([@jwodder](https://github.com/jwodder))
+- Satrajit Ghosh ([@satra](https://github.com/satra))
+- Yaroslav Halchenko ([@yarikoptic](https://github.com/yarikoptic))
+
+---
+
 # 0.17.0 (Wed May 12 2021)
 
 #### 🚀 Enhancement
diff --git a/MANIFEST.in b/MANIFEST.in
index 553a45fa..f6a9c1bf 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,6 +1,7 @@
 # This line includes versioneer.py in sdists, which is necessary for wheels
 # built from sdists to have the version set in their metadata.
 include versioneer.py
+include CHANGELOG.md tox.ini
 
 graft dandi
 
diff --git a/dandi/cli/cmd_download.py b/dandi/cli/cmd_download.py
index 165994b5..71a3daa7 100644
--- a/dandi/cli/cmd_download.py
+++ b/dandi/cli/cmd_download.py
@@ -43,12 +43,12 @@ class ChoiceList(click.ParamType):
     "-e",
     "--existing",
     type=click.Choice(
-        ["error", "skip", "overwrite", "refresh"]
+        ["error", "skip", "overwrite", "overwrite-different", "refresh"]
     ),  # TODO: verify-reupload (to become default)
     help="What to do if a file found existing locally. 'refresh': verify "
     "that according to the size and mtime, it is the same file, if not - "
     "download and overwrite.",
-    default="refresh",
+    default="error",
     show_default=True,
 )
 @click.option(
diff --git a/dandi/download.py b/dandi/download.py
index d0f8abf3..aca81dba 100644
--- a/dandi/download.py
+++ b/dandi/download.py
@@ -15,6 +15,7 @@ from .consts import dandiset_metadata_file
 from .dandiarchive import DandisetURL, MultiAssetURL, SingleAssetURL, parse_dandi_url
 from .dandiset import Dandiset
 from . import get_logger
+from .support.digests import get_digest
 from .support.pyout import naturalsize
 from .utils import (
     abbrev_prompt,
@@ -23,6 +24,7 @@ from .utils import (
     flattened,
     is_same_time,
     on_windows,
+    path_is_subpath,
     pluralize,
 )
 
@@ -207,6 +209,10 @@ def download_generator(
                 raise RuntimeError(
                     f"dandi-etag not available for asset. Known digests: {d}"
                 )
+            try:
+                digests["sha256"] = d["dandi:sha2-256"]
+            except KeyError:
+                pass
 
             path = asset["path"].lstrip("/")  # make into relative path
             path = op.normpath(path)
@@ -240,6 +246,7 @@ def download_generator(
             _download_generator = _download_file(
                 downloader,
                 download_path,
+                toplevel_path=output_path,
                 # size and modified generally should be there but better to redownload
                 # than to crash
                 size=asset.get("size"),
@@ -392,6 +399,10 @@ def _populate_dandiset_yaml(dandiset_path, dandiset_data, existing):
     if op.lexists(dandiset_yaml):
         if existing == "error":
             raise FileExistsError(dandiset_yaml)
+        elif existing == "refresh" and op.lexists(
+            op.join(dandiset_path, ".git", "annex")
+        ):
+            raise RuntimeError("Not refreshing path in git annex repository")
         elif existing == "skip" or (
             existing == "refresh"
             and os.lstat(dandiset_yaml).st_mtime >= mtime.timestamp()
@@ -410,7 +421,13 @@ def _populate_dandiset_yaml(dandiset_path, dandiset_data, existing):
 
 
 def _download_file(
-    downloader, path, size=None, mtime=None, existing="error", digests=None
+    downloader,
+    path,
+    toplevel_path,
+    size=None,
+    mtime=None,
+    existing="error",
+    digests=None,
 ):
     """Common logic for downloading a single file
 
@@ -434,6 +451,7 @@ def _download_file(
     """
     if op.lexists(path):
         block = f"File {path!r} already exists"
+        annex_path = op.join(toplevel_path, ".git", "annex")
         if existing == "error":
             raise FileExistsError(block)
         elif existing == "skip":
@@ -441,7 +459,39 @@ def _download_file(
             return
         elif existing == "overwrite":
             pass
+        elif existing == "overwrite-different":
+            realpath = op.realpath(path)
+            key_parts = op.basename(realpath).split("-")
+            if size is not None and os.stat(realpath).st_size != size:
+                lgr.debug(
+                    "Size of %s does not match size on server; redownloading", path
+                )
+            elif (
+                op.lexists(annex_path)
+                and op.islink(path)
+                and path_is_subpath(realpath, op.abspath(annex_path))
+                and key_parts[0] == "SHA256E"
+                and digests
+                and "sha256" in digests
+            ):
+                if key_parts[-1].partition(".")[0] == digests["sha256"]:
+                    yield _skip_file("already exists")
+                    return
+                else:
+                    lgr.debug(
+                        "%s is in git-annex, and hash does not match hash on server; redownloading",
+                        path,
+                    )
+            elif get_digest(path, "dandi-etag") == digests["dandi-etag"]:
+                yield _skip_file("already exists")
+                return
+            else:
+                lgr.debug(
+                    "Etag of %s does not match etag on server; redownloading", path
+                )
         elif existing == "refresh":
+            if op.lexists(annex_path):
+                raise RuntimeError("Not refreshing path in git annex repository")
             if mtime is None:
                 lgr.warning(
                     f"{path!r} - no mtime or ctime in the record, redownloading"
@@ -487,6 +537,7 @@ def _download_file(
 
     # TODO: how do we discover the total size????
     # TODO: do not do it in-place, but rather into some "hidden" file
+    resuming = False
     for attempt in range(3):
         try:
             if digester:
@@ -495,6 +546,7 @@ def _download_file(
             # I wonder if we could make writing async with downloader
             with DownloadDirectory(path, digests) as dldir:
                 downloaded = dldir.offset
+                resuming = downloaded > 0
                 if size is not None and downloaded == size:
                     # Exit early when downloaded == size, as making a Range
                     # request in such a case results in a 416 error from S3.
@@ -542,7 +594,7 @@ def _download_file(
             )
             time.sleep(random.random() * 5)
 
-    if downloaded_digest:
+    if downloaded_digest and not resuming:
         downloaded_digest = downloaded_digest.hexdigest()  # we care only about hex
         if digest != downloaded_digest:
             msg = f"{algo}: downloaded {downloaded_digest} != {digest}"
diff --git a/dandi/utils.py b/dandi/utils.py
index aa20c4a1..03b016a7 100644
--- a/dandi/utils.py
+++ b/dandi/utils.py
@@ -350,7 +350,12 @@ def find_files(
         paths = (op.join(dirpath, name) for name in names)
         for path in filter(re.compile(regex).search, paths):
             if not exclude_path(path):
-                yield path
+                if op.islink(path) and op.isdir(path):
+                    lgr.warning(
+                        "%s: Ignoring unsupported symbolic link to directory", path
+                    )
+                else:
+                    yield path
 
 
 _cp_supports_reflink = None
@@ -610,48 +615,6 @@ def get_instance(dandi_instance_id):
         )
 
 
-TITLE_CASE_LOWER = {
-    "a",
-    "an",
-    "and",
-    "as",
-    "but",
-    "by",
-    "for",
-    "in",
-    "nor",
-    "of",
-    "on",
-    "or",
-    "the",
-    "to",
-    "with",
-}
-
-
-def name2title(name):
-    # For use in autopopulating the titles of model schema fields
-    words = []
-    for w in split_camel_case(name):
-        w = w.lower()
-        if w == "id" or w == "url":
-            w = w.upper()
-        elif not words or w not in TITLE_CASE_LOWER:
-            w = w.capitalize()
-        words.append(w)
-    return " ".join(words)
-
-
-def split_camel_case(s):
-    last_start = 0
-    # Don't split apart "ID":
-    for m in re.finditer(r"(?<=I)[A-CE-Z]|(?<=[^I])[A-Z]", s):
-        yield s[last_start : m.start()]
-        last_start = m.start()
-    if last_start < len(s):
-        yield s[last_start:]
-
-
 def try_multiple(ntrials, retry, base):
     """
     ``try_multiple(ntrials, retry, base)(f, *args, **kwargs)`` calls ``f``
diff --git a/setup.cfg b/setup.cfg
index 9e816595..c9f2dc4d 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -30,7 +30,7 @@ install_requires =
     appdirs
     click
     click-didyoumean
-    dandischema
+    dandischema ~= 0.1.0
     etelemetry >= 0.2.0
     fasteners
     fscacher


Test patch:
diff --git a/dandi/cli/tests/test_download.py b/dandi/cli/tests/test_download.py
index 21ded7aa..27bf197c 100644
--- a/dandi/cli/tests/test_download.py
+++ b/dandi/cli/tests/test_download.py
@@ -16,7 +16,7 @@ def test_download_defaults(mocker):
     mock_download.assert_called_once_with(
         (),
         os.curdir,
-        existing="refresh",
+        existing="error",
         format="pyout",
         jobs=6,
         get_metadata=True,
@@ -32,7 +32,7 @@ def test_download_all_types(mocker):
     mock_download.assert_called_once_with(
         (),
         os.curdir,
-        existing="refresh",
+        existing="error",
         format="pyout",
         jobs=6,
         get_metadata=True,
@@ -48,7 +48,7 @@ def test_download_metadata_only(mocker):
     mock_download.assert_called_once_with(
         (),
         os.curdir,
-        existing="refresh",
+        existing="error",
         format="pyout",
         jobs=6,
         get_metadata=True,
@@ -64,7 +64,7 @@ def test_download_assets_only(mocker):
     mock_download.assert_called_once_with(
         (),
         os.curdir,
-        existing="refresh",
+        existing="error",
         format="pyout",
         jobs=6,
         get_metadata=False,
@@ -95,7 +95,7 @@ def test_download_gui_instance_in_dandiset(mocker):
     mock_download.assert_called_once_with(
         ["https://gui.dandiarchive.org/#/dandiset/123456/draft"],
         os.curdir,
-        existing="refresh",
+        existing="error",
         format="pyout",
         jobs=6,
         get_metadata=True,
@@ -117,7 +117,7 @@ def test_download_api_instance_in_dandiset(mocker):
     mock_download.assert_called_once_with(
         ["http://localhost:8000/api/dandisets/123456/"],
         os.curdir,
-        existing="refresh",
+        existing="error",
         format="pyout",
         jobs=6,
         get_metadata=True,
@@ -143,7 +143,7 @@ def test_download_url_instance_match(mocker):
     mock_download.assert_called_once_with(
         ("http://localhost:8000/api/dandisets/123456/",),
         os.curdir,
-        existing="refresh",
+        existing="error",
         format="pyout",
         jobs=6,
         get_metadata=True,
diff --git a/dandi/tests/test_utils.py b/dandi/tests/test_utils.py
index d9426ebc..761c6513 100644
--- a/dandi/tests/test_utils.py
+++ b/dandi/tests/test_utils.py
@@ -21,7 +21,6 @@ from ..utils import (
     get_module_version,
     get_utcnow_datetime,
     is_same_time,
-    name2title,
     on_windows,
     remap_dict,
 )
@@ -343,26 +342,6 @@ def test_server_info():
     assert "services" in data
 
 
-@pytest.mark.parametrize(
-    "name,title",
-    [
-        ("relatedResource", "Related Resource"),
-        ("identifier", "Identifier"),
-        ("wasGeneratedBy", "Was Generated by"),
-        ("sameAs", "Same as"),
-        ("includeInCitation", "Include in Citation"),
-        ("anExtraField", "An Extra Field"),
-        ("propertyID", "Property ID"),
-        ("fieldINeed", "Field I Need"),
-        ("needsADatum", "Needs a Datum"),
-        ("contentUrl", "Content URL"),
-        ("ContactPoint", "Contact Point"),
-    ],
-)
-def test_name2title(name, title):
-    assert name2title(name) == title
-
-
 def test_get_module_version():
     import pynwb