Repository: datalad/datalad

Issue:
`datalad get` error message suboptimal
@mih 

#### What is the problem?
 Confusing error from a failed `datalad get`.

#### What steps will reproduce the problem?

```
$ datalad clone ria+file:///p/fastdata/inm7/data_store/tmp/store#cf8bf616-6cee-11ea-8199-7cd30ae51ae8 test4
[INFO   ] Configure additional publication dependency on "store-storage"                                                                                                                               
configure-sibling(ok): . (sibling)
install(ok): /p/fastdata/inm7/data_store/tmp/test4 (dataset)
action summary:
  configure-sibling (ok: 1)
  install (ok: 1)

$ cd test4/

$ ls
file1

$ datalad get file1
[WARNING] Running get resulted in stderr output: 
ERROR: No more files
/p/fastdata/inm7/data_store/tmp/alt_store/cf8/bf616-6cee-11ea-8199-7cd30ae51ae8/archives/archive.7z



System ERROR:
Unknown error -2147024872
git-annex: get: 1 failed
 
[ERROR  ] verification of content failed [get(/p/fastdata/inm7/data_store/tmp/test4/file1)] 
get(error): file1 (file) [verification of content failed]
```
#### What version of DataLad are you using?
<details><summary>datalad wtf</summary>
<p>

```
$ datalad wtf
# WTF
## configuration <SENSITIVE, report disabled by configuration>
## datalad 
  - full_version: 0.12.4.dev537-g7bb9
  - version: 0.12.4.dev537
## dependencies 
  - appdirs: 1.4.3
  - boto: 2.49.0
  - cmd:7z: 16.02
  - cmd:annex: 8.20200309-g07fcace
  - cmd:bundled-git: 2.23.0
  - cmd:git: 2.23.0
  - cmd:system-git: 2.23.0
  - cmd:system-ssh: 7.9p1
  - exifread: 2.1.2
  - git: 3.1.0
  - gitdb: 4.0.2
  - humanize: 2.1.0
  - iso8601: 0.1.12
  - keyring: 21.1.1
  - keyrings.alt: 3.1
  - msgpack: 1.0.0
  - mutagen: 1.41.1
  - requests: 2.22.0
  - wrapt: 1.12.1
## environment 
  - LANG: en_US.UTF-8
  - PATH: /p/home/jusers/waite1/judac/shared/miniconda3/bin:/p/home/jusers/waite1/judac/shared/miniconda3/condabin:/opt/lenovo/onecli:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/usr/lpp/mmfs/bin
## extensions 
  - ria: 
    - description: Helper for the remote indexed archive (RIA) special remote
    - entrypoints: 
      - ria_remote.export_archive.ExportArchive: 
        - class: ExportArchive
        - load_error: None
        - module: ria_remote.export_archive
        - names: 
          - ria-export-archive
          - ria_export_archive
      - ria_remote.install.Install: 
        - class: Install
        - load_error: function has argument 'alt_sources' not described as a parameter [base.py:update_docstring_with_parameters:374]
        - module: ria_remote.install
        - names: 
          - ria-install
          - ria_install
    - load_error: None
    - module: ria_remote
    - version: 0.7
## git-annex 
  - build flags: 
    - Assistant
    - Webapp
    - Pairing
    - S3
    - WebDAV
    - Inotify
    - DBus
    - DesktopNotify
    - TorrentParser
    - MagicMime
    - Feeds
    - Testsuite
  - dependency versions: 
    - aws-0.21.1
    - bloomfilter-2.0.1.0
    - cryptonite-0.25
    - DAV-1.3.4
    - feed-1.2.0.1
    - ghc-8.6.5
    - http-client-0.6.4
    - persistent-sqlite-2.9.3
    - torrent-10000.1.1
    - uuid-1.3.13
    - yesod-1.6.0.1
  - key/value backends: 
    - SHA256E
    - SHA256
    - SHA512E
    - SHA512
    - SHA224E
    - SHA224
    - SHA384E
    - SHA384
    - SHA3_256E
    - SHA3_256
    - SHA3_512E
    - SHA3_512
    - SHA3_224E
    - SHA3_224
    - SHA3_384E
    - SHA3_384
    - SKEIN256E
    - SKEIN256
    - SKEIN512E
    - SKEIN512
    - BLAKE2B256E
    - BLAKE2B256
    - BLAKE2B512E
    - BLAKE2B512
    - BLAKE2B160E
    - BLAKE2B160
    - BLAKE2B224E
    - BLAKE2B224
 - BLAKE2B384E
    - BLAKE2B384
    - BLAKE2BP512E
    - BLAKE2BP512
    - BLAKE2S256E
    - BLAKE2S256
    - BLAKE2S160E
    - BLAKE2S160
    - BLAKE2S224E
    - BLAKE2S224
    - BLAKE2SP256E
    - BLAKE2SP256
    - BLAKE2SP224E
    - BLAKE2SP224
    - SHA1E
    - SHA1
    - MD5E
    - MD5
    - WORM
    - URL
  - operating system: linux x86_64
  - remote types: 
    - git
    - gcrypt
    - p2p
    - S3
    - bup
    - directory
    - rsync
    - web
    - bittorrent
    - webdav
    - adb
    - tahoe
    - glacier
    - ddar
    - git-lfs
    - hook
    - external
  - supported repository versions: 
    - 8
  - upgrade supported from repository versions: 
    - 0
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
  - version: 8.20200309-g07fcace
## location 
  - path: /p/fastdata/inm7/data_store
  - type: directory
## metadata_extractors 
  - annex: 
    - load_error: None
    - module: datalad.metadata.extractors.annex
    - version: None
  - audio: 
    - load_error: None
    - module: datalad.metadata.extractors.audio
    - version: None
  - datacite: 
    - load_error: None
    - module: datalad.metadata.extractors.datacite
    - version: None
  - datalad_core: 
    - load_error: None
    - module: datalad.metadata.extractors.datalad_core
    - version: None
  - datalad_rfc822: 
    - load_error: None
    - module: datalad.metadata.extractors.datalad_rfc822
    - version: None
  - exif: 
    - load_error: None
    - module: datalad.metadata.extractors.exif
    - version: None
  - frictionless_datapackage: 
    - load_error: None
    - module: datalad.metadata.extractors.frictionless_datapackage
    - version: None
  - image: 
    - load_error: None
    - module: datalad.metadata.extractors.image
    - version: None
  - xmp: 
    - load_error: No module named 'libxmp' [xmp.py:<module>:20]
    - module: datalad.metadata.extractors.xmp
## python 
  - implementation: CPython
  - version: 3.7.6
## system 
  - distribution: CentOS Linux/7.7.1908/Core
  - encoding: 
    - default: utf-8
    - filesystem: utf-8
    - locale.prefered: UTF-8
  - max_path_length: 283
  - name: Linux
  - release: 3.10.0-1062.1.2.el7.x86_64
  - type: posix
  - version: #1 SMP Mon Sep 30 14:19:46 UTC 2019
```
</p>
</details>

Comments:

Patch:
diff --git a/datalad/distributed/ora_remote.py b/datalad/distributed/ora_remote.py
index a397806dd..dc80befb2 100644
--- a/datalad/distributed/ora_remote.py
+++ b/datalad/distributed/ora_remote.py
@@ -188,6 +188,12 @@ class LocalIO(IOBase):
         )
 
     def get_from_archive(self, archive, src, dst, progress_cb):
+        # Upfront check to avoid cryptic error output
+        # https://github.com/datalad/datalad/issues/4336
+        if not self.exists(archive):
+            raise RIARemoteError("archive {arc} does not exist."
+                                 "".format(arc=archive))
+
         # this requires python 3.5
         with open(dst, 'wb') as target_file:
             subprocess.run([
@@ -1197,6 +1203,9 @@ class RIARemote(SpecialRemote):
     def transfer_store(self, key, filename):
         self._ensure_writeable()
 
+        # we need a file-system compatible name for the key
+        key = _sanitize_key(key)
+
         dsobj_dir, archive_path, key_path = self._get_obj_location(key)
         key_path = dsobj_dir / key_path
 
@@ -1234,6 +1243,8 @@ class RIARemote(SpecialRemote):
 
     @handle_errors
     def transfer_retrieve(self, key, filename):
+        # we need a file-system compatible name for the key
+        key = _sanitize_key(key)
 
         if isinstance(self.io, HTTPRemoteIO):
             self.io.get(PurePosixPath(self.annex.dirhash(key)) / key / key,
@@ -1259,6 +1270,8 @@ class RIARemote(SpecialRemote):
 
     @handle_errors
     def checkpresent(self, key):
+        # we need a file-system compatible name for the key
+        key = _sanitize_key(key)
 
         if isinstance(self.io, HTTPRemoteIO):
             return self.io.checkpresent(
@@ -1278,6 +1291,9 @@ class RIARemote(SpecialRemote):
 
     @handle_errors
     def remove(self, key):
+        # we need a file-system compatible name for the key
+        key = _sanitize_key(key)
+
         self._ensure_writeable()
 
         dsobj_dir, archive_path, key_path = self._get_obj_location(key)
@@ -1304,6 +1320,8 @@ class RIARemote(SpecialRemote):
 
     @handle_errors
     def whereis(self, key):
+        # we need a file-system compatible name for the key
+        key = _sanitize_key(key)
 
         if isinstance(self.io, HTTPRemoteIO):
             # display the URL for a request
@@ -1352,6 +1370,35 @@ class RIARemote(SpecialRemote):
     # TODO: implement method 'error'
 
 
+def _sanitize_key(key):
+    """Returns a sanitized key that is a suitable directory/file name
+
+    Documentation from the analog implementation in git-annex
+    Annex/Locations.hs
+
+    Converts a key into a filename fragment without any directory.
+
+    Escape "/" in the key name, to keep a flat tree of files and avoid
+    issues with keys containing "/../" or ending with "/" etc.
+
+    "/" is escaped to "%" because it's short and rarely used, and resembles
+        a slash
+    "%" is escaped to "&s", and "&" to "&a"; this ensures that the mapping
+        is one to one.
+    ":" is escaped to "&c", because it seemed like a good idea at the time.
+
+    Changing what this function escapes and how is not a good idea, as it
+    can cause existing objects to get lost.
+    """
+    esc = {
+        '/': '%',
+        '%': '&s',
+        '&': '&a',
+        ':': '&c',
+    }
+    return ''.join(esc.get(c, c) for c in key)
+
+
 def main():
     """cmdline entry point"""
     from annexremote import Master


Test patch:
diff --git a/datalad/distributed/tests/test_ria_basics.py b/datalad/distributed/tests/test_ria_basics.py
index 885d75558..e470e057e 100644
--- a/datalad/distributed/tests/test_ria_basics.py
+++ b/datalad/distributed/tests/test_ria_basics.py
@@ -11,6 +11,7 @@ import logging
 from datalad.api import (
     Dataset,
     clone,
+    create_sibling_ria,
 )
 from datalad.utils import Path
 from datalad.tests.utils import (
@@ -35,7 +36,8 @@ from datalad.tests.utils import (
 )
 from datalad.distributed.ora_remote import (
     LocalIO,
-    SSHRemoteIO
+    SSHRemoteIO,
+    _sanitize_key,
 )
 from datalad.support.exceptions import (
     CommandError,
@@ -624,3 +626,51 @@ def test_push_url(storepath, dspath, blockfile):
     known_sources = ds.repo.whereis('one.txt')
     assert_in(here_uuid, known_sources)
     assert_in(store_uuid, known_sources)
+
+
+@known_failure_windows
+@with_tempfile
+@with_tempfile
+def test_url_keys(dspath, storepath):
+    ds = Dataset(dspath).create()
+    repo = ds.repo
+    filename = 'url_no_size.html'
+    # URL-type key without size
+    repo.call_annex([
+        'addurl', '--relaxed', '--raw', '--file', filename, 'http://example.com',
+    ])
+    ds.save()
+    # copy target
+    ds.create_sibling_ria(
+        name='ria',
+        url='ria+file://{}'.format(storepath),
+        storage_sibling='only',
+    )
+    ds.get(filename)
+    repo.call_annex(['copy', '--to', 'ria', filename])
+    ds.drop(filename)
+    # in the store and on the web
+    assert_equal(len(ds.repo.whereis(filename)), 2)
+    # try download, but needs special permissions to even be attempted
+    ds.config.set('annex.security.allow-unverified-downloads', 'ACKTHPPT', where='local')
+    repo.call_annex(['copy', '--from', 'ria', filename])
+    assert_equal(len(ds.repo.whereis(filename)), 3)
+    # smoke tests that execute the remaining pieces with the URL key
+    repo.call_annex(['fsck', '-f', 'ria'])
+    assert_equal(len(ds.repo.whereis(filename)), 3)
+    # mapped key in whereis output
+    assert_in('%%example', repo.call_annex(['whereis', filename]))
+
+    repo.call_annex(['move', '-f', 'ria', filename])
+    # check that it does not magically reappear, because it actually
+    # did not drop the file
+    repo.call_annex(['fsck', '-f', 'ria'])
+    assert_equal(len(ds.repo.whereis(filename)), 2)
+
+
+def test_sanitize_key():
+    for i, o in (
+                ('http://example.com/', 'http&c%%example.com%'),
+                ('/%&:', '%&s&a&c'),
+            ):
+        assert_equal(_sanitize_key(i), o)