Repository: datalad/datalad-next

Issue:
archivist special remote: add support for tar archives with `.tgz` extension
I'm working on building a dataset from `.tgz` archives using the replacement for `add-archive-content` demonstrated [here](https://github.com/datalad/datalad-next/issues/183#issuecomment-1539943754) in combination with the archivist special remote. The demo below works if the archive is a `.tar.gz` extension but not with `.tgz`. With `.tgz`, I need to configure the `archivist.legacy-mode` for a successful `datalad get`. Here's a quick demo:

```shell
% mkdir project
% touch project/file1.txt project/file2.txt project/file3.txt
% tar -czvf project.tgz project
```

```shell
% datalad create tmp && cd tmp
% cp ../project.tgz ./
% datalad save -m "add archive" project.tgz
% git annex initremote archivist type=external externaltype=archivist encryption=none autoenable=true
% archivekey=$(git annex lookupkey project.tgz)
% datalad -f json ls-file-collection tarfile project.tgz --hash md5 | jq '. | select(.type == "file")' | jq --slurp . | datalad addurls --key 'et:MD5-s{size}--{hash-md5}' - "dl+archive:${archivekey}#path={item}&size={size}" '{item}'
% filekey=$(git annex lookupkey project/file1.txt)
% archivist_uuid=$(git annex info archivist | grep 'uuid' | cut -d ' ' -f 2)
% git annex setpresentkey $filekey $archivist_uuid 1
% datalad get project/file1.txt
get(error): project/file1.txt (file) [Could not obtain 'MD5E-s0--d41d8cd98f00b204e9800998ecf8427e.txt' -caused by- NotImplementedError]
```

```shell
% datalad configuration --scope local set datalad.archivist.legacy-mode=yes                                                1 !
set_configuration(ok): . [datalad.archivist.legacy-mode=yes]
% datalad get project/file1.txt                                            
[INFO   ] datalad-archives special remote is using an extraction cache under /playground/loj/abcd/tmp3/.git/datalad/tmp/archives/8bc4249de3. Remove it with DataLad's 'clean' command to save disk space. 
get(ok): project/file1.txt (file) [from archivist...]
```

<details><summary>datalad wtf</summary>

```
# WTF
## configuration <SENSITIVE, report disabled by configuration>
## credentials 
  - keyring: 
    - active_backends: 
      - PlaintextKeyring with no encyption v.1.0 at /home/loj/.local/share/python_keyring/keyring_pass.cfg
    - config_file: /home/loj/.config/python_keyring/keyringrc.cfg
    - data_root: /home/loj/.local/share/python_keyring
## datalad 
  - version: 0.19.3
## dependencies 
  - annexremote: 1.6.0
  - boto: 2.49.0
  - cmd:7z: 16.02
  - cmd:annex: 10.20221003
  - cmd:bundled-git: UNKNOWN
  - cmd:git: 2.39.2
  - cmd:ssh: 8.4p1
  - cmd:system-git: 2.39.2
  - cmd:system-ssh: 8.4p1
  - humanize: 4.8.0
  - iso8601: 2.1.0
  - keyring: 24.2.0
  - keyrings.alt: 5.0.0
  - msgpack: 1.0.7
  - platformdirs: 3.11.0
  - requests: 2.31.0
## environment 
  - LANG: en_US.UTF-8
  - LANGUAGE: en_US.UTF-8
  - LC_ALL: en_US.UTF-8
  - LC_CTYPE: en_US.UTF-8
  - PATH: /home/loj/.venvs/abcd-long/bin:/home/loj/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
## extensions 
  - container: 
    - description: Containerized environments
    - entrypoints: 
      - datalad_container.containers_add.ContainersAdd: 
        - class: ContainersAdd
        - module: datalad_container.containers_add
        - names: 
          - containers-add
          - containers_add
      - datalad_container.containers_list.ContainersList: 
        - class: ContainersList
        - module: datalad_container.containers_list
        - names: 
          - containers-list
          - containers_list
      - datalad_container.containers_remove.ContainersRemove: 
        - class: ContainersRemove
        - module: datalad_container.containers_remove
        - names: 
          - containers-remove
          - containers_remove
      - datalad_container.containers_run.ContainersRun: 
        - class: ContainersRun
        - module: datalad_container.containers_run
        - names: 
          - containers-run
          - containers_run
    - module: datalad_container
    - version: 1.2.3
  - next: 
    - description: What is next in DataLad
    - entrypoints: 
      - datalad_next.commands.create_sibling_webdav.CreateSiblingWebDAV: 
        - class: CreateSiblingWebDAV
        - module: datalad_next.commands.create_sibling_webdav
        - names: 
          - create-sibling-webdav
      - datalad_next.commands.credentials.Credentials: 
        - class: Credentials
        - module: datalad_next.commands.credentials
        - names: 
      - datalad_next.commands.download.Download: 
        - class: Download
        - module: datalad_next.commands.download
        - names: 
          - download
      - datalad_next.commands.ls_file_collection.LsFileCollection: 
        - class: LsFileCollection
        - module: datalad_next.commands.ls_file_collection
        - names: 
          - ls-file-collection
      - datalad_next.commands.tree.TreeCommand: 
        - class: TreeCommand
        - module: datalad_next.commands.tree
        - names: 
          - tree
    - module: datalad_next
    - version: 1.0.1
## git-annex 
  - build flags: 
    - Assistant
    - Webapp
    - Pairing
    - Inotify
    - DBus
    - DesktopNotify
    - TorrentParser
    - MagicMime
    - Benchmark
    - Feeds
    - Testsuite
    - S3
    - WebDAV
  - dependency versions: 
    - aws-0.22
    - bloomfilter-2.0.1.0
    - cryptonite-0.26
    - DAV-1.3.4
    - feed-1.3.0.1
    - ghc-8.8.4
    - http-client-0.6.4.1
    - persistent-sqlite-2.10.6.2
    - torrent-10000.1.1
    - uuid-1.3.13
    - yesod-1.6.1.0
  - key/value backends: 
    - SHA256E
    - SHA256
    - SHA512E
    - SHA512
    - SHA224E
    - SHA224
    - SHA384E
    - SHA384
    - SHA3_256E
    - SHA3_256
    - SHA3_512E
    - SHA3_512
    - SHA3_224E
    - SHA3_224
    - SHA3_384E
    - SHA3_384
    - SKEIN256E
    - SKEIN256
    - SKEIN512E
    - SKEIN512
    - BLAKE2B256E
    - BLAKE2B256
    - BLAKE2B512E
    - BLAKE2B512
    - BLAKE2B160E
    - BLAKE2B160
    - BLAKE2B224E
    - BLAKE2B224
    - BLAKE2B384E
    - BLAKE2B384
    - BLAKE2BP512E
    - BLAKE2BP512
    - BLAKE2S256E
    - BLAKE2S256
    - BLAKE2S160E
    - BLAKE2S160
    - BLAKE2S224E
    - BLAKE2S224
    - BLAKE2SP256E
    - BLAKE2SP256
    - BLAKE2SP224E
    - BLAKE2SP224
    - SHA1E
    - SHA1
    - MD5E
    - MD5
    - WORM
    - URL
    - X*
  - operating system: linux x86_64
  - remote types: 
    - git
    - gcrypt
    - p2p
    - S3
    - bup
    - directory
    - rsync
    - web
    - bittorrent
    - webdav
    - adb
    - tahoe
    - glacier
    - ddar
    - git-lfs
    - httpalso
    - borg
    - hook
    - external
  - supported repository versions: 
    - 8
    - 9
    - 10
  - upgrade supported from repository versions: 
    - 0
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
    - 8
    - 9
    - 10
  - version: 10.20221003
## location 
  - path: /playground/loj/abcd
  - type: directory
## metadata.extractors 
  - container_inspect: 
    - distribution: datalad-container 1.2.3
    - load_error: ModuleNotFoundError(No module named 'datalad_metalad')
    - module: datalad_container.extractors.metalad_container
## metadata.filters 
## metadata.indexers 
## python 
  - implementation: CPython
  - version: 3.9.2
## system 
  - distribution: debian/11/bullseye
  - encoding: 
    - default: utf-8
    - filesystem: utf-8
    - locale.prefered: UTF-8
  - filesystem: 
    - CWD: 
      - path: /playground/loj/abcd
    - HOME: 
      - path: /home/loj
    - TMP: 
      - path: /tmp
  - max_path_length: 276
  - name: Linux
  - release: 5.10.0-23-amd64
  - type: posix
  - version: #1 SMP Debian 5.10.179-1 (2023-05-12)

```
</details>

Discussion:

Patch:
diff --git a/changelog.d/20231026_185357_michael.hanke_archivist_tgz.md b/changelog.d/20231026_185357_michael.hanke_archivist_tgz.md
new file mode 100644
index 0000000..f41dadc
--- /dev/null
+++ b/changelog.d/20231026_185357_michael.hanke_archivist_tgz.md
@@ -0,0 +1,6 @@
+### 💫 Enhancements and new features
+
+- The `archivist` remote now supports archive type detection
+  from `*E`-type annex keys for `.tgz` archives too.
+  Fixes https://github.com/datalad/datalad-next/issues/517 via
+  https://github.com/datalad/datalad-next/pull/518 (by @mih)
diff --git a/datalad_next/types/archivist.py b/datalad_next/types/archivist.py
index 12e9b2b..17c538d 100644
--- a/datalad_next/types/archivist.py
+++ b/datalad_next/types/archivist.py
@@ -74,7 +74,7 @@ class ArchivistLocator:
     """
     akey: AnnexKey
     member: PurePosixPath
-    size: int
+    size: int | None = None
     # datalad-archives did not have the type info, we want to be
     # able to handle those too, make optional
     atype: ArchiveType | None = None
@@ -91,21 +91,21 @@ class ArchivistLocator:
     @classmethod
     def from_str(cls, url: str):
         """Return ``ArchivistLocator`` from ``str`` form"""
-        url_matched = _recognized_urls.match(url)
-        if not url_matched:
+        url_match = _recognized_urls.match(url)
+        if not url_match:
             raise ValueError('Unrecognized dl+archives locator syntax')
-        url_matched = url_matched.groupdict()
+        url_matched = url_match.groupdict()
         # convert to desired type
         akey = AnnexKey.from_str(url_matched['key'])
 
         # archive member properties
-        props_matched = _archive_member_props.match(url_matched['props'])
-        if not props_matched:
+        props_match = _archive_member_props.match(url_matched['props'])
+        if not props_match:
             # without at least a 'path' there is nothing we can do here
             raise ValueError(
                 'dl+archives locator contains invalid archive member '
                 f'specification: {url_matched["props"]!r}')
-        props_matched = props_matched.groupdict()
+        props_matched = props_match.groupdict()
         amember_path = PurePosixPath(props_matched['path'])
         if amember_path.is_absolute():
             raise ValueError(
@@ -116,6 +116,8 @@ class ArchivistLocator:
 
         # size is optional, regex ensure that it is an int
         size = props_matched.get('size')
+        if size is not None:
+            size = int(size)
 
         # archive type, could be None
         atype = props_matched.get('atype')
@@ -134,6 +136,8 @@ class ArchivistLocator:
                 atype = ArchiveType.zip
             elif '.tar' in suf:
                 atype = ArchiveType.tar
+            elif '.tgz' in suf:
+                atype = ArchiveType.tar
 
         return cls(
             akey=akey,


Test patch:
diff --git a/datalad_next/types/tests/test_archivist.py b/datalad_next/types/tests/test_archivist.py
index 8f78163..b3d03ac 100644
--- a/datalad_next/types/tests/test_archivist.py
+++ b/datalad_next/types/tests/test_archivist.py
@@ -23,6 +23,12 @@ def test_archivistlocator():
     assert ArchivistLocator.from_str(
         'dl+archive:MD5E-s1--e9f624eb778e6f945771c543b6e9c7b2.tar#path=f.txt'
     ).atype == ArchiveType.tar
+    assert ArchivistLocator.from_str(
+        'dl+archive:MD5E-s1--e9f624eb778e6f945771c543b6e9c7b2.tgz#path=f.txt'
+    ).atype == ArchiveType.tar
+    assert ArchivistLocator.from_str(
+        'dl+archive:MD5E-s1--e9f624eb778e6f945771c543b6e9c7b2.tar.gz#path=f.txt'
+    ).atype == ArchiveType.tar
     assert ArchivistLocator.from_str(
         'dl+archive:MD5E-s1--e9f624eb778e6f945771c543b6e9c7b2.zip#path=f.txt'
     ).atype == ArchiveType.zip