Repository: dandi/dandi-cli

Issue:
automate switching to `-i dandi-api` via `dandiarchive.org/server-info` information
ATM that one reports
```json
{
"version": "1.0.0",
"cli-minimal-version": "0.6.0",
"cli-bad-versions": [],
"services": {
"girder": {
"url": "https://girder.dandiarchive.org"
},
"webui": {
"url": "https://gui.dandiarchive.org"
},
"api": {
"url": "https://publish.dandiarchive.org/api"
},
"jupyterhub": {
"url": "https://hub.dandiarchive.org"
}
}
}
```

I think  we should 
- ~~change redirector it to remove entry for `"api"` since that service is long gone (and boost `version` to e.g. `1.1.0`)~~. that was a bad idea, replaced with
  -  [x] make "api" empty (null == None)
- [ ] double check that `dandi-cli` would refuse to interact with the archive if its version is below minimal one specified in that `server-info`
- [ ] adjust `dandi-cli` so if there is a url for `"api"`, we would use `dandi-api` instance by default and not the girder one. (alternative - do that if there is no "girder")
- [ ] release `dandi-cli` 0.14.0

Upon migration
- in redirector - adjust minimal version of the client to become `0.14.0` (assuming that is the one which it would be with default switch) and add `"url"` for the `"api"` and remove "girder", so released client would automagically switch to use `dandi-api` instance 

any flaw in above plan/logic @satra ?

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/dandi/dandi-cli/pull/565?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) Report
> Merging [#565](https://codecov.io/gh/dandi/dandi-cli/pull/565?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) (7de8e40) into [master](https://codecov.io/gh/dandi/dandi-cli/commit/7a843bd4e5b75bbd2d547221943cd65ab45b77bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) (7a843bd) will **decrease** coverage by `20.88%`.
> The diff coverage is `90.00%`.

[![Impacted file tree graph](https://codecov.io/gh/dandi/dandi-cli/pull/565/graphs/tree.svg?width=650&height=150&src=pr&token=cBhomxJ9Ua&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi)](https://codecov.io/gh/dandi/dandi-cli/pull/565?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi)

```diff
@@             Coverage Diff             @@
##           master     #565       +/-   ##
===========================================
- Coverage   83.63%   62.74%   -20.89%     
===========================================
  Files          62       62               
  Lines        6500     6426       -74     
===========================================
- Hits         5436     4032     -1404     
- Misses       1064     2394     +1330     
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `62.74% <90.00%> (-20.89%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/dandi/dandi-cli/pull/565?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) | Coverage Δ | |
|---|---|---|
| [dandi/utils.py](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvdXRpbHMucHk=) | `74.44% <80.00%> (-5.50%)` | :arrow_down: |
| [dandi/delete.py](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvZGVsZXRlLnB5) | `32.55% <100.00%> (-64.39%)` | :arrow_down: |
| [dandi/tests/test\_utils.py](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvdGVzdHMvdGVzdF91dGlscy5weQ==) | `100.00% <100.00%> (ø)` | |
| [dandi/tests/test\_dandiapi.py](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvdGVzdHMvdGVzdF9kYW5kaWFwaS5weQ==) | `12.24% <0.00%> (-87.76%)` | :arrow_down: |
| [dandi/tests/test\_delete.py](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvdGVzdHMvdGVzdF9kZWxldGUucHk=) | `20.16% <0.00%> (-79.84%)` | :arrow_down: |
| [dandi/tests/test\_upload.py](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvdGVzdHMvdGVzdF91cGxvYWQucHk=) | `17.55% <0.00%> (-74.64%)` | :arrow_down: |
| [dandi/upload.py](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvdXBsb2FkLnB5) | `2.37% <0.00%> (-73.40%)` | :arrow_down: |
| [dandi/dandiapi.py](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvZGFuZGlhcGkucHk=) | `19.53% <0.00%> (-68.41%)` | :arrow_down: |
| [dandi/support/generatorify.py](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvc3VwcG9ydC9nZW5lcmF0b3JpZnkucHk=) | `0.00% <0.00%> (-64.37%)` | :arrow_down: |
| ... and [31 more](https://codecov.io/gh/dandi/dandi-cli/pull/565/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/dandi/dandi-cli/pull/565?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/dandi/dandi-cli/pull/565?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi). Last update [7a843bd...7de8e40](https://codecov.io/gh/dandi/dandi-cli/pull/565?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi).


Patch:
diff --git a/dandi/dandiarchive.py b/dandi/dandiarchive.py
index 8a097af6..f50280df 100644
--- a/dandi/dandiarchive.py
+++ b/dandi/dandiarchive.py
@@ -139,6 +139,7 @@ class _dandi_url_parser:
         # - server_type:
         #   - 'girder' - underlying requests should go to girder server
         #   - 'api' - the "new" API service
+        #   - 'redirect' - use redirector's server-info
         # - rewrite:
         #   - callable -- which would rewrite that "URI"
         # - map_instance
@@ -223,7 +224,7 @@ class _dandi_url_parser:
                 r"(/(?P<version>draft))?"
                 rf"(/files(\?_id={id_grp}(&_modelType=folder)?)?)?"
             ),
-            {"server_type": "girder"},
+            {"server_type": "redirect"},
             "https://<server>[/api]#/dandiset/<dandiset id>[/draft]"
             "[/files[?_id=<id>[&_modelType=folder]]]",
         ),
@@ -375,7 +376,24 @@ class _dandi_url_parser:
             )
 
         url_server = groups["server"]
-        server = cls.map_to[server_type].get(url_server.rstrip("/"), url_server)
+        if server_type == "redirect":
+            try:
+                instance_name = known_instances_rev[url_server.rstrip("/")]
+            except KeyError:
+                raise UnknownURLError(f"{url} does not map to a known instance")
+            instance = get_instance(instance_name)
+            if instance.metadata_version == 0:
+                server_type = "girder"
+                server = instance.girder
+            elif instance.metadata_version == 1:
+                server_type = "api"
+                server = instance.api
+            else:
+                raise RuntimeError(
+                    f"Unknown instance metadata_version: {instance.metadata_version}"
+                )
+        else:
+            server = cls.map_to[server_type].get(url_server.rstrip("/"), url_server)
 
         if not server.endswith("/"):
             server += "/"  # we expected '/' to be there so let it be
diff --git a/dandi/delete.py b/dandi/delete.py
index 52174da7..22beec21 100644
--- a/dandi/delete.py
+++ b/dandi/delete.py
@@ -6,11 +6,11 @@ from typing import Iterable, Iterator, NamedTuple, Optional, Set, Tuple
 import click
 import requests
 
-from .consts import dandiset_metadata_file, known_instances
+from .consts import dandiset_metadata_file
 from .dandiapi import DandiAPIClient
 from .dandiarchive import parse_dandi_url
 from .exceptions import NotFoundError
-from .utils import is_url
+from .utils import get_instance, is_url
 
 
 class RemoteAsset(NamedTuple):
@@ -115,7 +115,7 @@ class Deleter:
             raise RuntimeError(f"Unexpected asset type for {url}: {asset_type}")
 
     def register_local_path_equivalent(self, instance_name: str, filepath: str) -> None:
-        instance = known_instances[instance_name]
+        instance = get_instance(instance_name)
         if instance.girder is not None:
             raise NotImplementedError("Cannot delete assets from Girder instances")
         api_url = instance.api
diff --git a/dandi/utils.py b/dandi/utils.py
index afa17219..4b5fb20f 100644
--- a/dandi/utils.py
+++ b/dandi/utils.py
@@ -618,14 +618,27 @@ def get_instance(dandi_instance_id):
         raise CliVersionTooOldError(our_version, minversion, bad_versions)
     if our_version in bad_versions:
         raise BadCliVersionError(our_version, minversion, bad_versions)
-    services = server_info["services"]
-    return dandi_instance(
-        metadata_version=0,
-        girder=services.get("girder", {}).get("url"),
-        gui=services.get("webui", {}).get("url"),
-        redirector=redirector_url,
-        api=services.get("api", {}).get("url"),
-    )
+    if server_info["services"].get("girder"):
+        return dandi_instance(
+            metadata_version=0,
+            girder=server_info["services"].get("girder", {}).get("url"),
+            gui=server_info["services"].get("webui", {}).get("url"),
+            redirector=redirector_url,
+            api=None,
+        )
+    elif server_info["services"].get("api"):
+        return dandi_instance(
+            metadata_version=1,
+            girder=None,
+            gui=server_info["services"].get("webui", {}).get("url"),
+            redirector=redirector_url,
+            api=server_info["services"].get("api", {}).get("url"),
+        )
+    else:
+        raise RuntimeError(
+            "redirector's server-info returned unknown set of services keys: "
+            + ", ".join(k for k, v in server_info["services"].items() if v is not None)
+        )
 
 
 TITLE_CASE_LOWER = {


Test patch:
diff --git a/dandi/tests/test_dandiarchive.py b/dandi/tests/test_dandiarchive.py
index 4a6ee420..76087cfa 100644
--- a/dandi/tests/test_dandiarchive.py
+++ b/dandi/tests/test_dandiarchive.py
@@ -1,4 +1,5 @@
 import pytest
+import responses
 
 from dandi.consts import known_instances
 from dandi.dandiarchive import follow_redirect, parse_dandi_url
@@ -174,3 +175,53 @@ def test_follow_redirect():
         follow_redirect("https://bit.ly/dandi12")
         == "https://gui.dandiarchive.org/#/file-browser/folder/5e72b6ac3da50caa9adb0498"
     )
+
+
+@responses.activate
+def test_parse_gui_old_redirect():
+    responses.add(
+        responses.GET,
+        "https://dandiarchive.org/server-info",
+        json={
+            "version": "1.2.0",
+            "cli-minimal-version": "0.6.0",
+            "cli-bad-versions": [],
+            "services": {
+                "girder": {"url": "https://girder.dandiarchive.org"},
+                "webui": {"url": "https://gui.dandirchive.org"},
+                "api": None,
+                "jupyterhub": {"url": "https://hub.dandiarchive.org"},
+            },
+        },
+    )
+    assert parse_dandi_url("https://gui.dandiarchive.org/#/dandiset/000003") == (
+        "girder",
+        "https://girder.dandiarchive.org/",
+        "dandiset",
+        {"dandiset_id": "000003", "version": "draft"},
+    )
+
+
+@responses.activate
+def test_parse_gui_new_redirect():
+    responses.add(
+        responses.GET,
+        "https://dandiarchive.org/server-info",
+        json={
+            "version": "1.2.0",
+            "cli-minimal-version": "0.6.0",
+            "cli-bad-versions": [],
+            "services": {
+                "girder": None,
+                "webui": {"url": "https://gui.dandirchive.org"},
+                "api": {"url": "https://api.dandiarchive.org/api"},
+                "jupyterhub": {"url": "https://hub.dandiarchive.org"},
+            },
+        },
+    )
+    assert parse_dandi_url("https://gui.dandiarchive.org/#/dandiset/000003") == (
+        "api",
+        "https://api.dandiarchive.org/api/",
+        "dandiset",
+        {"dandiset_id": "000003", "version": None},
+    )
diff --git a/dandi/tests/test_utils.py b/dandi/tests/test_utils.py
index b833444f..92c5d8cf 100644
--- a/dandi/tests/test_utils.py
+++ b/dandi/tests/test_utils.py
@@ -168,7 +168,7 @@ def test_get_instance_dandi():
             "services": {
                 "girder": {"url": "https://girder.dandi"},
                 "webui": {"url": "https://gui.dandi"},
-                "api": {"url": "https://publish.dandi/api"},
+                "api": None,
                 "jupyterhub": {"url": "https://hub.dandi"},
             },
         },
@@ -178,7 +178,33 @@ def test_get_instance_dandi():
         girder="https://girder.dandi",
         gui="https://gui.dandi",
         redirector="https://dandiarchive.org",
-        api="https://publish.dandi/api",
+        api=None,
+    )
+
+
+@responses.activate
+def test_get_instance_dandi_with_api():
+    responses.add(
+        responses.GET,
+        "https://dandiarchive.org/server-info",
+        json={
+            "version": "1.0.0",
+            "cli-minimal-version": "0.5.0",
+            "cli-bad-versions": [],
+            "services": {
+                "girder": None,
+                "webui": {"url": "https://gui.dandi"},
+                "api": {"url": "https://api.dandi"},
+                "jupyterhub": {"url": "https://hub.dandi"},
+            },
+        },
+    )
+    assert get_instance("dandi") == dandi_instance(
+        metadata_version=1,
+        girder=None,
+        gui="https://gui.dandi",
+        redirector="https://dandiarchive.org",
+        api="https://api.dandi",
     )
 
 
@@ -194,7 +220,7 @@ def test_get_instance_url():
             "services": {
                 "girder": {"url": "https://girder.dandi"},
                 "webui": {"url": "https://gui.dandi"},
-                "api": {"url": "https://publish.dandi/api"},
+                "api": None,
                 "jupyterhub": {"url": "https://hub.dandi"},
             },
         },
@@ -204,7 +230,7 @@ def test_get_instance_url():
         girder="https://girder.dandi",
         gui="https://gui.dandi",
         redirector="https://example.dandi/",
-        api="https://publish.dandi/api",
+        api=None,
     )
 
 
@@ -220,7 +246,7 @@ def test_get_instance_cli_version_too_old():
             "services": {
                 "girder": {"url": "https://girder.dandi"},
                 "webui": {"url": "https://gui.dandi"},
-                "api": {"url": "https://publish.dandi/api"},
+                "api": None,
                 "jupyterhub": {"url": "https://hub.dandi"},
             },
         },
@@ -245,7 +271,7 @@ def test_get_instance_bad_cli_version():
             "services": {
                 "girder": {"url": "https://girder.dandi"},
                 "webui": {"url": "https://gui.dandi"},
-                "api": {"url": "https://publish.dandi/api"},
+                "api": None,
                 "jupyterhub": {"url": "https://hub.dandi"},
             },
         },
@@ -312,7 +338,7 @@ def test_get_instance_bad_version_from_server():
             "services": {
                 "girder": {"url": "https://girder.dandi"},
                 "webui": {"url": "https://gui.dandi"},
-                "api": {"url": "https://publish.dandi/api"},
+                "api": None,
                 "jupyterhub": {"url": "https://hub.dandi"},
             },
         },