Repository: dandi/dandi-cli

Issue:
Interface new implementation (API server) in dandi-cli top-level interfaces
A decision was made in https://github.com/dandi/dandi-cli/pull/283 to aim to get `master` in a shape to support interaction with both current girder-based deployed dandiarchive **and** WiP https://github.com/dandi/dandi-api implementations within the same code base. I think it is doable.  #283 will provide "internal" interfaces for an upload/download cycle and then we would need to use/expose  them through top level interfaces.  For that, I think we should

- to `known_instances` add a new entry `"dandi-api"` which would have no girder, we ui, redirector and only API. 
  - we would need also at least a `dandi-api-local-docker-tests` to point to the fixture'd instance
  - we should not occlude other existing fixtures/instances such as `local-docker-tests` since they reflect the instances of currently deployed dandiarchive setup (albeit now without any API which was provided by publish. for a bit)
    -  edit: https://github.com/dandi/dandi-cli/pull/283/files#diff-96e695b73dd490b5f194b670b03c04d549000cb6c3e9b3fef76e8065c68c041aR100 suggests to reuse the same record, which seems to be Ok ATM since we have no longer `publish.` API to worry about.  But I think it would just delay establishing such a dedicated record, whenever we do get a test deployment of new web ui/redirector which would work with new API.  So IMHO might be better to be explicit and have separate dedicated records
  - most likely we should add `metadata-version` "field" to those records, where for old ones it would be 0 (unversioned ad-hoc), and then `1` for the new version of metadata (ideally it should [be the API server](https://github.com/dandi/dandi-api/issues/36) which tells which version of metadata schema it expects, but that is -- later).

### upload

- `--dandi-instance` option of `dandi upload` (in DANDI_DEVEL mode) would automagically list those added instance(s)
  - `upload` code should be RFed (main logic is in [`process_path`](https://github.com/dandi/dandi-cli/blob/master/dandi/upload.py#L147) so it could support both upload to original girder-based client and new API-based one
     - for new API-based one, metadata extraction should use new metadata (version 1) schema

### download

for `download`, which does not rely on explicit specification of which instance to talk to, but parses from URL/identifier we need to come up with a "schema" on how to reference content from new API server, which would be easy on humans and flexible to support multiple instances/servers.  Specification/parsing of such urls should be added to `known_urls` (https://github.com/dandi/dandi-cli/blob/master/dandi/dandiarchive.py#L109).

So what about just taking what we had/(have) on a recent iteration with `publish` API and tune it to correspond to our API (`/dandisets/{version__dandiset__pk}/versions/{version__version}/assets/paths/`) call:

```python
       f"{server_grp}#.*/(?P<asset_type>dandiset)s/{dandiset_id_grp}"
        "/(?P<version>versions/([.0-9]{5,}|draft))"
        "(/assets/path(?location=(?P<location>.*)?)?)?"
        "$": {"server_type": "dandi-api"}
```

NB initially I thought we might need to add some optional prefix `f"(dandi-api::)?" for early "decision making" but because of unique `/dandisets/` I think we do not need it

so sample url for download would look like   

- `https://api.dandiarchive.org/dandisets/000001`, `https://api.dandiarchive.org/dandisets/000001/versions/draft`, `https://api.dandiarchive.org/dandisets/000001/versions/draft/path/`  -- entire dataset, draft version
- `https://api.dandiarchive.org/dandisets/000001/0.0.1` -- entire dataset, `0.0.1` version
- `https://api.dandiarchive.org/dandisets/000001/draft/path/sub-XXX` -- sub-XXX folder (or a file if points to an asset) of draft version

instead of `api.dandiarchive.org`, urls could point to `localhost:port` in the test.

Later, whenever web UI starts using API, we would need to adjust other "schemas". redirector will start redirecting to new web ui, possibly different urls.  But for now, IMHO it is sensible to just follow the API URLs.  Those will not be really "visible" to users, but would allow us to test new functionality/interaction with API server.
 

Discussion:

Patch:
diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md
index 51f7fdf6..c9312dae 100644
--- a/DEVELOPMENT.md
+++ b/DEVELOPMENT.md
@@ -42,7 +42,7 @@ development command line options.
   All those options would otherwise be hidden from the user visible (`--help`)
   interface, unless this env variable is set to non-empty value
 
-- `DANDI_API_KEY` -- avoids using keyrings, thus making it possible to
+- `DANDI_GIRDER_API_KEY` -- avoids using keyrings, thus making it possible to
   "temporarily" use another account etc.
 
 - `DANDI_LOG_LEVEL` -- set log level. By default `INFO`, should be an int (`10` - `DEBUG`).
diff --git a/dandi/cli/cmd_download.py b/dandi/cli/cmd_download.py
index e44e1c7d..a880e9f0 100644
--- a/dandi/cli/cmd_download.py
+++ b/dandi/cli/cmd_download.py
@@ -1,7 +1,7 @@
 import os
 
 import click
-from .base import devel_option, map_to_click_exceptions
+from .base import map_to_click_exceptions
 
 
 class ChoiceList(click.ParamType):
@@ -78,12 +78,12 @@ class ChoiceList(click.ParamType):
 #     type=click.Choice(["require", "skip", "ignore"]),
 #     default="require",
 # )
-@devel_option(
-    "--develop-debug",
-    help="For development: do not use pyout callbacks, do not swallow exception",
-    default=False,
-    is_flag=True,
-)
+# @devel_option(
+#     "--develop-debug",
+#     help="For development: do not use pyout callbacks, do not swallow exception",
+#     default=False,
+#     is_flag=True,
+# )
 @click.argument("url", nargs=-1)
 @map_to_click_exceptions
 def download(url, output_dir, existing, jobs, format, download_types):
diff --git a/dandi/consts.py b/dandi/consts.py
index 016539e8..2bb46c27 100644
--- a/dandi/consts.py
+++ b/dandi/consts.py
@@ -75,7 +75,9 @@ metadata_digests = ("sha1", "md5", "sha512", "sha256")
 dandiset_metadata_file = "dandiset.yaml"
 dandiset_identifier_regex = "^[0-9]{6}$"
 
-dandi_instance = namedtuple("dandi_instance", ("girder", "gui", "redirector", "api"))
+dandi_instance = namedtuple(
+    "dandi_instance", ("metadata_version", "girder", "gui", "redirector", "api")
+)
 
 # So it could be easily mapped to external IP (e.g. from within VM)
 # to test against instance running outside of current environment
@@ -83,10 +85,11 @@ instancehost = os.environ.get("DANDI_INSTANCEHOST", "localhost")
 
 known_instances = {
     "local-girder-only": dandi_instance(
-        f"http://{instancehost}:8080", None, None, None
+        0, f"http://{instancehost}:8080", None, None, None
     ),  # just pure girder
     # Redirector: TODO https://github.com/dandi/dandiarchive/issues/139
     "local-docker": dandi_instance(
+        0,
         f"http://{instancehost}:8080",
         f"http://{instancehost}:8085",
         None,
@@ -94,16 +97,28 @@ known_instances = {
         # may be https://github.com/dandi/dandi-publish/pull/71 would help
     ),
     "local-docker-tests": dandi_instance(
+        0,
         f"http://{instancehost}:8081",
         f"http://{instancehost}:8086",
         f"http://{instancehost}:8079",
-        f"http://{instancehost}:8000/api",
+        None,
     ),
     "dandi": dandi_instance(
+        0,
         "https://girder.dandiarchive.org",
         "https://gui.dandiarchive.org",
         "https://dandiarchive.org",
-        "https://publish.dandiarchive.org/api",  # ? might become api.
+        None,  # publish. is gone, superseded by API which did not yet fully superseded the rest
+    ),
+    "dandi-api": dandi_instance(
+        1,
+        None,
+        "https://gui-beta-dandiarchive-org.netlify.app",
+        None,
+        "https://api.dandiarchive.org/api",
+    ),
+    "dandi-api-local-docker-tests": dandi_instance(
+        1, None, None, None, f"http://{instancehost}:8000/api"
     ),
 }
 # to map back url: name
diff --git a/dandi/dandiapi.py b/dandi/dandiapi.py
index be2f658d..29f61395 100644
--- a/dandi/dandiapi.py
+++ b/dandi/dandiapi.py
@@ -5,7 +5,8 @@ from time import sleep
 import requests
 
 from . import get_logger
-from .consts import MAX_CHUNK_SIZE
+from .consts import MAX_CHUNK_SIZE, known_instances_rev
+from .girder import keyring_lookup
 from .support.digests import Digester
 
 lgr = get_logger()
@@ -219,7 +220,29 @@ class DandiAPIClient(RESTFullAPIClient):
     def __init__(self, api_url, token=None):
         super().__init__(api_url)
         if token is not None:
-            self._headers["Authorization"] = f"token {token}"
+            self.authenticate(token)
+
+    def authenticate(self, token):
+        self._headers["Authorization"] = f"token {token}"
+
+    def dandi_authenticate(self):
+        # Shortcut for advanced folks
+        api_key = os.environ.get("DANDI_API_KEY", None)
+        if api_key:
+            self.authenticate(api_key)
+            lgr.debug("Successfully authenticated using the key from the envvar")
+            return
+        if self.api_url in known_instances_rev:
+            client_name = known_instances_rev[self.api_url]
+        else:
+            raise NotImplementedError("TODO client name derivation for keyring")
+        app_id = f"dandi-api-{client_name}"
+        keyring_backend, api_key = keyring_lookup(app_id, "key")
+        if not api_key:
+            api_key = input(f"Please provide API Key for {client_name}: ")
+            keyring_backend.set_password(app_id, "key", api_key)
+            lgr.debug("Stored key in keyring")
+        self.authenticate(api_key)
 
     def get_asset(self, dandiset_id, version, uuid):
         """
@@ -309,6 +332,49 @@ class DandiAPIClient(RESTFullAPIClient):
         return dandiset
 
     def upload(self, dandiset_id, version_id, asset_path, asset_metadata, filepath):
+        """
+        Parameters
+        ----------
+        dandiset_id: str
+          the ID of the Dandiset to which to upload the file
+        version_id: str
+          the ID of the version of the Dandiset to which to upload the file
+        asset_path: str
+          the POSIX path at which the uploaded file will be placed on the
+          server
+        asset_metadata: dict
+          metadata for the uploaded asset file
+        filepath: str or PathLike
+          the path to the local file to upload
+        """
+        for r in self.iter_upload(
+            dandiset_id, version_id, asset_path, asset_metadata, filepath
+        ):
+            if r["status"] == "validating":
+                sleep(0.1)
+
+    def iter_upload(
+        self, dandiset_id, version_id, asset_path, asset_metadata, filepath
+    ):
+        """
+        Parameters
+        ----------
+        dandiset_id: str
+          the ID of the Dandiset to which to upload the file
+        version_id: str
+          the ID of the version of the Dandiset to which to upload the file
+        asset_path: str
+          the POSIX path at which the uploaded file will be placed on the
+          server
+        asset_metadata: dict
+          metadata for the uploaded asset file
+        filepath: str or PathLike
+          the path to the local file to upload
+
+        Returns
+        -------
+        a generator of `dict`s containing at least a ``"status"`` key
+        """
         filehash = Digester(["sha256"])(filepath)["sha256"]
         lgr.debug("Calculated sha256 digest of %s for %s", filehash, filepath)
         try:
@@ -323,51 +389,61 @@ class DandiAPIClient(RESTFullAPIClient):
             lgr.debug("Blob is already uploaded to server")
             blob_exists = True
         if not blob_exists:
+            total_size = os.path.getsize(filepath)
             lgr.debug("Beginning upload")
             resp = self.post(
                 "/uploads/initialize/",
                 json={
                     "file_name": f"{dandiset_id}/{version_id}/{asset_path}",
-                    "file_size": os.path.getsize(filepath),
+                    "file_size": total_size,
                 },
             )
             object_key = resp["object_key"]
             upload_id = resp["upload_id"]
             parts_out = []
-            with open(filepath, "rb") as fp:
-                for part in resp["parts"]:
-                    chunk = fp.read(part["size"])
-                    if len(chunk) != part["size"]:
-                        raise RuntimeError(
-                            f"End of file {filepath} reached unexpectedly early"
+            bytes_uploaded = 0
+            storage = RESTFullAPIClient("http://nil.nil")
+            with storage.session():
+                with open(filepath, "rb") as fp:
+                    for part in resp["parts"]:
+                        chunk = fp.read(part["size"])
+                        if len(chunk) != part["size"]:
+                            raise RuntimeError(
+                                f"End of file {filepath} reached unexpectedly early"
+                            )
+                        lgr.debug(
+                            "Uploading part %d (%d bytes)",
+                            part["part_number"],
+                            part["size"],
                         )
-                    lgr.debug(
-                        "Uploading part %d (%d bytes)",
-                        part["part_number"],
-                        part["size"],
-                    )
-                    r = self.put(part["upload_url"], data=chunk, json_resp=False)
-                    parts_out.append(
-                        {
-                            "part_number": part["part_number"],
-                            "size": part["size"],
-                            "etag": r.headers["ETag"],
+                        r = storage.put(part["upload_url"], data=chunk, json_resp=False)
+                        bytes_uploaded += len(chunk)
+                        yield {
+                            "status": "uploading",
+                            "upload": 100 * bytes_uploaded / total_size,
+                            "current": bytes_uploaded,
                         }
-                    )
-            lgr.debug("Completing upload")
-            resp = self.post(
-                "/uploads/complete/",
-                json={
-                    "object_key": object_key,
-                    "upload_id": upload_id,
-                    "parts": parts_out,
-                },
-            )
-            self.post(resp["complete_url"], data=resp["body"], json_resp=False)
-            self.post(
-                "/uploads/validate/",
-                json={"sha256": filehash, "object_key": object_key},
-            )
+                        parts_out.append(
+                            {
+                                "part_number": part["part_number"],
+                                "size": part["size"],
+                                "etag": r.headers["ETag"],
+                            }
+                        )
+                lgr.debug("Completing upload")
+                resp = self.post(
+                    "/uploads/complete/",
+                    json={
+                        "object_key": object_key,
+                        "upload_id": upload_id,
+                        "parts": parts_out,
+                    },
+                )
+                storage.post(resp["complete_url"], data=resp["body"], json_resp=False)
+                self.post(
+                    "/uploads/validate/",
+                    json={"sha256": filehash, "object_key": object_key},
+                )
         while True:
             lgr.debug("Waiting for server-side validation to complete")
             resp = self.get(f"/uploads/validations/{filehash}/")
@@ -378,12 +454,14 @@ class DandiAPIClient(RESTFullAPIClient):
                         f"  Error reported: {resp.get('error')}"
                     )
                 break
-            sleep(0.1)
+            yield {"status": "validating"}
         lgr.debug("Assigning asset blob to dandiset & version")
+        yield {"status": "producing asset"}
         self.post(
             f"/dandisets/{dandiset_id}/versions/{version_id}/assets/",
             json={"path": asset_path, "metadata": asset_metadata, "sha256": filehash},
         )
+        yield {"status": "done"}
 
     def create_dandiset(self, name, metadata):
         return self.post("/dandisets/", json={"name": name, "metadata": metadata})
@@ -429,3 +507,15 @@ class DandiAPIClient(RESTFullAPIClient):
             self.download_asset(
                 dandiset_id, version, a["uuid"], filepath, chunk_size=chunk_size
             )
+
+    def get_asset_bypath(self, dandiset_id, version, asset_path):
+        try:
+            # Weed out any assets that happen to have the given path as a
+            # proper prefix:
+            (asset,) = (
+                a
+                for a in self.get_dandiset_assets(dandiset_id, version, path=asset_path)
+                if a["path"] == asset_path
+            )
+        except ValueError:
+            return None
diff --git a/dandi/dandiarchive.py b/dandi/dandiarchive.py
index 65c5f460..1c53aa6e 100644
--- a/dandi/dandiarchive.py
+++ b/dandi/dandiarchive.py
@@ -41,18 +41,20 @@ def navigate_url(url):
 
     # We could later try to "dandi_authenticate" if run into permission issues.
     # May be it could be not just boolean but the "id" to be used?
-    # TODO: remove whenever API starts to support drafts in an unknown version
-    if server_type == "api" and asset_id.get("version") == "draft":
+    if server_type == "girder":
         asset_id, asset_type, client, server_type = _map_to_girder(url)
         args = asset_id, asset_type
-    elif server_type == "girder":
-        client = girder.get_client(
-            server_url, authenticate=False, progressbars=True  # TODO: redo all this
-        )
-        args = asset_id, asset_type
     elif server_type == "api":
         client = DandiAPIClient(server_url)
-        args = (asset_id["dandiset_id"], asset_id["version"], asset_id.get("location"))
+        args = (asset_id["dandiset_id"], asset_id["version"])
+        if asset_id.get("location"):
+            # API interface was RFed in 6ba45daf7c00d6cbffd33aed91a984ad28419f56
+            # and no longer takes "location" kwarg. There is `get_dandiset_assets`
+            # but it doesn't provide dandiset... review the use of navigate_url
+            # to see if we should keep its interface as is and provide dandiset...
+            raise NotImplementedError(
+                "No support for path specific handling via API yet"
+            )
     else:
         raise NotImplementedError(
             f"Download from server of type {server_type} is not yet implemented"
@@ -67,9 +69,7 @@ def navigate_url(url):
 
 def _map_to_girder(url):
     """
-    "draft" datasets are not yet supported through our DANDI API. So we need to
-    perform special handling for now: discover girder_id for it and then proceed
-    with girder
+    discover girder_id for a draft dataset
     """
     # This is a duplicate call from above but it is cheap, so decided to just redo
     # it here instead of passing all the variables + url
@@ -105,7 +105,9 @@ class _dandi_url_parser:
     id_regex = "[a-f0-9]{24}"
     id_grp = f"(?P<id>{id_regex})"
     dandiset_id_grp = "(?P<dandiset_id>[0-9]{6})"
-    server_grp = "(?P<server>(?P<protocol>https?)://(?P<hostname>[^/]+)/)"
+    server_grp = (
+        "(?P<server>(?P<protocol>https?)://(?P<hostname>[^/]+)/(api/)?)"
+    )  # should absorb port and api/
     known_urls = {
         # Those we first redirect and then handle the redirected URL
         # TODO: Later should better conform to our API, so we could allow
@@ -114,33 +116,39 @@ class _dandi_url_parser:
         #   - 'pass' - would continue with original url if no redirect happen
         #   - 'only' - would interrupt if no redirection happens
         # server_type:
-        #   - 'girder' - the default/old
-        #   - 'api' - the "new" (as of 20200715 state of various PRs)
+        #   - 'girder' - underlying requests should go to girder server
+        #   - 'api' - the "new" API service
         # rewrite:
         #   - callable -- which would rewrite that "URI"
         "DANDI:": {"rewrite": lambda x: "https://identifiers.org/" + x},
         "https?://dandiarchive.org/.*": {"handle_redirect": "pass"},
         "https?://identifiers.org/DANDI:.*": {"handle_redirect": "pass"},
+        # New DANDI API, ATM can be reached only via enable('DJANGO_API')  in browser console
+        # https://gui.dandiarchive.org/#/dandiset/000001/0.201104.2302/files
+        # TODO: upload something to any dandiset to see what happens when there are files
+        # and adjust for how path is provided (if not ?location=)
+        f"(?P<server>(?P<protocol>https?)://(?P<hostname>gui-beta-dandiarchive-org.netlify.app)/)"
+        f"#/(?P<asset_type>dandiset)/{dandiset_id_grp}"
+        "(/(?P<version>([.0-9]{5,}|draft)))?"
+        f"(/files(\\?location=(?P<location>.*)?)?)?"
+        "$": {"server_type": "api"},
+        #
+        # PRs are also on netlify - so above takes precedence. TODO: make more specific?
         "https?://[^/]*dandiarchive-org.netlify.app/.*": {"map_instance": "dandi"},
-        # Girder-inflicted urls to folders etc based on the IDs
-        # For those we will completely ignore domain - it will be "handled"
-        f"{server_grp}#.*/(?P<asset_type>folder|collection|dandiset)/{id_grp}$": {},
-        # Nothing special
-        # Multiple items selected - will need custom handling of 'multiitem'
-        f"{server_grp}#/folder/{id_regex}/selected(?P<multiitem>(/item\\+{id_grp})+)$": {},
-        # Direct girder urls to items
-        f"{server_grp}api/v1/(?P<asset_type>item)/{id_grp}/download$": {},
-        # New DANDI API
-        # https://deploy-preview-341--gui-dandiarchive-org.netlify.app/#/dandiset/000006/0.200714.1807
-        # https://deploy-preview-341--gui-dandiarchive-org.netlify.app/#/dandiset/000006/0.200714.1807/files
-        # https://deploy-preview-341--gui-dandiarchive-org.netlify.app/#/dandiset/000006/0.200714.1807/files?location=%2Fsub-anm369962%2F
-        # But for drafts files navigator it is a different beast:
+        #
+        # Direct urls to our new API
+        f"{server_grp}"
+        f"(?P<asset_type>dandiset)s/{dandiset_id_grp}/?"
+        "(versions(/(?P<version>([.0-9]{5,}|draft)))?)?"
+        "$": {"server_type": "api"},
+        # But for drafts files navigator it is a bit different beast and there
+        # could be no versions, only draft
         # https://deploy-preview-341--gui-dandiarchive-org.netlify.app/#/dandiset/000027/draft/files?_id=5f176583f63d62e1dbd06943&_modelType=folder
-        f"{server_grp}#.*/(?P<asset_type>dandiset)/{dandiset_id_grp}"
-        "/(?P<version>([.0-9]{5,}|draft))"
-        "(/files(\\?location=(?P<location>.*)?)?)?"
+        f"{server_grp}"
+        f"#/(?P<asset_type>dandiset)/{dandiset_id_grp}"
+        "(/(?P<version>draft))?"
         f"(/files(\\?_id={id_grp}(&_modelType=folder)?)?)?"
-        "$": {"server_type": "api"},
+        "$": {"server_type": "girder"},
         # https://deploy-preview-341--gui-dandiarchive-org.netlify.app/#/dandiset/000006/draft
         # (no API yet)
         "https?://.*": {"handle_redirect": "only"},
@@ -149,11 +157,19 @@ class _dandi_url_parser:
     map_asset_types = {"dandiset": "folder"}
     # And lets create our mapping into girder instances from known_instances:
     map_to = {"girder": {}, "api": {}}
-    for girder, gui, redirector, api in known_instances.values():  # noqa: F402
+    for (
+        metadata_version,
+        girder,  # noqa: F402
+        gui,
+        redirector,
+        api,
+    ) in known_instances.values():
         for h in (gui, redirector):
             if h:
-                map_to["girder"][h] = girder
-                map_to["api"][h] = api
+                if girder:
+                    map_to["girder"][h] = girder
+                if api:
+                    map_to["api"][h] = api
 
     @classmethod
     def parse(cls, url, *, map_instance=True):
@@ -162,24 +178,17 @@ class _dandi_url_parser:
         Example URLs (as of 20200310):
         - User public: (Users -> bendichter/Public/Tolias2020)
           [seems to be visible only if logged in]
-          https://gui.dandiarchive.org/#/folder/5e5593cc1a343161ff7c5a92
-          https://girder.dandiarchive.org/#user/5da4b8fe51c340795cb18fd0/folder/5e5593cc1a343161ff7c5a92
+          old (girder inflicted): https://gui.dandiarchive.org/#/folder/5e5593cc1a343161ff7c5a92
         - Collection top level (Collections -> yarik):
           https://gui.dandiarchive.org/#/collection/5daa5ca7e3489855a3027682
-          https://girder.dandiarchive.org/#collection/5daa5ca7e3489855a3027682
         - Collections: (Collections -> yarik/svoboda)
-          https://gui.dandiarchive.org/#/folder/5dab0830f377535c7d96c2b4
-          https://girder.dandiarchive.org/#collection/5daa5ca7e3489855a3027682/folder/5dab0830f377535c7d96c2b4
+          old (girder inflicted): https://gui.dandiarchive.org/#/folder/5dab0830f377535c7d96c2b4
         - Dataset landing page metadata
-          https://gui.dandiarchive.org/#/dandiset/5e6d5c6976569eb93f451e4f
+          old (girder inflicted): https://gui.dandiarchive.org/#/dandiset/5e6d5c6976569eb93f451e4f
+          now (20210119): https://gui.dandiarchive.org/#/dandiset/000003
 
         Individual and multiple files:
           - dandi???
-          - girder -- we don't support:
-            https://girder.dandiarchive.org/api/v1/item/5dab0972f377535c7d96c392/download
-          - gui.: support single or multiple
-            # if there is a selection, we could get multiple items
-            https://gui.dandiarchive.org/#/folder/5e60c14f81bc3e47d94aa012/selected/item+5e60c19381bc3e47d94aa014
 
         Multiple selected files + folders -- we do not support ATM, then further
         RFing would be due, probably making this into a generator or returning a
@@ -264,51 +273,38 @@ class _dandi_url_parser:
         if not server.endswith("/"):
             server += "/"  # we expected '/' to be there so let it be
 
-        if server_type == "girder":
-            if "multiitem" not in groups:
-                # we must be all set
-                asset_ids = [groups["id"]]
-                asset_type = groups["asset_type"]
-                asset_type = cls.map_asset_types.get(asset_type, asset_type)
+        asset_type = groups.get("asset_type")
+        dandiset_id = groups.get("dandiset_id")
+        version = groups.get("version")
+        location = groups.get("location")
+        if location:
+            location = urlunquote(location)
+            # ATM carries leading '/' which IMHO is not needed/misguiding somewhat, so
+            # I will just strip it
+            location = location.lstrip("/")
+        if not (asset_type == "dandiset" and dandiset_id):
+            raise ValueError(f"{url} does not point to a dandiset")
+        if not version:
+            version = "draft"
+            # TODO: verify since web UI might have different opinion: it does show e.g.
+            # released version for 000001 now, but that one could be produced only from draft
+            # so we should be ok.  Otherwise we should always then query "dandiset_read" endpoint
+            # to figure out what is the "most recent one"
+        # Let's just return a structured record for the requested asset
+        asset_ids = {"dandiset_id": dandiset_id, "version": version}
+        # if location is not degenerate -- it would be a folder or a file
+        if location:
+            if location.endswith("/"):
+                asset_type = "folder"
             else:
-                # we need to split/parse them and return a list
-                asset_ids = [
-                    i.split("+")[1] for i in groups["multiitem"].split("/") if i
-                ]
                 asset_type = "item"
-        elif server_type == "api":
-            asset_type = groups.get("asset_type")
-            dandiset_id = groups.get("dandiset_id")
-            version = groups.get("version")
-            location = groups.get("location")
-            if location:
-                location = urlunquote(location)
-                # ATM carries leading '/' which IMHO is not needed/misguiding somewhat, so
-                # I will just strip it
-                location = location.lstrip("/")
-            if not (asset_type == "dandiset" and dandiset_id):
-                raise ValueError(f"{url} does not point to a dandiset")
-            if not version:
-                raise NotImplementedError(
-                    f"{url} does not point to a specific version (or draft). DANDI ppl should "
-                    f"decide what should be a behavior in such cases"
-                )
-            # Let's just return a structured record for the requested asset
-            asset_ids = {"dandiset_id": dandiset_id, "version": version}
-            # if location is not degenerate -- it would be a folder or a file
-            if location:
-                if location.endswith("/"):
-                    asset_type = "folder"
-                else:
-                    asset_type = "item"
-                asset_ids["location"] = location
-            # TODO: remove whenever API supports "draft" and this type of url
-            if groups.get("id"):
-                assert version == "draft"
-                asset_ids["folder_id"] = groups["id"]
-                asset_type = "folder"
-        else:
-            raise RuntimeError(f"must not happen. We got {server_type}")
+            asset_ids["location"] = location
+        # TODO: remove whenever API supports "draft" and this type of url
+        if groups.get("id"):
+            assert version == "draft"
+            asset_ids["folder_id"] = groups["id"]
+            asset_type = "folder"
+
         ret = server_type, server, asset_type, asset_ids
         lgr.debug("Parsed into %s", ret)
         return ret
diff --git a/dandi/dandiset.py b/dandi/dandiset.py
index 478fd8f7..f98c5489 100644
--- a/dandi/dandiset.py
+++ b/dandi/dandiset.py
@@ -92,11 +92,23 @@ class Dandiset(object):
         # and from published versions, we will just test both locations
         id_ = metadata.get("dandiset", {}).get("identifier")
         if id_:
+            # very old but might still be present... TODO: API-migration-remove
             lgr.debug("Found identifier %s in 'dandiset.identifier'", id_)
 
         if not id_ and "identifier" in metadata:
+            # girder-based, used before migration to API  TODO: API-migration-remove
             id_ = metadata["identifier"]
-            lgr.debug("Found identifier %s in top level 'identifier'", id_)
+            lgr.debug("Found identifier %s in top level 'identifier'", str(id_))
+
+        if isinstance(id_, dict):
+            # New formalized model.
+            # TODO: add schemaVersion handling
+            if id_.get("propertyID") != "DANDI":
+                raise ValueError(
+                    f"Got following identifier record when was expecting a record "
+                    f"with 'propertyID: DANDI': {id_}"
+                )
+            id_ = str(id_.get("value", ""))
 
         return id_
 
diff --git a/dandi/girder.py b/dandi/girder.py
index 66c3baa1..640f25fb 100644
--- a/dandi/girder.py
+++ b/dandi/girder.py
@@ -142,7 +142,7 @@ class GirderCli(gcl.GirderClient):
 
     def dandi_authenticate(self):
         # Shortcut for advanced folks
-        api_key = os.environ.get("DANDI_API_KEY", None)
+        api_key = os.environ.get("DANDI_GIRDER_API_KEY", None)
         if api_key:
             self.authenticate(apiKey=api_key)
             lgr.debug("Successfully authenticated using the key from the envvar")
diff --git a/dandi/upload.py b/dandi/upload.py
index 3bf79148..54c3b6b3 100644
--- a/dandi/upload.py
+++ b/dandi/upload.py
@@ -1,4 +1,9 @@
 from datetime import datetime
+import json
+import os.path
+
+# PurePosixPath to be cast to for paths on girder
+from pathlib import Path, PurePosixPath
 import re
 import sys
 import time
@@ -26,8 +31,7 @@ def upload(
     allow_any_path=False,
     devel_debug=False,
 ):
-    # PurePosixPath to be cast to for paths on girder
-    from pathlib import Path, PurePosixPath
+    from . import girder
     from .dandiset import Dandiset
     from .support.digests import Digester
 
@@ -41,6 +45,22 @@ def upload(
     # Should no longer be needed
     # dandiset_path = Path(dandiset_path).resolve()
 
+    instance = get_instance(dandi_instance)
+    if instance.girder is None:
+        assert instance.api is not None
+        return _new_upload(
+            instance.api,
+            dandiset,
+            paths,
+            existing,
+            validation,
+            dandiset_path,
+            allow_any_path,
+            devel_debug,
+        )
+
+    client = girder.get_client(instance.girder)
+
     # Girder side details:
 
     if not girder_collection:
@@ -70,7 +90,6 @@ def upload(
         )
 
     import multiprocessing
-    from . import girder
     from .pynwb_utils import ignore_benign_pynwb_warnings, get_object_id
     from .metadata import get_metadata
     from .validate import validate_file
@@ -80,8 +99,6 @@ def upload(
 
     ignore_benign_pynwb_warnings()  # so validate doesn't whine
 
-    client = girder.get_client(get_instance(dandi_instance).girder)
-
     try:
         collection_rec = girder.ensure_collection(client, girder_collection)
     except girder.gcl.HttpError as exc:
@@ -573,3 +590,275 @@ def upload(
     #     lgr.error("Following errors were detected while uploading")
     #     for e, paths in errors.items():
     #         lgr.error(" %s: %d paths", e, len(paths))
+
+
+def _new_upload(
+    api_url,
+    dandiset,
+    paths,
+    existing,
+    validation,
+    dandiset_path,
+    allow_any_path,
+    devel_debug,
+):
+    from .dandiapi import DandiAPIClient
+    from .support.digests import Digester
+
+    client = DandiAPIClient(api_url)
+    client.dandi_authenticate()
+
+    ds_identifier = dandiset.identifier
+    # this is a path not a girder id
+
+    if not re.match(dandiset_identifier_regex, str(ds_identifier)):
+        raise ValueError(
+            f"Dandiset identifier {ds_identifier} does not follow expected "
+            f"convention {dandiset_identifier_regex!r}.  Use "
+            f"'dandi register' to get a legit identifier"
+        )
+
+    from .pynwb_utils import ignore_benign_pynwb_warnings
+    from .metadata import nwb2asset
+    from .validate import validate_file
+    from .utils import find_dandi_files, find_files, path_is_subpath
+    from .support.pyout import naturalsize
+
+    ignore_benign_pynwb_warnings()  # so validate doesn't whine
+
+    #
+    # Treat paths
+    #
+    if not paths:
+        paths = [dandiset.path]
+
+    # Expand and validate all paths -- they should reside within dandiset
+    paths = find_files(".*", paths) if allow_any_path else find_dandi_files(paths)
+    paths = list(map(Path, paths))
+    npaths = len(paths)
+    lgr.info(f"Found {npaths} files to consider")
+    for path in paths:
+        if not (
+            allow_any_path
+            or path.name == dandiset_metadata_file
+            or path.name.endswith(".nwb")
+        ):
+            raise NotImplementedError(
+                f"ATM only .nwb and dandiset.yaml should be in the paths to upload. Got {path}"
+            )
+        if not path_is_subpath(str(path.absolute()), dandiset.path):
+            raise ValueError(f"{path} is not under {dandiset.path}")
+
+    # We will keep a shared set of "being processed" paths so
+    # we could limit the number of them until
+    #   https://github.com/pyout/pyout/issues/87
+    # properly addressed
+    process_paths = set()
+    from collections import defaultdict
+
+    uploaded_paths = defaultdict(lambda: {"size": 0, "errors": []})
+
+    def skip_file(msg):
+        return {"status": "skipped", "message": str(msg)}
+
+    # TODO: we might want to always yield a full record so no field is not
+    # provided to pyout to cause it to halt
+    def process_path(path, relpath):
+        """
+
+        Parameters
+        ----------
+        path: Path
+          Non Pure (OS specific) Path
+        relpath:
+          For location on server.  Will be cast to PurePosixPath
+
+        Yields
+        ------
+        dict
+          Records for pyout
+        """
+        # Ensure consistent types
+        path = Path(path)
+        relpath = PurePosixPath(relpath)
+        try:
+            try:
+                path_stat = path.stat()
+                yield {"size": path_stat.st_size}
+            except FileNotFoundError:
+                yield skip_file("ERROR: File not found")
+                return
+            except Exception as exc:
+                # without limiting [:50] it might cause some pyout indigestion
+                yield skip_file("ERROR: %s" % str(exc)[:50])
+                return
+
+            #
+            # Compute checksums and possible other digests (e.g. for s3, ipfs - TODO)
+            #
+            yield {"status": "digesting"}
+            try:
+                # TODO: in theory we could also cache the result, but since it is
+                # critical to get correct checksums, safer to just do it all the time.
+                # Should typically be faster than upload itself ;-)
+                digester = Digester(["sha256"])
+                sha256_digest = digester(path)["sha256"]
+            except Exception as exc:
+                yield skip_file("failed to compute digests: %s" % str(exc))
+                return
+
+            extant = client.get_asset_bypath(ds_identifier, "draft", relpath)
+            if extant is not None and extant["sha256"] == sha256_digest:
+                if existing == "error":
+                    # as promised -- not gentle at all!
+                    raise FileExistsError("file exists")
+                if existing == "skip":
+                    yield skip_file("file exists")
+                    return
+                # Logic below only for overwrite and reupload
+                if existing == "overwrite":
+                    if extant["sha256"] == sha256_digest:
+                        yield skip_file("file exists")
+                        return
+                elif existing == "refresh":
+                    pass
+                elif existing == "force":
+                    pass
+                else:
+                    raise ValueError("existing")
+
+            #
+            # Validate first, so we do not bother server at all if not kosher
+            #
+            # TODO: enable back validation of dandiset.yaml
+            if path.name != dandiset_metadata_file and validation != "skip":
+                yield {"status": "validating"}
+                validation_errors = validate_file(path)
+                yield {"errors": len(validation_errors)}
+                # TODO: split for dandi, pynwb errors
+                if validation_errors:
+                    if validation == "require":
+                        yield skip_file("failed validation")
+                        return
+                else:
+                    yield {"status": "validated"}
+            else:
+                # yielding empty causes pyout to get stuck or crash
+                # https://github.com/pyout/pyout/issues/91
+                # yield {"errors": '',}
+                pass
+
+            #
+            # Special handling for dandiset.yaml
+            # Yarik hates it but that is life for now. TODO
+            #
+            if path.name == dandiset_metadata_file:
+                # TODO This is a temporary measure to avoid breaking web UI
+                # dandiset metadata schema assumptions.  All edits should happen
+                # online.
+                yield skip_file("should be edited online")
+                return
+
+            #
+            # Extract metadata - delayed since takes time, but is done before
+            # actual upload, so we could skip if this fails
+            #
+            # Extract metadata before actual upload and skip if fails
+            # TODO: allow for for non-nwb files to skip this step
+            # ad-hoc for dandiset.yaml for now
+            yield {"status": "extracting metadata"}
+            try:
+                asset_metadata = nwb2asset(
+                    path, digest=sha256_digest, digest_type="SHA256"
+                )
+            except Exception as exc:
+                if allow_any_path:
+                    yield {"status": "failed to extract metadata"}
+                    metadata = {
+                        "contentSize": os.path.getsize(path),
+                        "digest": sha256_digest,
+                        "digest_type": "SHA256",
+                        # "encodingFormat": # TODO
+                    }
+                else:
+                    yield skip_file("failed to extract metadata: %s" % str(exc))
+                    return
+            else:
+                # We need to convert to a `dict` this way instead of with
+                # `.dict()` so that enums will be converted to strings.
+                metadata = json.loads(
+                    asset_metadata.json(exclude_unset=True, exclude_none=True)
+                )
+
+            #
+            # Upload file
+            #
+            yield {"status": "uploading"}
+            for r in client.iter_upload(
+                ds_identifier, "draft", str(relpath), metadata, str(path)
+            ):
+                if r["status"] == "uploading":
+                    uploaded_paths[str(path)]["size"] = r["current"]
+                yield r
+            yield {"status": "done"}
+
+        except Exception as exc:
+            if devel_debug:
+                raise
+            # Custom formatting for some exceptions we know to extract
+            # user-meaningful message
+            message = str(exc)
+            uploaded_paths[str(path)]["errors"].append(message)
+            yield {"status": "ERROR", "message": message}
+        finally:
+            process_paths.remove(str(path))
+
+    # We will again use pyout to provide a neat table summarizing our progress
+    # with upload etc
+    import pyout
+    from .support import pyout as pyouts
+
+    # for the upload speeds we need to provide a custom  aggregate
+    t0 = time.time()
+
+    def upload_agg(*ignored):
+        dt = time.time() - t0
+        total = sum(v["size"] for v in uploaded_paths.values())
+        if not total:
+            return ""
+        speed = total / dt if dt else 0
+        return "%s/s" % naturalsize(speed)
+
+    pyout_style = pyouts.get_style(hide_if_missing=False)
+    pyout_style["upload"]["aggregate"] = upload_agg
+
+    rec_fields = ["path", "size", "errors", "upload", "status", "message"]
+    out = pyout.Tabular(style=pyout_style, columns=rec_fields)
+
+    with out, client.session():
+        for path in paths:
+            while len(process_paths) >= 10:
+                lgr.log(2, "Sleep waiting for some paths to finish processing")
+                time.sleep(0.5)
+
+            rec = {"path": str(path)}
+            process_paths.add(str(path))
+
+            try:
+                relpath = path.absolute().relative_to(dandiset.path)
+
+                rec["path"] = str(relpath)
+                if devel_debug:
+                    # DEBUG: do serially
+                    for v in process_path(path, relpath):
+                        print(str(v), flush=True)
+                else:
+                    rec[tuple(rec_fields[1:])] = process_path(path, relpath)
+            except ValueError as exc:
+                if "does not start with" in str(exc):
+                    # if top_path is not the top path for the path
+                    # Provide more concise specific message without path details
+                    rec.update(skip_file("must be a child of top path"))
+                else:
+                    rec.update(skip_file(exc))
+            out(rec)
diff --git a/dandi/utils.py b/dandi/utils.py
index 25f4441c..2107249b 100644
--- a/dandi/utils.py
+++ b/dandi/utils.py
@@ -617,6 +617,7 @@ def get_instance(dandi_instance_id):
     if our_version in bad_versions:
         raise BadCliVersionError(our_version, minversion, bad_versions)
     return dandi_instance(
+        metadata_version=0,
         girder=server_info["services"]["girder"]["url"],
         gui=server_info["services"]["webui"]["url"],
         redirector=redirector_url,


Test patch:
diff --git a/dandi/tests/fixtures.py b/dandi/tests/fixtures.py
index b0339c44..538e0293 100644
--- a/dandi/tests/fixtures.py
+++ b/dandi/tests/fixtures.py
@@ -147,15 +147,15 @@ LOCAL_DOCKER_ENV = LOCAL_DOCKER_DIR.name
 
 
 @pytest.fixture(scope="session")
-def local_docker_compose():
-    instance_id = "local-docker-tests"
-    instance = known_instances[instance_id]
+def docker_compose_setup():
+    GIRDER_URL = known_instances["local-docker-tests"].girder
+    API_URL = known_instances["dandi-api-local-docker-tests"].api
 
     # if api_key is specified, we are reusing some already running instance
     # so we would not bother starting/stopping a new one here
     api_key = os.environ.get("DANDI_REUSE_LOCAL_DOCKER_TESTS_API_KEY")
     if api_key:
-        yield {"api_key": api_key, "instance": instance, "instance_id": instance_id}
+        yield {"api_key": api_key}
         return
 
     skipif.no_network()
@@ -176,7 +176,7 @@ def local_docker_compose():
         # Should we check that the output of `docker wait` is 0?
 
         r = requests.get(
-            f"{instance.girder}/api/v1/user/authentication", auth=("admin", "letmein")
+            f"{GIRDER_URL}/api/v1/user/authentication", auth=("admin", "letmein")
         )
         r.raise_for_status()
         initial_api_key = r.json()["authToken"]["token"]
@@ -184,7 +184,7 @@ def local_docker_compose():
         # Get an unscoped/full permissions API key that can be used for
         # uploading:
         r = requests.post(
-            f"{instance.girder}/api/v1/api_key",
+            f"{GIRDER_URL}/api/v1/api_key",
             params={"name": "testkey", "tokenDuration": 1},
             headers={"Girder-Token": initial_api_key},
         )
@@ -193,7 +193,7 @@ def local_docker_compose():
 
         # dandi-publish requires an admin named "publish":
         requests.post(
-            f"{instance.girder}/api/v1/user",
+            f"{GIRDER_URL}/api/v1/user",
             data={
                 "login": "publish",
                 "email": "nil@nil.nil",
@@ -206,8 +206,7 @@ def local_docker_compose():
         ).raise_for_status()
 
         r = requests.get(
-            f"{instance.girder}/api/v1/user/authentication",
-            auth=("publish", "Z1lT4Fh7Kj"),
+            f"{GIRDER_URL}/api/v1/user/authentication", auth=("publish", "Z1lT4Fh7Kj")
         )
         r.raise_for_status()
         publish_api_key = r.json()["authToken"]["token"]
@@ -286,7 +285,7 @@ def local_docker_compose():
         )
 
         requests.put(
-            f"{instance.girder}/api/v1/system/setting",
+            f"{GIRDER_URL}/api/v1/system/setting",
             data={
                 "key": "dandi.publish_api_url",
                 "value": "http://django:8000/api/",  # django or localhost?
@@ -295,14 +294,14 @@ def local_docker_compose():
         ).raise_for_status()
 
         requests.put(
-            f"{instance.girder}/api/v1/system/setting",
+            f"{GIRDER_URL}/api/v1/system/setting",
             data={"key": "dandi.publish_api_key", "value": django_api_key},
             headers={"Girder-Token": publish_api_key},
         ).raise_for_status()
 
         for _ in range(10):
             try:
-                requests.get(f"{instance.api}/dandisets/")
+                requests.get(f"{API_URL}/dandisets/")
             except requests.ConnectionError:
                 sleep(1)
             else:
@@ -310,17 +309,34 @@ def local_docker_compose():
         else:
             raise RuntimeError("Django container did not start up in time")
 
-        yield {
-            "api_key": api_key,
-            "instance": instance,
-            "instance_id": instance_id,
-            "django_api_key": django_api_key,
-        }
+        yield {"girder_api_key": api_key, "django_api_key": django_api_key}
     finally:
         run(["docker-compose", "down", "-v"], cwd=str(LOCAL_DOCKER_DIR), check=True)
 
 
+@pytest.fixture()
+def local_docker_compose(docker_compose_setup):
+    instance_id = "local-docker-tests"
+    instance = known_instances[instance_id]
+    return {
+        "api_key": docker_compose_setup["girder_api_key"],
+        "instance": instance,
+        "instance_id": instance_id,
+    }
+
+
 @pytest.fixture()
 def local_docker_compose_env(local_docker_compose, monkeypatch):
-    monkeypatch.setenv("DANDI_API_KEY", local_docker_compose["api_key"])
+    monkeypatch.setenv("DANDI_GIRDER_API_KEY", local_docker_compose["api_key"])
     return local_docker_compose
+
+
+@pytest.fixture()
+def local_dandi_api(docker_compose_setup):
+    instance_id = "dandi-api-local-docker-tests"
+    instance = known_instances[instance_id]
+    return {
+        "api_key": docker_compose_setup["django_api_key"],
+        "instance": instance,
+        "instance_id": instance_id,
+    }
diff --git a/dandi/tests/test_dandiapi.py b/dandi/tests/test_dandiapi.py
index e3675845..e5be5c86 100644
--- a/dandi/tests/test_dandiapi.py
+++ b/dandi/tests/test_dandiapi.py
@@ -2,10 +2,9 @@ import os.path
 from ..dandiapi import DandiAPIClient
 
 
-def test_upload(local_docker_compose, simple1_nwb, tmp_path):
+def test_upload(local_dandi_api, simple1_nwb, tmp_path):
     client = DandiAPIClient(
-        api_url=local_docker_compose["instance"].api,
-        token=local_docker_compose["django_api_key"],
+        api_url=local_dandi_api["instance"].api, token=local_dandi_api["api_key"]
     )
     with client.session():
         r = client.create_dandiset(name="Upload Test", metadata={})
diff --git a/dandi/tests/test_dandiarchive.py b/dandi/tests/test_dandiarchive.py
index 46c428dc..2531809b 100644
--- a/dandi/tests/test_dandiarchive.py
+++ b/dandi/tests/test_dandiarchive.py
@@ -13,99 +13,78 @@ def _assert_parse_girder_url(url):
     return a, aid
 
 
-def _assert_parse_api_url(url):
+def _assert_parse_api_url(url, instance="dandi-api"):
     st, s, a, aid = parse_dandi_url(url)
     assert st == "api"
-    assert s == known_instances["dandi"].api + "/"
+    assert s == known_instances[instance].api + "/"
     return a, aid
 
 
 def test_parse_dandi_url():
     # ATM we point to drafts, so girder
-    assert _assert_parse_api_url("DANDI:000027") == (
+    assert _assert_parse_girder_url("DANDI:000027") == (
         "dandiset",
         {"dandiset_id": "000027", "version": "draft"},
     )
 
-    # user
-    # we do not care about user -- we care about folder id
+    # Example of current web ui (with girder backend) as of 20210119
     assert _assert_parse_girder_url(
-        "https://girder.dandiarchive.org/"
-        "#user/5da4b8fe51c340795cb18fd0/folder/5e5593cc1a343161ff7c5a92"
-    ) == ("folder", ["5e5593cc1a343161ff7c5a92"])
-    # dandiset
-
-    # folder
-    assert _assert_parse_girder_url(
-        "https://gui.dandiarchive.org/#/folder/5e5593cc1a343161ff7c5a92"
-    ) == ("folder", ["5e5593cc1a343161ff7c5a92"])
-
-    # selected folder(s)
-    # "https://gui.dandiarchive.org/#/folder/5d978d9ecc10d1bc31040bca/selected/folder+5e8672e06cce296e2e817318"
-
-    # selected items and folders
-    # "https://gui.dandiarchive.org/#/folder/5d978d9ecc10d1bc31040bca/selected/item+5e8674dc6cce296e2e8173c5/folder+5e8672e06cce296e2e817318"
-
-    # Selected multiple items
-    assert _assert_parse_girder_url(
-        "https://gui.dandiarchive.org/#/folder/5e7b9e43529c28f35128c745/selected/"
-        "item+5e7b9e44529c28f35128c747/item+5e7b9e43529c28f35128c746"
-    ) == ("item", ["5e7b9e44529c28f35128c747", "5e7b9e43529c28f35128c746"])
-
-    # new (v1? not yet tagged) web UI, and as it comes from a PR,
-    # so we need to provide yet another mapping to stock girder
-    assert _assert_parse_girder_url(
-        "https://refactor--gui-dandiarchive-org.netlify.app/#/file-browser"
-        "/folder/5e9f9588b5c9745bad9f58fe"
-    ) == ("folder", ["5e9f9588b5c9745bad9f58fe"])
-
-    # New DANDI web UI driven by DANDI API.  Again no version assigned/planned!
-    # see https://github.com/dandi/dandiarchive/pull/341
-    url1 = (
-        "https://deploy-preview-341--gui-dandiarchive-org.netlify.app/"
-        "#/dandiset/000006/0.200714.1807"
+        "https://gui.dandiarchive.org/#/dandiset/000003/draft/"
+        "files?_id=5e74ee41368d3c79a8006d29&_modelType=folder"
+    ) == (
+        "folder",
+        {
+            "dandiset_id": "000003",
+            "version": "draft",
+            "folder_id": "5e74ee41368d3c79a8006d29",
+        },
     )
+
+    # New DANDI web UI driven by DANDI API.
+    url1 = "https://gui-beta-dandiarchive-org.netlify.app/#/dandiset/000001"
     assert _assert_parse_api_url(url1) == (
         "dandiset",
-        {"dandiset_id": "000006", "version": "0.200714.1807"},
+        # TODO: in -cli we assume draft, but web ui might be taking the "latest"
+        {"dandiset_id": "000001", "version": "draft"},
     )
-    assert _assert_parse_api_url(url1 + "/files") == (
+    assert _assert_parse_api_url(url1 + "/0.201104.2302") == (
         "dandiset",
-        {"dandiset_id": "000006", "version": "0.200714.1807"},
+        {"dandiset_id": "000001", "version": "0.201104.2302"},
     )
-    assert _assert_parse_api_url(url1 + "/files?location=%2F") == (
+    assert _assert_parse_api_url(url1 + "/0.201104.2302/files") == (
         "dandiset",
-        {"dandiset_id": "000006", "version": "0.200714.1807"},
+        {"dandiset_id": "000001", "version": "0.201104.2302"},
     )
-    assert _assert_parse_api_url(url1 + "/files?location=%2Fsub-anm369962%2F") == (
-        "folder",
-        {
-            "dandiset_id": "000006",
-            "version": "0.200714.1807",
-            "location": "sub-anm369962/",
-        },
-    )
-    # no trailing / - Yarik considers it to be an item (file)
-    assert _assert_parse_api_url(url1 + "/files?location=%2Fsub-anm369962") == (
-        "item",
-        {
-            "dandiset_id": "000006",
-            "version": "0.200714.1807",
-            "location": "sub-anm369962",
-        },
-    )
-    # And the hybrid for "drafts" where it still goes by girder ID
+
     assert _assert_parse_api_url(
-        "https://deploy-preview-341--gui-dandiarchive-org.netlify.app/#/dandiset/000027"
-        "/draft/files?_id=5f176583f63d62e1dbd06943&_modelType=folder"
-    ) == (
-        "folder",
-        {
-            "dandiset_id": "000027",
-            "version": "draft",
-            "folder_id": "5f176583f63d62e1dbd06943",
-        },
-    )
+        "http://localhost:8000/api/dandisets/000002/versions/draft",
+        instance="dandi-api-local-docker-tests",
+    ) == ("dandiset", {"dandiset_id": "000002", "version": "draft"})
+    # TODO: bring it inline with how it would look whenever there is a folder
+    # ATM there is not a single dataset with a folder to see how it looks
+    # no trailing / - Yarik considers it to be an item (file)
+    # assert _assert_parse_api_url(url1 + "/files?location=%2Fsub-anm369962") == (
+    #     "item",
+    #     {
+    #         "dandiset_id": "000006",
+    #         "version": "0.200714.1807",
+    #         "location": "sub-anm369962",
+    #     },
+    # )
+
+    # TODO: bring back a test on deploy-preview-
+    # # And the hybrid for "drafts" where it still goes by girder ID
+    # assert _assert_parse_api_url(
+    #     "https://deploy-preview-341--gui-dandiarchive-org.netlify.app/#/dandiset/000027"
+    #     "/draft/files?_id=5f176583f63d62e1dbd06943&_modelType=folder"
+    # ) == (
+    #     "folder",
+    #     {
+    #         "dandiset_id": "000027",
+    #         "version": "draft",
+    #         "folder_id": "5f176583f63d62e1dbd06943",
+    #     },
+    # )
 
 
 @mark.skipif_no_network
@@ -114,7 +93,7 @@ def test_parse_dandi_url_redirect():
     with pytest.raises(NotFoundError):
         parse_dandi_url("https://dandiarchive.org/dandiset/999999")
     # Is there ATM
-    assert _assert_parse_api_url("https://dandiarchive.org/dandiset/000003") == (
+    assert _assert_parse_girder_url("https://dandiarchive.org/dandiset/000003") == (
         "dandiset",
         {"dandiset_id": "000003", "version": "draft"},
     )
diff --git a/dandi/tests/test_download.py b/dandi/tests/test_download.py
index b6d4e907..79276981 100644
--- a/dandi/tests/test_download.py
+++ b/dandi/tests/test_download.py
@@ -2,71 +2,14 @@ import json
 import os
 import os.path as op
 
-import time
 import tqdm
 
 from ..download import download
-from ..tests.skip import mark
-
-from ..girder import GirderCli, gcl, TQDMProgressReporter
+from ..girder import TQDMProgressReporter
 
 import pytest
 
 
-@mark.skipif_no_network
-def test_download_multiple_files(monkeypatch, tmpdir):
-    url = (
-        "https://gui.dandiarchive.org/#/folder/5e70d3173da50caa9adaf334/selected/"
-        "item+5e70d3173da50caa9adaf335/item+5e70d3183da50caa9adaf336"
-    )
-
-    # In 0.6 RF of download we stopped using girder's downloadFile.
-    # But we still do up to 3 tries also while getting the downloadFileAsIterator,
-    # to this test will test those retries.
-    # While at it we will also test girder downloadFile to retry at least 3 times
-    # in case of some errors, and that it sleeps between retries
-    orig_sendRestRequest = GirderCli.sendRestRequest
-
-    class Mocks:
-        ntries = 0
-        sleeps = 0
-
-        @staticmethod
-        def sendRestRequest(self, *args, **kwargs):
-            if (
-                len(args) > 1
-                and args[1].startswith("file/")
-                and args[1].endswith("/download")
-            ):
-                Mocks.ntries += 1
-                if Mocks.ntries < 3:
-                    raise gcl.HttpError(
-                        text="Failing to download", url=url, method="GET", status=500
-                    )
-            return orig_sendRestRequest(self, *args, **kwargs)
-
-        @staticmethod
-        def sleep(duration):
-            Mocks.sleeps += duration
-            # no actual sleeping
-
-    monkeypatch.setattr(GirderCli, "sendRestRequest", Mocks.sendRestRequest)
-    monkeypatch.setattr(time, "sleep", Mocks.sleep)  # to not sleep in the test
-
-    ret = download(url, tmpdir)
-    assert not ret  # we return nothing ATM, might want to "generate"
-
-    assert Mocks.ntries == 3 + 1  # 3 on the first since 2 fail + 1 on 2nd file
-    assert Mocks.sleeps >= 2  # slept at least 1 sec each time
-
-    downloads = (x.basename for x in tmpdir.listdir())
-    assert sorted(downloads) == [
-        "sub-anm372795_ses-20170714.nwb",
-        "sub-anm372795_ses-20170715.nwb",
-    ]
-    assert all(x.lstat().size > 1e5 for x in tmpdir.listdir())  # all bigish files
-
-
 # both urls point to 000027 (lean test dataset), and both draft and "released"
 # version have only a single file ATM
 @pytest.mark.parametrize(
diff --git a/dandi/tests/test_girder.py b/dandi/tests/test_girder.py
index 0ed4f359..190672e0 100644
--- a/dandi/tests/test_girder.py
+++ b/dandi/tests/test_girder.py
@@ -64,7 +64,7 @@ def test_lock_dandiset_unlock_within(local_docker_compose_env):
 
 
 def test_dandi_authenticate_no_env_var(local_docker_compose_env, monkeypatch, mocker):
-    monkeypatch.delenv("DANDI_API_KEY", raising=False)
+    monkeypatch.delenv("DANDI_GIRDER_API_KEY", raising=False)
     monkeypatch.setenv("PYTHON_KEYRING_BACKEND", "keyring.backends.null.Keyring")
     inputmock = mocker.patch(
         "dandi.girder.input", return_value=local_docker_compose_env["api_key"]
@@ -79,7 +79,7 @@ def test_dandi_authenticate_no_env_var(local_docker_compose_env, monkeypatch, mo
 def test_dandi_authenticate_no_env_var_ask_twice(
     local_docker_compose_env, monkeypatch, mocker
 ):
-    monkeypatch.delenv("DANDI_API_KEY", raising=False)
+    monkeypatch.delenv("DANDI_GIRDER_API_KEY", raising=False)
     monkeypatch.setenv("PYTHON_KEYRING_BACKEND", "keyring.backends.null.Keyring")
     keyiter = iter(["badkey", local_docker_compose_env["api_key"]])
     inputmock = mocker.patch("dandi.girder.input", side_effect=lambda _: next(keyiter))
diff --git a/dandi/tests/test_upload.py b/dandi/tests/test_upload.py
index 1632bae3..073cba9e 100644
--- a/dandi/tests/test_upload.py
+++ b/dandi/tests/test_upload.py
@@ -6,6 +6,7 @@ import responses
 
 from .. import girder
 from ..consts import collection_drafts, dandiset_metadata_file
+from ..dandiapi import DandiAPIClient
 from ..dandiset import Dandiset
 from ..download import download
 from ..register import register
@@ -242,3 +243,28 @@ def test_enormous_upload_breaks_girder(
             bigfile.unlink()
         except FileNotFoundError:
             pass
+
+
+def test_new_upload_download(
+    local_dandi_api, mocker, monkeypatch, organized_nwb_dir, tmp_path
+):
+    client = DandiAPIClient(
+        api_url=local_dandi_api["instance"].api, token=local_dandi_api["api_key"]
+    )
+    with client.session():
+        r = client.create_dandiset("Test Dandiset", {})
+    dandiset_id = r["identifier"]
+    nwb_file, = organized_nwb_dir.glob(f"*{os.sep}*.nwb")
+    (organized_nwb_dir / dandiset_metadata_file).write_text(
+        f"identifier: '{dandiset_id}'\n"
+    )
+    monkeypatch.chdir(organized_nwb_dir)
+    monkeypatch.setenv("DANDI_API_KEY", local_dandi_api["api_key"])
+    upload(paths=[], dandi_instance=local_dandi_api["instance_id"], devel_debug=True)
+    download(
+        f"{local_dandi_api['instance'].api}/dandisets/{dandiset_id}/versions/draft",
+        tmp_path,
+    )
+    nwb_file2, = tmp_path.glob(f"{dandiset_id}{os.sep}*{os.sep}*.nwb")
+    assert nwb_file.name == nwb_file2.name
+    assert nwb_file.parent.name == nwb_file2.parent.name
diff --git a/dandi/tests/test_utils.py b/dandi/tests/test_utils.py
index 4f878f84..19153b1e 100644
--- a/dandi/tests/test_utils.py
+++ b/dandi/tests/test_utils.py
@@ -169,6 +169,7 @@ def test_get_instance_dandi():
         },
     )
     assert get_instance("dandi") == dandi_instance(
+        metadata_version=0,
         girder="https://girder.dandi",
         gui="https://gui.dandi",
         redirector="https://dandiarchive.org",
@@ -194,6 +195,7 @@ def test_get_instance_url():
         },
     )
     assert get_instance("https://example.dandi/") == dandi_instance(
+        metadata_version=0,
         girder="https://girder.dandi",
         gui="https://gui.dandi",
         redirector="https://example.dandi/",