Repository: NeurodataWithoutBorders/nwbinspector (language: python)

Issue:
Add DANDI Inspector API; replace ros3 with remfile
## Motivation

To make a smooth deprecation cycle of `nwbinspector` CLI with removal of ros3, it will need to route to the intended way of running things, so this needs to be implemented first


Discussion / hints:
@stephprince This is ready - the introduction of API functions specific to DANDI streaming, the removal of those parts from the NWB inspection, and the integration of the API functions to replace the previous ones

While this technically removes the usage for ROS3 driver in one fell swoop, the full purge of the `robust_s3_read` is likely best done in an isolated PR
Looks good! Just very minor stylistic comments. `if x is True` is fine if x can take on types other than boolean, but in all of these cases the variable has been typehinted as bool, so we can just use `if x`. Keeping the `if x is True` would to be hint that x is allowed to take a non-boolean value, so I would avoid that pattern if that is not the case.
@stephprince Resolved all comments - ready for another round
> Are the dandi-dev and dandi-dev-live failures related to this PR?

I wondered that too, but have no clue how they possibly could be. Will keep an eye on it when tomorrows dailies run (no cutting of releases until well after then)

I suspect it's an environment resolution issue
## [Codecov](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) Report
Attention: Patch coverage is `88.67925%` with `18 lines` in your changes missing coverage. Please review.
> Project coverage is 84.75%. Comparing base [(`a642a3c`)](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/commit/a642a3ca0e06980c82d894d42fcb609368fd092a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) to head [(`36a5650`)](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/commit/36a5650ae77c5394d60fa347d847c81712335323?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders).

| [Files with missing lines](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) | Patch % | Lines |
|---|---|---|
| [src/nwbinspector/\_nwbinspector\_cli.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2F_nwbinspector_cli.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci9fbndiaW5zcGVjdG9yX2NsaS5weQ==) | 89.18% | [8 Missing :warning: ](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) |
| [src/nwbinspector/\_nwb\_inspection.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2F_nwb_inspection.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci9fbndiX2luc3BlY3Rpb24ucHk=) | 75.00% | [7 Missing :warning: ](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) |
| [src/nwbinspector/\_dandi\_inspection.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2F_dandi_inspection.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci9fZGFuZGlfaW5zcGVjdGlvbi5weQ==) | 98.07% | [1 Missing :warning: ](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) |
| [src/nwbinspector/nwbinspector/\_\_init\_\_.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2Fnwbinspector%2F__init__.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci9ud2JpbnNwZWN0b3IvX19pbml0X18ucHk=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) |
| [src/nwbinspector/testing/\_testing.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2Ftesting%2F_testing.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci90ZXN0aW5nL190ZXN0aW5nLnB5) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) |

<details><summary>Additional details and impacted files</summary>


[![Impacted file tree graph](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490/graphs/tree.svg?width=650&height=150&src=pr&token=772QROR5F0&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders)](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders)

```diff
@@            Coverage Diff             @@
##              dev     #490      +/-   ##
==========================================
+ Coverage   82.16%   84.75%   +2.58%     
==========================================
  Files          47       48       +1     
  Lines        1357     1423      +66     
==========================================
+ Hits         1115     1206      +91     
+ Misses        242      217      -25     
```

| [Flag](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) | `?` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Files with missing lines](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) | Coverage Δ | |
|---|---|---|
| [src/nwbinspector/\_configuration.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2F_configuration.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci9fY29uZmlndXJhdGlvbi5weQ==) | `96.22% <100.00%> (ø)` | |
| [src/nwbinspector/\_dandi\_inspection.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2F_dandi_inspection.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci9fZGFuZGlfaW5zcGVjdGlvbi5weQ==) | `98.07% <98.07%> (ø)` | |
| [src/nwbinspector/nwbinspector/\_\_init\_\_.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2Fnwbinspector%2F__init__.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci9ud2JpbnNwZWN0b3IvX19pbml0X18ucHk=) | `0.00% <0.00%> (ø)` | |
| [src/nwbinspector/testing/\_testing.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2Ftesting%2F_testing.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci90ZXN0aW5nL190ZXN0aW5nLnB5) | `83.63% <66.66%> (ø)` | |
| [src/nwbinspector/\_nwb\_inspection.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2F_nwb_inspection.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci9fbndiX2luc3BlY3Rpb24ucHk=) | `83.70% <75.00%> (ø)` | |
| [src/nwbinspector/\_nwbinspector\_cli.py](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490?src=pr&el=tree&filepath=src%2Fnwbinspector%2F_nwbinspector_cli.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL253Ymluc3BlY3Rvci9fbndiaW5zcGVjdG9yX2NsaS5weQ==) | `89.18% <89.18%> (ø)` | |

... and [2 files with indirect coverage changes](https://app.codecov.io/gh/NeurodataWithoutBorders/nwbinspector/pull/490/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders)

</details>

Fix patch:
diff --git a/.github/workflows/dailies.yml b/.github/workflows/dailies.yml
index 0a579039a..3716ab909 100644
--- a/.github/workflows/dailies.yml
+++ b/.github/workflows/dailies.yml
@@ -16,13 +16,13 @@ jobs:
     uses: neurodatawithoutborders/nwbinspector/.github/workflows/doc-link-checks.yml@dev
 
   test-dandi-latest:
-    uses: neurodatawithoutborders/nwbinspector/.github/workflows/dandi_release.yml@dev
+    uses: neurodatawithoutborders/nwbinspector/.github/workflows/dandi-release.yml@dev
 
   test-dandi-dev:
-    uses: neurodatawithoutborders/nwbinspector/.github/workflows/dandi_dev.yml@dev
+    uses: neurodatawithoutborders/nwbinspector/.github/workflows/dandi-dev.yml@dev
 
   test-dandi-dev-live:
-    uses: neurodatawithoutborders/nwbinspector/.github/workflows/dandi_dev_live_service.yml@dev
+    uses: neurodatawithoutborders/nwbinspector/.github/workflows/dandi-dev-live-services.yml@dev
 
   notify:
     runs-on: ubuntu-latest
diff --git a/.github/workflows/dandi-dev-live-services.yml b/.github/workflows/dandi-dev-live-services.yml
new file mode 100644
index 000000000..0f17fb43a
--- /dev/null
+++ b/.github/workflows/dandi-dev-live-services.yml
@@ -0,0 +1,32 @@
+name: Test against DANDI dev branch - all tests, including live services
+on: workflow_call
+
+jobs:
+  build-and-test:
+    name: Testing against current DANDI release and dev branch
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+
+    steps:
+      - name: Checkout local branch of source repository
+        uses: actions/checkout@v4
+      - run: git fetch --prune --unshallow --tags
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Test against DANDI dev
+        run: |
+          pip install virtualenv
+          virtualenv --system-site-packages --python=python3 venvs/dandi_dev_tests
+          source venvs/dandi_dev_tests/bin/activate
+          pip install -e .[dandi]
+          cd ..
+          git clone https://github.com/dandi/dandi-cli
+          cd dandi-cli
+          pip install -e .[test]
+          pip install hdmf==3.14.3  # temporary: https://github.com/dandi/dandi-cli/issues/1494
+          pytest -vv
diff --git a/.github/workflows/dandi-dev.yml b/.github/workflows/dandi-dev.yml
new file mode 100644
index 000000000..0e7692270
--- /dev/null
+++ b/.github/workflows/dandi-dev.yml
@@ -0,0 +1,35 @@
+name: Test against DANDI dev branch - no network
+on: workflow_call
+
+env:
+  DANDI_TESTS_NONETWORK: "1"
+
+jobs:
+  build-and-test:
+    name: Testing against current DANDI release and dev branch
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+
+    steps:
+      - name: Checkout local branch of source repository
+        uses: actions/checkout@v4
+      - run: git fetch --prune --unshallow --tags
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Test local branch of NWB Inspector against DANDI dev branch
+        run: |
+          pip install virtualenv
+          virtualenv --system-site-packages --python=python3 venvs/dandi_dev_tests
+          source venvs/dandi_dev_tests/bin/activate
+          pip install -e .[dandi]
+          cd ..
+          git clone https://github.com/dandi/dandi-cli
+          cd dandi-cli
+          pip install -e .[test]
+          pip install hdmf==3.14.3  # temporary: https://github.com/dandi/dandi-cli/issues/1494
+          pytest -rsx
diff --git a/.github/workflows/dandi_release.yml b/.github/workflows/dandi-release.yml
similarity index 77%
rename from .github/workflows/dandi_release.yml
rename to .github/workflows/dandi-release.yml
index 7697a71dc..cf7fa9ee5 100644
--- a/.github/workflows/dandi_release.yml
+++ b/.github/workflows/dandi-release.yml
@@ -1,4 +1,4 @@
-name: Test against latest DANDI release - no network
+name: Run NWB Inspector tests against latest DANDI release - no network
 on: workflow_call
 
 env:
@@ -25,7 +25,7 @@ jobs:
           - '3.11'
 
     steps:
-    - name: Set up environment
+    - name: Checkout local branch of source repository
       uses: actions/checkout@v4
       with:
         # Fetch all commits so that versioneer will return something compatible
@@ -43,7 +43,9 @@ jobs:
         pip install "dandi[extras,test]"
 
     - name: Install this branch of NWB Inspector
-      run: pip install -e .
+      run: |
+        pip install -e .
+        pip install hdmf==3.14.3  # temporary: https://github.com/dandi/dandi-cli/issues/1494
 
     - name: Run all NWB Inspector tests
-      run: python -m pytest -s -v
+      run: pytest -rsx
diff --git a/.github/workflows/dandi_dev.yml b/.github/workflows/dandi_dev.yml
deleted file mode 100644
index 315e85fdb..000000000
--- a/.github/workflows/dandi_dev.yml
+++ /dev/null
@@ -1,38 +0,0 @@
-name: Test against DANDI dev branch - no network
-on: workflow_call
-
-env:
-  DANDI_TESTS_NONETWORK: "1"
-
-jobs:
-  build-and-test:
-    name: Testing against current DANDI release and dev branch
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-    steps:
-      - uses: s-weigand/setup-conda@v1
-        with:
-          update-conda: true
-          python-version: 3.9
-          conda-channels: conda-forge
-      - uses: actions/checkout@v3
-      - run: git fetch --prune --unshallow --tags
-
-      - name: Test against DANDI dev
-        run: |
-          pip install virtualenv
-          git clone https://github.com/NeurodataWithoutBorders/nwbinspector
-          git clone https://github.com/dandi/dandi-cli
-          cd dandi-cli
-          # Test against a specific branch or commit if needed
-          # git checkout a50c4fe243f9a3b72fbcecaea095943be02616d6  # error_fix branch with iteration issue
-          virtualenv --system-site-packages --python=python3 venvs/dev3
-          source venvs/dev3/bin/activate
-          pip install -e .[test]
-          cd ../nwbinspector
-          # Test against a specific branch or commit if needed
-          # git checkout add_downstream_ci
-          pip install -e .
-          cd ../dandi-cli
-          pytest -vv
diff --git a/.github/workflows/dandi_dev_live_service.yml b/.github/workflows/dandi_dev_live_service.yml
deleted file mode 100644
index 85f765b14..000000000
--- a/.github/workflows/dandi_dev_live_service.yml
+++ /dev/null
@@ -1,35 +0,0 @@
-name: Test against DANDI dev branch - all tests, including live service
-on: workflow_call
-
-jobs:
-  build-and-test:
-    name: Testing against current DANDI release and dev branch
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-    steps:
-      - uses: s-weigand/setup-conda@v1
-        with:
-          update-conda: true
-          python-version: 3.9
-          conda-channels: conda-forge
-      - uses: actions/checkout@v3
-      - run: git fetch --prune --unshallow --tags
-
-      - name: Test against DANDI dev
-        run: |
-          pip install virtualenv
-          git clone https://github.com/NeurodataWithoutBorders/nwbinspector
-          git clone https://github.com/dandi/dandi-cli
-          cd dandi-cli
-          # Test against a specific branch or commit if needed
-          # git checkout a50c4fe243f9a3b72fbcecaea095943be02616d6  # error_fix branch with iteration issue
-          virtualenv --system-site-packages --python=python3 venvs/dev3
-          source venvs/dev3/bin/activate
-          pip install -e .[test]
-          cd ../nwbinspector
-          # Test against a specific branch or commit if needed
-          # git checkout add_downstream_ci
-          pip install -e .
-          cd ../dandi-cli
-          pytest -vv


Test patch:
diff --git a/.github/workflows/deploy-tests.yml b/.github/workflows/deploy-tests.yml
index b8de4531b..38505e729 100644
--- a/.github/workflows/deploy-tests.yml
+++ b/.github/workflows/deploy-tests.yml
@@ -3,13 +3,12 @@ name: Deploy tests
 on:
   pull_request:
 
-jobs:
-  cancel-previous-runs:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Cancel Previous Runs
-        uses: styfle/cancel-workflow-action@0.11.0
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
 
+
+jobs:
   assess-file-changes:
     uses: catalystneuro/neuroconv/.github/workflows/assess-file-changes.yml@main
 
@@ -58,17 +57,17 @@ jobs:
   test-dandi-latest:
     needs: assess-file-changes
     if: ${{ needs.assess-file-changes.outputs.SOURCE_CHANGED == 'true' }}
-    uses: ./.github/workflows/dandi_release.yml
+    uses: ./.github/workflows/dandi-release.yml
 
   test-dandi-dev:
     needs: assess-file-changes
     if: ${{ needs.assess-file-changes.outputs.SOURCE_CHANGED == 'true' }}
-    uses: ./.github/workflows/dandi_dev.yml
+    uses: ./.github/workflows/dandi-dev.yml
 
   test-dandi-dev-live:
     needs: assess-file-changes
     if: ${{ needs.assess-file-changes.outputs.SOURCE_CHANGED == 'true' }}
-    uses: ./.github/workflows/dandi_dev_live_service.yml
+    uses: ./.github/workflows/dandi-dev-live-services.yml
 
   check-final-status:
     name: All tests passing