Repository: datalad/datalad

Issue:
eval_func() leads to one `git-config` call per cmd call
This is not a proper analysis, just recording a realization.

In order to figure out whether there are any result hooks, `eval_func()` consults a `ConfigManager` instance. That in itself would not be an issue. What is an issue is that this instance is create anew for each call to `eval_func()`. That means its `reload()` is called with `force=True` leading to an unconditional `git-config` call, regardless of whether the last one was milliseconds ago. There is no mtime checking, no caching.

Two directions to mitigate come to my mind:

1. ConfigManager singletons
2. Have two instances as dataset members. One  for the full config, and one for non-committed config only.

Discussion:
yarikoptic: FWIW: common gotcha from travis (below is just a sample failed tests among a good number):
```
======================================================================
ERROR: datalad.tests.test_tests_utils.test_probe_known_failure
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/datalad/tests/test_tests_utils.py", line 633, in test_probe_known_failure
    with patch_config({'datalad.tests.knownfailures.probe': True}):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/datalad/tests/utils.py", line 1921, in patch_config
    return patch.dict(dl_cfg._store, vars)
AttributeError: 'ConfigManager' object has no attribute '_store'
```
mih: So this seems to be functional in general. In order to get #4794 incorporated as well, something like the following patch should be sufficient:

```diff
diff --git a/datalad/config.py b/datalad/config.py
index bf921f16c..92223a584 100644
--- a/datalad/config.py
+++ b/datalad/config.py
@@ -9,6 +9,7 @@
 """
 """
 
+from concurrent.futures import ThreadPoolExecutor
 from fasteners import InterProcessLock
 from functools import lru_cache
 import datalad
@@ -279,9 +280,10 @@ class ConfigManager(object):
                 else run_args
 
         # reload everything that was found todo
-        while to_run:
-            store_id, runargs = to_run.popitem()
-            self._stores[store_id] = self._reload(runargs)
+        with ThreadPoolExecutor(max_workers=len(to_run)) as executor:
+            store_ids = list(to_run.keys())
+            res = executor.map(self._reload, (to_run[k] for k in store_ids))
+            self._stores = dict(zip(store_ids, res))
 
         # always update the merged representation, even if we did not reload
         # anything from a file. ENV or overrides could change independently
```

However, that just hangs. If I am using the same approach as in #4794, adjusted for this PR, I get the Python segfault that I am already familiar with from previous parallelization attempts.

Any insight would be much appreciated.
codecov[bot]: # [Codecov](https://codecov.io/gh/datalad/datalad/pull/4829?src=pr&el=h1) Report
> Merging [#4829](https://codecov.io/gh/datalad/datalad/pull/4829?src=pr&el=desc) into [master](https://codecov.io/gh/datalad/datalad/commit/8df310629c389f67618406e957978423a89f1cf8?el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `93.81%`.

[![Impacted file tree graph](https://codecov.io/gh/datalad/datalad/pull/4829/graphs/tree.svg?width=650&height=150&src=pr&token=IlkOD0D6oP)](https://codecov.io/gh/datalad/datalad/pull/4829?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4829      +/-   ##
==========================================
- Coverage   89.67%   89.51%   -0.17%     
==========================================
  Files         289      289              
  Lines       40455    40457       +2     
==========================================
- Hits        36278    36214      -64     
- Misses       4177     4243      +66     
```


| [Impacted Files](https://codecov.io/gh/datalad/datalad/pull/4829?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datalad/config.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC9jb25maWcucHk=) | `96.56% <92.77%> (-1.25%)` | :arrow_down: |
| [datalad/core/local/resulthooks.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC9jb3JlL2xvY2FsL3Jlc3VsdGhvb2tzLnB5) | `70.49% <100.00%> (ø)` | |
| [datalad/interface/run\_procedure.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC9pbnRlcmZhY2UvcnVuX3Byb2NlZHVyZS5weQ==) | `90.90% <100.00%> (+0.11%)` | :arrow_up: |
| [datalad/interface/tests/test\_ls\_webui.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC9pbnRlcmZhY2UvdGVzdHMvdGVzdF9sc193ZWJ1aS5weQ==) | `100.00% <100.00%> (ø)` | |
| [datalad/interface/utils.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC9pbnRlcmZhY2UvdXRpbHMucHk=) | `94.11% <100.00%> (-0.05%)` | :arrow_down: |
| [datalad/support/gitrepo.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC9zdXBwb3J0L2dpdHJlcG8ucHk=) | `90.49% <100.00%> (+0.07%)` | :arrow_up: |
| [datalad/tests/test\_config.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC90ZXN0cy90ZXN0X2NvbmZpZy5weQ==) | `100.00% <100.00%> (ø)` | |
| [datalad/tests/utils.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC90ZXN0cy91dGlscy5weQ==) | `87.15% <100.00%> (ø)` | |
| [datalad/customremotes/base.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC9jdXN0b21yZW1vdGVzL2Jhc2UucHk=) | `69.35% <0.00%> (-14.52%)` | :arrow_down: |
| [datalad/customremotes/tests/\_\_init\_\_.py](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree#diff-ZGF0YWxhZC9jdXN0b21yZW1vdGVzL3Rlc3RzL19faW5pdF9fLnB5) | `91.66% <0.00%> (-8.34%)` | :arrow_down: |
| ... and [15 more](https://codecov.io/gh/datalad/datalad/pull/4829/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/datalad/datalad/pull/4829?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/datalad/datalad/pull/4829?src=pr&el=footer). Last update [8df3106...0f74693](https://codecov.io/gh/datalad/datalad/pull/4829?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

mih: I looked into the stalled crawler test by raising the log level. At level 5 I see the stalling (more info below). At level 1 no stalling the test passes :roll_eyes: This smells like an async IO problem again. Here is why:

with log level 5:

```
2020-08-26T09:22:13.7729059Z [DEBUG] Async run ['git', '-c', 'annex.alwayscommit=false', 'annex', 'lookupkey', '-c', 'annex.dotfiles=true', '-c', 'annex.retry=3', '--debug', '--', '1-copy.dat'] 
2020-08-26T09:22:13.7730095Z [DEBUG] Launching process ['git', '-c', 'annex.alwayscommit=false', 'annex', 'lookupkey', '-c', 'annex.dotfiles=true', '-c', 'annex.retry=3', '--debug', '--', '1-copy.dat'] 
2020-08-26T09:22:13.7790392Z [DEBUG] Closing stdin of <subprocess.Popen object at 0x7f4c00471410> and waiting process to finish 
```

this is the last output before the stall. Fractions of a second earlier there was a successful command execution (same command different file):

```
2020-08-26T09:22:12.6683512Z [DEBUG] Async run ['git', '-c', 'annex.alwayscommit=false', 'annex', 'lookupkey', '-c', 'annex.dotfiles=true', '-c', 'annex.retry=3', '--debug', '--', 'sample.txt'] 
2020-08-26T09:22:12.6684546Z [DEBUG] Launching process ['git', '-c', 'annex.alwayscommit=false', 'annex', 'lookupkey', '-c', 'annex.dotfiles=true', '-c', 'annex.retry=3', '--debug', '--', 'sample.txt'] 
2020-08-26T09:22:12.6731619Z [DEBUG] Process 18683 started 
2020-08-26T09:22:12.6733804Z [DEBUG] Waiting for process 18683 to complete 
2020-08-26T09:22:12.7020802Z [Level 5] Read 219 bytes from 18683[stderr] 
2020-08-26T09:22:12.7069603Z [Level 5] Read 57 bytes from 18683[stderr] 
2020-08-26T09:22:12.7070271Z [Level 5] Read 195 bytes from 18683[stderr] 
2020-08-26T09:22:12.7074682Z [Level 5] Read 243 bytes from 18683[stderr] 
2020-08-26T09:22:12.7261366Z [DEBUG] Process 18683 exited with return code 1 
2020-08-26T09:22:12.7266622Z [Level 8] Finished running ['git', '-c', 'annex.alwayscommit=false', 'annex', 'lookupkey', '-c', 'annex.dotfiles=true', '-c', 'annex.retry=3', '--debug', '--', 'sample.txt'] with status 1 
```

I assume that raising to log level 1 causes the call frequency to drop, due to much more processing of output being done.
mih: So after my "fix" here is what the logs say (before I turn the log level back to normal):

```
2020-08-26T11:44:43.1602034Z [DEBUG] Async run ['git', '-c', 'annex.alwayscommit=false', 'annex', 'init', '-c', 'annex.dotfiles=true', '--debug'] 
2020-08-26T11:44:43.1602342Z [DEBUG] Launching process ['git', '-c', 'annex.alwayscommit=false', 'annex', 'init', '-c', 'annex.dotfiles=true', '--debug'] 
2020-08-26T11:44:43.1629761Z [DEBUG] Closing stdin of <subprocess.Popen object at 0x7fe51f32f9d0> and waiting process to finish 
2020-08-26T11:44:46.1652556Z [WARNING] Caught Command '['git', '-c', 'annex.merge-annex-branches=false', 'annex', 'info', '--bytes', '--json', '--json-error-messages', '--batch']' timed out after 3.0 seconds [subprocess.py:_wait:1645] on trial #1. Sleeping 1.000000 and retrying 
2020-08-26T11:44:50.1667437Z [WARNING] Caught Command '['git', '-c', 'annex.merge-annex-branches=false', 'annex', 'info', '--bytes', '--json', '--json-error-messages', '--batch']' timed out after 3.0 seconds [subprocess.py:_wait:1645] on trial #2. Sleeping 1.000000 and retrying 
2020-08-26T11:44:54.1681564Z [WARNING] Batched process <subprocess.Popen object at 0x7fe51f32f9d0> did not finish, abandoning it without killing it 
2020-08-26T11:44:54.1684485Z [DEBUG] Closing stdin of <subprocess.Popen object at 0x7fe51f300450> and waiting process to finish 
2020-08-26T11:44:57.1694455Z [WARNING] Caught Command '['git', '-c', 'annex.merge-annex-branches=false', 'annex', 'info', '--bytes', '--json', '--json-error-messages', '--batch']' timed out after 3.0 seconds [subprocess.py:_wait:1645] on trial #1. Sleeping 1.000000 and retrying 
2020-08-26T11:45:01.1710080Z [WARNING] Caught Command '['git', '-c', 'annex.merge-annex-branches=false', 'annex', 'info', '--bytes', '--json', '--json-error-messages', '--batch']' timed out after 3.0 seconds [subprocess.py:_wait:1645] on trial #2. Sleeping 1.000000 and retrying 
2020-08-26T11:45:05.1723954Z [WARNING] Batched process <subprocess.Popen object at 0x7fe51f300450> did not finish, abandoning it without killing it 
2020-08-26T11:45:05.1733630Z [DEBUG] Process 18882 started 
```

This time it happened twice, not with `lookupkey` but with `info`. I do not think, this has anything to do with the scope of this PR.
kyleam: > In any case, I think this PR has utility even without parallelization .

In my view it'd be better to not tack on that change here.  Ideally that'd come in a dedicated PR with timings so that we have some sense what the added complexity is buying us.
mih: After I figured out #4832 I tried the patch from https://github.com/datalad/datalad/pull/4829#issuecomment-679995627 again. Unfortunately, it still hangs trying to acquire a thread lock.

I then tried a port of @yarikoptic 's original approach, which now also hangs at the same step (instead of segfaulting):

```diff
diff --git a/datalad/config.py b/datalad/config.py
index bf921f16c..37ee64e1e 100644
--- a/datalad/config.py
+++ b/datalad/config.py
@@ -279,9 +279,17 @@ class ConfigManager(object):
                 else run_args
 
         # reload everything that was found todo
-        while to_run:
-            store_id, runargs = to_run.popitem()
-            self._stores[store_id] = self._reload(runargs)
+        if to_run:
+            res = {}
+            with ThreadPoolExecutor(max_workers=len(to_run)) as executor:
+                for k, args in to_run.items():
+                    res[k] = executor.submit(self._reload, args)
+                for r in res.values():
+                    exc = r.exception()
+                    if exc:
+                        raise exc
+                for k, r in res.items():
+                    self._stores[k] = r.result()
 
         # always update the merged representation, even if we did not reload
         # anything from a file. ENV or overrides could change independently
```
mih: Thanks for the reviews @kyleam @bpoldrack !

@kyleam I think I would prefer to keep the RF that establishes the point where parallel execution could happen -- even though I cannot get it to work yet.
yarikoptic: On one hand I am ok with delaying and pretty much ignoring (but then why to delay) failure to close a batched process... 
but on the other -- it seems to be some kind of a regression where changes cause unrelated (? BatchedAnnex is not for config reading right?) process to stall.  May be 'strace' (at the log level appropriate to cause a stall) and looking at that git-annex process -- what is it waiting for? (some lock? not closed file? dunno) could shine some light and inform us.  Otherwise it is not unlikely to resurface in the future in some other fashion and we would spend time chasing it down "afresh" until relating to these changes.  I will try now to reproduce as well and `strace` it... will report on success/fail shortly
yarikoptic: <details>
<summary>Initial observations. TL;DR: two annex info batched processes are out there; disabling coverage avoids stall (racy racy?)</summary> 

- stalls consistently only if I have full invocation as in travis:
```
mkdir -p __testhome__
  cd __testhome__
  python -m nose -s -v --with-cov --cover-package datalad $(echo datalad-crawler | tr '-' '_')
```
  - if I disable coverage - no stall
  - if I point to the test file -- no stall
  - if I specify test via `--tests` -- no stall
- running from upstairs (not `__testhome__`) -- stalls, so unrelated

### stracing

```
$> strace -f -o /tmp/stall.log -ttt python -m nose -s -v --with-cov --cover-package datalad datalad_crawler
datalad_crawler.dbs.tests.test_files.test_AnnexDBs(<class 'datalad_crawler.dbs.files.PhysicalFileStatusesDB'>,) ... ok
datalad_crawler.dbs.tests.test_files.test_AnnexDBs(<class 'datalad_crawler.dbs.files.JsonFileStatusesDB'>,) ... ok
datalad_crawler.nodes.tests.test_annex.test_annex_file('full',) ... 
```

looking at strace:
```shell
$> tail -f stall.log       
317776 1598468686.314884 poll([{fd=16, events=POLLIN}], 1, 500) = 0 (Timeout)
317776 1598468686.815950 poll([{fd=16, events=POLLIN}], 1, 500) = 0 (Timeout)
317776 1598468687.317215 poll([{fd=16, events=POLLIN}], 1, 500) = 0 (Timeout)
317776 1598468687.818528 poll([{fd=16, events=POLLIN}], 1, 500) = 0 (Timeout)
```
and more interesting `ps auxw -H`
```
yoh       316711  2.7  0.0   7652  3772 pts/12   S+   15:03   0:04         strace -f -o /tmp/stall.log -ttt python -m nose -s -v --with-cov --cover-package datalad datalad_crawler
yoh       316714  1.5  0.2 190616 75748 pts/12   S+   15:03   0:02           python -m nose -s -v --with-cov --cover-package datalad datalad_crawler
yoh       317224  0.0  0.0   8824  3548 pts/12   S+   15:03   0:00             /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git -c annex.merge-annex-branches=false annex info --bytes --json --json-error-messages --batch
yoh       317239  0.0  0.1 1074079100 40644 pts/12 Sl+ 15:03   0:00               /usr/lib/git-annex.linux/exe/git-annex --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git-annex/git-annex info --bytes --json --json-error-messages --batch
yoh       317246  0.0  0.0   8900  4248 pts/12   S+   15:03   0:00                 /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
yoh       317247  0.0  0.0   8824  3564 pts/12   S+   15:03   0:00                 /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize)
yoh       317248  0.0  0.0   8896  3956 pts/12   S+   15:03   0:00                 /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
yoh       317249  0.0  0.0   8824  4080 pts/12   S+   15:03   0:00                 /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize)
yoh       317689  0.0  0.0   8824  3568 pts/12   S+   15:03   0:00             /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git -c annex.merge-annex-branches=false annex info --bytes --json --json-error-messages --batch
yoh       317704  0.0  0.1 1074079100 39284 pts/12 Sl+ 15:03   0:00               /usr/lib/git-annex.linux/exe/git-annex --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git-annex/git-annex info --bytes --json --json-error-messages --batch
yoh       317711  0.0  0.0   8900  4100 pts/12   S+   15:03   0:00                 /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
yoh       317712  0.0  0.0   8824  3560 pts/12   S+   15:03   0:00                 /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize)
yoh       317713  0.0  0.0   8824  3560 pts/12   S+   15:03   0:00                 /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
yoh       317714  0.0  0.0   8824  3836 pts/12   S+   15:03   0:00                 /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize)
yoh       317776  0.0  0.1 190708 62564 pts/12   Sl+  15:03   0:00             python -m nose -s -v --with-cov --cover-package datalad datalad_crawler
yoh       317854  0.0  0.0      0     0 pts/12   Z+   15:03   0:00             [git] <defunct>
```

so -- my guess we are waiting on one of those **two**? `git annex info` processes:
```
yoh       317224  0.0  0.0   8824  3548 pts/12   S+   15:03   0:00             /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git -c annex.merge-annex-branches=false annex info --bytes --json --json-error-messages --batch
yoh       317689  0.0  0.0   8824  3568 pts/12   S+   15:03   0:00             /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git -c annex.merge-annex-branches=false annex info --bytes --json --json-error-messages --batch
```
**both** "working" in now deleted temporary different directories so unrelated runs?
```
$> ls -l /proc/{317224,317689}/cwd
lrwxrwxrwx 1 yoh yoh 0 Aug 26 15:08 /proc/317224/cwd -> /home/yoh/.tmp/datalad_temp_tree__test_AnnexDBs4h2bub1\ (deleted)
lrwxrwxrwx 1 yoh yoh 0 Aug 26 15:08 /proc/317689/cwd -> /home/yoh/.tmp/datalad_temp_tree__test_AnnexDB8hzkgm8y\ (deleted)
```
and stracing them doesn't work:
```
lena:/tmp
$> sudo strace -f -p 317689
[sudo] password for yoh: 
strace: attach: ptrace(PTRACE_SEIZE, 317689): Operation not permitted
```

</details>
now I will try to run
yarikoptic: FWIW
- for me doesn't stall with strace + log level 5; racy racy
- it is having first those test_files tests which create those two batched processes we saw, so `python -m nose -s -v --with-cov --cover-package datalad ../datalad_crawler/dbs/tests/test_files.py ../datalad_crawler/nodes/tests/test_annex.py` reproduces the stall
- it is those two annex info batched process which get "cleaned up" only those tests complete (so directory removed) and we get into test_annex_file which (via some `datalad/config.py(284)reload()`) only then causes garbage collection on those batched processes:
<details>
<summary>traceback when failing to wait already in `test_annex_file`</summary> 

```shell
$> python -m nose -s -v --with-cov --cover-package datalad ../datalad_crawler/dbs/tests/test_files.py ../datalad_crawler/nodes/tests/test_annex.py
datalad_crawler.dbs.tests.test_files.test_AnnexDBs(<class 'datalad_crawler.dbs.files.PhysicalFileStatusesDB'>,) ... ok
datalad_crawler.dbs.tests.test_files.test_AnnexDBs(<class 'datalad_crawler.dbs.files.JsonFileStatusesDB'>,) ... ok
datalad_crawler.nodes.tests.test_annex.test_annex_file('full',) ... > /home/yoh/proj/datalad/datalad-master/datalad/cmd.py(1343)close()
-> lgr.warning(
(Pdb) p self
BatchedAnnex(cmd=<<['git', '-c', ++105 chars++ch']>>, output_proc=<function>, path=<<'/home/yoh/.tm++38 chars++_aa'>>)
(Pdb) p self.path
'/home/yoh/.tmp/datalad_temp_tree__test_AnnexDBf1z92_aa'
(Pdb) bt
  /usr/lib/python3.8/runpy.py(194)_run_module_as_main()
-> return _run_code(code, main_globals, None,
  /usr/lib/python3.8/runpy.py(87)_run_code()
-> exec(code, run_globals)
  /usr/lib/python3/dist-packages/nose/__main__.py(8)<module>()
-> run_exit()
  /usr/lib/python3/dist-packages/nose/core.py(118)__init__()
-> unittest.TestProgram.__init__(
  /usr/lib/python3.8/unittest/main.py(101)__init__()
-> self.runTests()
  /usr/lib/python3/dist-packages/nose/core.py(207)runTests()
-> result = self.testRunner.run(self.test)
  /usr/lib/python3/dist-packages/nose/core.py(62)run()
-> test(result)
  /usr/lib/python3/dist-packages/nose/suite.py(177)__call__()
-> return self.run(*arg, **kw)
  /usr/lib/python3/dist-packages/nose/suite.py(224)run()
-> test(orig)
  /usr/lib/python3/dist-packages/nose/suite.py(177)__call__()
-> return self.run(*arg, **kw)
  /usr/lib/python3/dist-packages/nose/suite.py(224)run()
-> test(orig)
  /usr/lib/python3/dist-packages/nose/suite.py(177)__call__()
-> return self.run(*arg, **kw)
  /usr/lib/python3/dist-packages/nose/suite.py(224)run()
-> test(orig)
  /usr/lib/python3/dist-packages/nose/suite.py(177)__call__()
-> return self.run(*arg, **kw)
  /usr/lib/python3/dist-packages/nose/suite.py(224)run()
-> test(orig)
  /usr/lib/python3/dist-packages/nose/case.py(45)__call__()
-> return self.run(*arg, **kwarg)
  /usr/lib/python3/dist-packages/nose/case.py(133)run()
-> self.runTest(result)
  /usr/lib/python3/dist-packages/nose/case.py(151)runTest()
-> test(result)
  /usr/lib/python3.8/unittest/case.py(736)__call__()
-> return self.run(*args, **kwds)
  /usr/lib/python3.8/unittest/case.py(676)run()
-> self._callTestMethod(testMethod)
  /usr/lib/python3.8/unittest/case.py(633)_callTestMethod()
-> method()
  /usr/lib/python3/dist-packages/nose/case.py(197)runTest()
-> self.test(*self.arg)
  /home/yoh/proj/datalad/datalad-master/datalad/tests/utils.py(554)newfunc()
-> return t(*(arg + (d,)), **kw)
  /home/yoh/proj/datalad/datalad-master/datalad/tests/utils.py(666)newfunc()
-> return tfunc(*(args + (path, url)), **kwargs)
  /home/yoh/proj/datalad/datalad-master/datalad/tests/utils.py(731)newfunc()
-> return t(*(arg + (filename,)), **kw)
  /home/yoh/proj/datalad/datalad-crawler/datalad_crawler/nodes/tests/test_annex.py(160)_test_annex_file()
-> output = list(annex(input_with_stats))
  /home/yoh/proj/datalad/datalad-crawler/datalad_crawler/nodes/annex.py(574)__call__()
-> changed = set().union(*self._get_status(args=[fpath]))
  /home/yoh/proj/datalad/datalad-crawler/datalad_crawler/nodes/annex.py(942)_get_status()
-> if isinstance(self.repo, AnnexRepo) and self.repo.is_direct_mode():
  /home/yoh/proj/datalad/datalad-master/datalad/support/annexrepo.py(1158)is_direct_mode()
-> self._direct_mode = self._is_direct_mode_from_config()
  /home/yoh/proj/datalad/datalad-master/datalad/support/annexrepo.py(1144)_is_direct_mode_from_config()
-> self.config.reload()
  /home/yoh/proj/datalad/datalad-master/datalad/config.py(284)reload()
-> self._stores[store_id] = self._reload(runargs)
  /home/yoh/proj/datalad/datalad-master/datalad/config.py(317)_reload()
-> stdout, stderr = self._run(
  /home/yoh/proj/datalad/datalad-master/datalad/config.py(635)_run()
-> return self._runner.run(self._config_cmd + args, **kwargs)
  /home/yoh/proj/datalad/datalad-master/datalad/cmd.py(1139)run()
-> out, err = super().run(
  /home/yoh/proj/datalad/datalad-master/datalad/cmd.py(905)run()
-> proc = subprocess.Popen(cmd,
  /usr/lib/python3.8/subprocess.py(854)__init__()
-> self._execute_child(args, executable, preexec_fn, close_fds,
  /usr/lib/python3.8/subprocess.py(1650)_execute_child()
-> self._close_pipe_fds(p2cread, p2cwrite,
  /usr/lib/python3.8/subprocess.py(1118)_close_pipe_fds()
-> stack.callback(os.close, errwrite)
  /usr/lib/python3.8/contextlib.py(449)callback()
-> _exit_wrapper = self._create_cb_wrapper(callback, *args, **kwds)
  /home/yoh/proj/datalad/datalad-master/datalad/support/annexrepo.py(382)__del__()
-> self._batched.close()
  /home/yoh/proj/datalad/datalad-master/datalad/support/annexrepo.py(3733)close()
-> p.close()
> /home/yoh/proj/datalad/datalad-master/datalad/cmd.py(1343)close()
-> lgr.warning(

```
</details>

So
- I wonder if somehow may be now `config` entities manage to hold off on those AnnexRepo (with their batched annexes) instances `__del__`?

#### A removed path is not the cause for the stall

I thought that we had already some prior git-annex odd behaviors whenever a directory is removed from under its feet while it is still running, which is what is happening here and may be what causes annex to not die gracefully (but insofar I failed to find relevant report/discussion in annex, and didn't try to reproduce with a minimal example)
   -  I incorrectly assumed that DATALAD_TESTS_TEMP_KEEP=1 is not in effect with `@with_tree` since I do not see those log msgs:
<details>
<summary>no log msgs</summary> 

```shell
$> DATALAD_TESTS_TEMP_KEEP=1 python -m nose -s -v --with-cov --cover-package datalad ../datalad_crawler/dbs/tests/test_files.py ../datalad_crawler/nodes/tests/test_annex.py
[INFO   ] Keeping temp file: /home/yoh/.tmp/datalad_temp_get_most_obscure_supported_namewdl6khdx 
[INFO   ] Keeping temp file: /home/yoh/.tmp/datalad_temp_m5fubgx2 
datalad_crawler.dbs.tests.test_files.test_AnnexDBs(<class 'datalad_crawler.dbs.files.PhysicalFileStatusesDB'>,) ... ok
datalad_crawler.dbs.tests.test_files.test_AnnexDBs(<class 'datalad_crawler.dbs.files.JsonFileStatusesDB'>,) ... ok
datalad_crawler.nodes.tests.test_annex.test_annex_file('full',) ... > /home/yoh/proj/datalad/datalad-master/datalad/cmd.py(1343)close()
-> lgr.warning(
(Pdb) 
...
```
</details>
  - but attaching to pdb and seeing those directories still present still and us failing to 'wait' for annex still while directories present suggests that issue is different!
yarikoptic: damn, giving up for now -- I do not really see an excuse for git annex to stall.  Overall change in behavior from before though -- those batched processes were closed at the end of the `_test_AnnexDB` tests, and now that is delayed into the runtime of the `test_annex_file` test. Overall I am ready to blame either python or git-annex (BTW using `8.20200810+git47-g27329f0bb-1~ndall+1` for all this ATM), but I do not fathom a minimal example to demonstrate that unfortunately.  

@joeyh -- what information could we provide to help troubleshoot batched annex process not exiting when we close our stdout, stderr goes into a file which our process also "closes" although it remains open by git annex process.

Some additional observations:
- with `--debug` added to its invocation -- goes through fine, no stalling
- if I `annex._batched.close()` at the end of those `_test_AnnexDB` tests -- all good.  Output in stderr from --debug etc looks identical (well -- subprocesses exit with 128 due to `fatal: Not a valid object name ...` regardless of what it is, shouldn't stall.
yarikoptic: crap -- I thought that may be in non-stuck case all those "Exit 127" from git-annex children would be gone in the "non-stuck" cases... but my modifcations somehow (or evening) lead just to the effect of `--debug` invocation getting stuck now too... bleh

<details>
<summary>some details with a diff (even if pdb is commented out -- stalls)</summary>

- when we get "stuck" we do get those stderr about invalid
```
yoh       461646 11.8  0.2 190572 74436 pts/12   S+   19:18   0:01  |   |   \_ python -m nose -s -v --with-cov --cover-package datalad ../datalad_crawler/dbs/tests/test_files.py ../datalad_crawler/nodes/tests/test_annex.py
yoh       462157  0.0  0.0   8824  3504 pts/12   S+   19:18   0:00  |   |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git -c annex.merge-annex-branches=false annex info --bytes --json --json-error-messages --batch
yoh       462172  0.2  0.1 1074079100 40420 pts/12 Sl+ 19:18   0:00  |   |       |   \_ /usr/lib/git-annex.linux/exe/git-annex --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git-annex/git-annex info --bytes --json --json-error-messages --batch
yoh       462179  0.0  0.0   8900  4128 pts/12   S+   19:18   0:00  |   |       |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
yoh       462180  0.0  0.0   8824  3532 pts/12   S+   19:18   0:00  |   |       |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(
yoh       462181  0.0  0.0   8896  4152 pts/12   S+   19:18   0:00  |   |       |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
yoh       462182  0.0  0.0   8824  4240 pts/12   S+   19:18   0:00  |   |       |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(
yoh       462632  0.0  0.0   8824  3500 pts/12   S+   19:18   0:00  |   |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git -c annex.merge-annex-branches=false annex info --bytes --json --json-error-messages --batch
yoh       462647  0.2  0.1 1074079100 39540 pts/12 Sl+ 19:18   0:00  |   |       |   \_ /usr/lib/git-annex.linux/exe/git-annex --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git-annex/git-annex info --bytes --json --json-error-messages --batch
yoh       462654  0.0  0.0   8900  4100 pts/12   S+   19:18   0:00  |   |       |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
yoh       462655  0.0  0.0   8824  3540 pts/12   S+   19:18   0:00  |   |       |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(
yoh       462656  0.0  0.0   8824  3532 pts/12   S+   19:18   0:00  |   |       |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
yoh       462657  0.0  0.0   8824  3796 pts/12   S+   19:18   0:00  |   |       |       \_ /usr/lib/git-annex.linux/exe/git --library-path /usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(
yoh       462730  0.0  0.1 187828 59840 pts/12   Sl+  19:18   0:00  |   |       \_ python -m nose -s -v --with-cov --cover-package datalad ../datalad_crawler/dbs/tests/test_files.py ../datalad_crawler/nodes/tests/test_annex.py
```

<details>
<summary>some straces</summary> 

```shell
$> strace -f -p 462157
strace: Process 462157 attached
wait4(462172, 0x7ffddea7f894, 0, NULL)  = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
wait4(462172, 0x7ffddea7f894, 0, NULL)  = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
wait4(462172, 0x7ffddea7f894, 0, NULL)  = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
wait4(462172, 0x7ffddea7f894, 0, NULL)  = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
wait4(462172, 0x7ffddea7f894, 0, NULL)  = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
wait4(462172, 0x7ffddea7f894, 0, NULL)  = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
wait4(462172, 0x7ffddea7f894, 0, NULL)  = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
wait4(462172, 0x7ffddea7f894, 0, NULL)  = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
wait4(462172, ^Cstrace: Process 462157 detached
 <detached ...>

(dev3) 1 17419 ->130.....................................:Wed 26 Aug 2020 07:20:54 PM EDT:.
(git)lena:~datalad/datalad-master[nf-cfgsrcquery]git
$> strace -f -p 462172
strace: Process 462172 attached with 5 threads
[pid 462177] futex(0x7f10cc000bac, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 462175] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 462173] futex(0x41c61cc, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 462174] futex(0x41c61cc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 462172] futex(0x555556c209fc, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 462173] <... futex resumed>)       = -1 EAGAIN (Resource temporarily unavailable)
[pid 462174] <... futex resumed>)       = 0
[pid 462173] futex(0x41c6160, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 462174] futex(0x41c6160, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 462173] <... futex resumed>)       = 0
[pid 462173] futex(0x41c6160, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 462174] epoll_wait(4,  <unfinished ...>
[pid 462173] <... futex resumed>)       = 0
[pid 462174] <... epoll_wait resumed>[], 64, 0) = 0
[pid 462173] read(3,  <unfinished ...>
[pid 462174] epoll_wait(4,  <unfinished ...>
[pid 462173] <... read resumed>"\2402\0\0\0\0\0\0", 8) = 8
[pid 462174] <... epoll_wait resumed>[], 64, 0) = 0
[pid 462173] read(3,  <unfinished ...>
[pid 462174] epoll_wait(4,  <unfinished ...>
[pid 462173] <... read resumed>"\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] write(10, "\377\0\0\0\0\0\0\0", 8 <unfinished ...>
[pid 462175] <... restart_syscall resumed>) = 1
[pid 462173] <... write resumed>)       = 8
[pid 462175] read(10,  <unfinished ...>
[pid 462173] read(3,  <unfinished ...>
[pid 462175] <... read resumed>"\377\0\0\0\0\0\0\0", 8) = 8
[pid 462175] futex(0x7f10cc000bac, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 462177] <... futex resumed>)       = 0
[pid 462177] futex(0x7f10cc000bb0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 462175] futex(0x7f10cc000bb0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 462177] <... futex resumed>)       = 0
[pid 462175] <... futex resumed>)       = 1
[pid 462177] futex(0x7f10cc000bb0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 462175] poll([{fd=8, events=POLLIN}, {fd=10, events=POLLIN}], 2, -1 <unfinished ...>
[pid 462177] <... futex resumed>)       = 0
[pid 462177] rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
[pid 462177] clock_gettime(CLOCK_PROCESS_CPUTIME_ID, {tv_sec=0, tv_nsec=49224083}) = 0
[pid 462173] <... read resumed>"\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] read(3,  <unfinished ...>
[pid 462177] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 462177] futex(0x7f10cc000ba8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 462173] <... read resumed>"\1\0\0\0\0\0\0\0", 8) = 8
[pid 462173] futex(0x41c61c8, FUTEX_WAIT_PRIVATE, 0, NULL^Cstrace: Process 462172 detached
strace: Process 462173 detached
 <detached ...>
strace: Process 462174 detached
strace: Process 462175 detached
strace: Process 462177 detached

(dev3) 1 17420 ->130.....................................:Wed 26 Aug 2020 07:21:17 PM EDT:.
(git)lena:~datalad/datalad-master[nf-cfgsrcquery]git
$> ls -l /proc/462172/fd/3
lrwx------ 1 yoh yoh 64 Aug 26 19:21 /proc/462172/fd/3 -> anon_inode:[timerfd]
```
</details>

-

<details>
<summary>crap -- with this diff for pdb in problematic case it started to stall even with --debug !</summary> 

```shell
$> git diff
diff --git a/datalad/cmd.py b/datalad/cmd.py
index 43e8fede5..77327963e 100644
--- a/datalad/cmd.py
+++ b/datalad/cmd.py
@@ -1195,7 +1195,7 @@ class BatchedCommand(SafeDelCloseMixin):
     def __init__(self, cmd, path=None, output_proc=None):
         if not isinstance(cmd, list):
             cmd = [cmd]
-        self.cmd = cmd
+        self.cmd = cmd  + ['--debug']
         self.path = path
         self.output_proc = output_proc if output_proc else readline_rstripped
         self._process = None
@@ -1317,7 +1317,14 @@ class BatchedCommand(SafeDelCloseMixin):
         ret = None
         if self._stderr_out:
             # close possibly still open fd
-            os.fdopen(self._stderr_out).close()
+            f = os.fdopen(self._stderr_out)
+            with open(self._stderr_out_fname) as f2:
+                stderr = f2.read()
+            lgr.warning(" STDERR: %s", stderr)
+            # our problem!
+            if 'fatal: Not a valid object name d/file2.txt' in stderr:
+                import pdb; pdb.set_trace()
+            f.close()
             lgr.warning(" CLOSED STDERR at %s", self._stderr_out)
             self._stderr_out = None
         if self._process:
```
</details>
</details>

Patch:
diff --git a/datalad/cmd.py b/datalad/cmd.py
index 1a7edfe52..c1002ea86 100644
--- a/datalad/cmd.py
+++ b/datalad/cmd.py
@@ -46,6 +46,7 @@ from .utils import (
     generate_file_chunks,
     get_tempfile_kwargs,
     split_cmdline,
+    try_multiple,
     unlink,
 )
 
@@ -1327,9 +1328,27 @@ class BatchedCommand(SafeDelCloseMixin):
                 "Closing stdin of %s and waiting process to finish", process)
             process.stdin.close()
             process.stdout.close()
-            process.wait()
+            # try waiting for the annex process to finish 3 times for 3 sec
+            # with 1s pause in between
+            try:
+                try_multiple(
+                    # ntrials
+                    3,
+                    # exception to catch
+                    subprocess.TimeoutExpired,
+                    # base waiting period
+                    1.0,
+                    # function to run
+                    process.wait,
+                    timeout=3.0,
+                )
+            except subprocess.TimeoutExpired:
+                lgr.warning(
+                    "Batched process %s did not finish, abandoning it without killing it",
+                    process)
+            if process.returncode is not None:
+                lgr.debug("Process %s has finished", process)
             self._process = None
-            lgr.debug("Process %s has finished", process)
         if self._stderr_out_fname and os.path.exists(self._stderr_out_fname):
             if return_stderr:
                 with open(self._stderr_out_fname, 'r') as f:
diff --git a/datalad/config.py b/datalad/config.py
index e75933391..bf921f16c 100644
--- a/datalad/config.py
+++ b/datalad/config.py
@@ -9,25 +9,18 @@
 """
 """
 
+from fasteners import InterProcessLock
 from functools import lru_cache
 import datalad
 from datalad.consts import (
     DATASET_CONFIG_FILE,
-    DATALAD_DOTDIR,
 )
 from datalad.cmd import GitRunner
 from datalad.dochelpers import exc_str
-from distutils.version import LooseVersion
 
 import re
 import os
-from os.path import (
-    join as opj,
-    exists,
-    getmtime,
-    abspath,
-    isabs,
-)
+from pathlib import Path
 from time import time
 
 import logging
@@ -68,26 +61,18 @@ def _where_reload(obj):
     return obj
 
 
-def _parse_gitconfig_dump(dump, store, fileset, replace, cwd=None):
-    if replace:
-        # if we want to replace existing values in the store
-        # collect into a new dict and `update` the store at the
-        # end. This way we get the desired behavior of multi-value
-        # keys, but only for the current source
-        dct = {}
-        fileset = set()
-    else:
-        # if we don't want to replace value, perform the multi-value
-        # preserving addition on the existing store right away
-        dct = store
+# TODO document and make "public" (used in GitRepo too)
+def _parse_gitconfig_dump(dump, cwd=None):
+    dct = {}
+    fileset = set()
     for line in dump.split('\0'):
         if not line:
             continue
         if line.startswith('file:'):
             # origin line
-            fname = line[5:]
-            if not isabs(fname):
-                fname = opj(cwd, fname) if cwd else abspath(fname)
+            fname = Path(line[5:])
+            if not fname.is_absolute():
+                fname = Path(cwd) / fname if cwd else Path.cwd() / fname
             fileset.add(fname)
             continue
         if line.startswith('command line:'):
@@ -108,19 +93,16 @@ def _parse_gitconfig_dump(dump, store, fileset, replace, cwd=None):
                 dct[k] = present_v + (v,)
             else:
                 dct[k] = (present_v, v)
-    if replace:
-        store.update(dct)
-    return store, fileset
+    return dct, fileset
 
 
-def _parse_env(store):
+def _update_from_env(store):
     dct = {}
     for k in os.environ:
         if not k.startswith('DATALAD_'):
             continue
         dct[k.replace('__', '-').replace('_', '.').lower()] = os.environ[k]
     store.update(dct)
-    return store
 
 
 def anything2bool(val):
@@ -157,17 +139,15 @@ class ConfigManager(object):
     configurations items at once.  Instead, each modification results in a
     dedicated call to `git config`. This author thinks this is OK, as he
     cannot think of a situation where a large number of items need to be
-    written during normal operation. If such need arises, various solutions are
-    possible (via GitPython, or an independent writer).
+    written during normal operation.
 
     Each instance carries a public `overrides` attribute. This dictionary
     contains variables that override any setting read from a file. The overrides
-    are persistent across reloads, and are not modified by any of the
-    manipulation methods, such as `set` or `unset`.
+    are persistent across reloads.
 
     Any DATALAD_* environment variable is also presented as a configuration
     item. Settings read from environment variables are not stored in any of the
-    configuration file, but are read dynamically from the environment at each
+    configuration files, but are read dynamically from the environment at each
     `reload()` call. Their values take precedence over any specification in
     configuration files, and even overrides.
 
@@ -196,15 +176,28 @@ class ConfigManager(object):
         if source not in ('any', 'local', 'dataset', 'dataset-local'):
             raise ValueError(
                 'Unkown ConfigManager(source=) setting: {}'.format(source))
-        # store in a simple dict
-        # no subclassing, because we want to be largely read-only, and implement
-        # config writing separately
-        self._store = {}
-        self._cfgfiles = set()
-        self._cfgmtimes = None
-        self._dataset_path = None
-        self._dataset_cfgfname = None
-        self._repo_cfgfname = None
+        store = dict(
+            # store in a simple dict
+            # no subclassing, because we want to be largely read-only, and implement
+            # config writing separately
+            cfg={},
+            # track the files that jointly make up the config in this store
+            files=set(),
+            # and their modification times to be able to avoid needless unforced reloads
+            mtimes=None,
+        )
+        self._stores = dict(
+            # populated with info from git
+            git=store,
+            # only populated with info from commited dataset config
+            dataset=store.copy(),
+        )
+        # merged representation (the only one that existed pre datalad 0.14)
+        # will be built on initial reload
+        self._merged_store = {}
+
+        self._repo = None if dataset is None else \
+            dataset if hasattr(dataset, 'dot_git') else dataset.repo
         self._config_cmd = ['git', 'config']
         # public dict to store variables that always override any setting
         # read from a file
@@ -223,13 +216,7 @@ class ConfigManager(object):
             # when calling 'git config' to prevent a repository in the current
             # working directory from leaking configuration into the output.
             self._config_cmd = ['git', '--git-dir=', 'config']
-        else:
-            self._dataset_path = dataset.path
-            if source != 'local':
-                self._dataset_cfgfname = opj(self._dataset_path,
-                                             DATASET_CONFIG_FILE)
-            if source != 'dataset':
-                self._repo_cfgfname = opj(self._dataset_path, '.git', 'config')
+
         self._src_mode = source
         # Since configs could contain sensitive information, to prevent
         # any "facilitated" leakage -- just disable logging of outputs for
@@ -240,16 +227,22 @@ class ConfigManager(object):
             # to pick up the right config files
             run_kwargs['cwd'] = dataset.path
         self._runner = GitRunner(**run_kwargs)
-        try:
-            self._gitconfig_has_showorgin = \
-                LooseVersion(get_git_version()) >= '2.8.0'
-        except Exception:
-            # no git something else broken, assume git is present anyway
-            # to not delay this, but assume it is old
-            self._gitconfig_has_showorgin = False
 
         self.reload(force=True)
 
+        if not ConfigManager._checked_git_identity:
+            for cfg, envs in (
+                    ('user.name', ('GIT_AUTHOR_NAME', 'GIT_COMMITTER_NAME')),
+                    ('user.email', ('GIT_AUTHOR_EMAIL', 'GIT_COMMITTER_EMAIL'))):
+                if cfg not in self \
+                        and not any(e in os.environ for e in envs):
+                    lgr.warning(
+                        "It is highly recommended to configure Git before using "
+                        "DataLad. Set both 'user.name' and 'user.email' "
+                        "configuration variables."
+                    )
+            ConfigManager._checked_git_identity = True
+
     def reload(self, force=False):
         """Reload all configuration items from the configured sources
 
@@ -258,82 +251,81 @@ class ConfigManager(object):
         is found for any file no reload is performed. This mechanism will not
         detect newly created global configuration files, use `force` in this case.
         """
-        if not force and self._cfgmtimes:
-            # we aren't forcing and we have read files before
-            # check if any file we read from has changed
-            current_time = time()
-            curmtimes = {c: getmtime(c) for c in self._cfgfiles if exists(c)}
-            if all(curmtimes[c] == self._cfgmtimes.get(c) and
-                   # protect against low-res mtimes (FAT32 has 2s, EXT3 has 1s!)
-                   # if mtime age is less than worst resolution assume modified
-                   (current_time - curmtimes[c]) > 2.0
-                   for c in curmtimes):
-                # all the same, nothing to do, except for
-                # superimpose overrides, could have changed in the meantime
-                self._store.update(self.overrides)
-                # reread env, is quick
-                self._store = _parse_env(self._store)
-                return
-
-        self._store = {}
+        run_args = ['-z', '-l', '--show-origin']
+
+        # update from desired config sources only
         # 2-step strategy:
         #   - load datalad dataset config from dataset
         #   - load git config from all supported by git sources
         # in doing so we always stay compatible with where Git gets its
         # config from, but also allow to override persistent information
         # from dataset locally or globally
-        run_args = ['-z', '-l']
-        if self._gitconfig_has_showorgin:
-            run_args.append('--show-origin')
-
-        if self._dataset_cfgfname:
-            if exists(self._dataset_cfgfname):
-                stdout, stderr = self._run(
-                    run_args + ['--file', self._dataset_cfgfname],
-                    log_stderr=True
-                )
-                # overwrite existing value, do not amend to get multi-line
-                # values
-                self._store, self._cfgfiles = _parse_gitconfig_dump(
-                    stdout, self._store, self._cfgfiles, replace=False,
-                    cwd=self._runner.cwd)
-
-        if self._src_mode == 'dataset':
-            # superimpose overrides, and stop early
-            self._store.update(self.overrides)
-            return
-
-        if self._src_mode == 'dataset-local':
-            run_args.append('--local')
-        stdout, stderr = self._run(run_args, log_stderr=True)
-        self._store, self._cfgfiles = _parse_gitconfig_dump(
-            stdout, self._store, self._cfgfiles, replace=True,
-            cwd=self._runner.cwd)
-
-        # always monitor the dataset cfg location, we know where it is in all cases
-        if self._dataset_cfgfname:
-            self._cfgfiles.add(self._dataset_cfgfname)
-            self._cfgfiles.add(self._repo_cfgfname)
-        self._cfgmtimes = {c: getmtime(c) for c in self._cfgfiles if exists(c)}
 
+        # figure out what needs to be reloaded at all
+        to_run = {}
+        # committed dataset config
+        dataset_cfgfile = self._repo.pathobj / DATASET_CONFIG_FILE \
+            if self._repo else None
+        if (self._src_mode != 'local' and
+                dataset_cfgfile and
+                dataset_cfgfile.exists()) and (
+                force or self._need_reload(self._stores['dataset'])):
+            to_run['dataset'] = run_args + ['--file', str(dataset_cfgfile)]
+
+        if self._src_mode != 'dataset' and (
+                force or self._need_reload(self._stores['git'])):
+            to_run['git'] = run_args + ['--local'] \
+                if self._src_mode == 'dataset-local' \
+                else run_args
+
+        # reload everything that was found todo
+        while to_run:
+            store_id, runargs = to_run.popitem()
+            self._stores[store_id] = self._reload(runargs)
+
+        # always update the merged representation, even if we did not reload
+        # anything from a file. ENV or overrides could change independently
+        # start with the commit dataset config
+        merged = self._stores['dataset']['cfg'].copy()
+        # local config always takes precedence
+        merged.update(self._stores['git']['cfg'])
         # superimpose overrides
-        self._store.update(self.overrides)
+        merged.update(self.overrides)
+        # override with environment variables, unless we only want to read the
+        # dataset's commit config
+        if self._src_mode != 'dataset':
+            _update_from_env(merged)
+        self._merged_store = merged
+
+    def _need_reload(self, store):
+        if not store['mtimes']:
+            return True
+        # we have read files before
+        # check if any file we read from has changed
+        current_time = time()
+        curmtimes = {c: c.stat().st_mtime for c in store['files'] if c.exists()}
+        if all(curmtimes[c] == store['mtimes'].get(c) and
+               # protect against low-res mtimes (FAT32 has 2s, EXT3 has 1s!)
+               # if mtime age is less than worst resolution assume modified
+               (current_time - curmtimes[c]) > 2.0
+               for c in curmtimes):
+            return False
+        return True
 
-        # override with environment variables
-        self._store = _parse_env(self._store)
+    def _reload(self, run_args):
+        # query git-config
+        stdout, stderr = self._run(
+            run_args,
+            log_stderr=True
+        )
+        store = {}
+        store['cfg'], store['files'] = _parse_gitconfig_dump(
+            stdout, cwd=self._runner.cwd)
 
-        if not ConfigManager._checked_git_identity:
-            for cfg, envs in (
-                    ('user.name', ('GIT_AUTHOR_NAME', 'GIT_COMMITTER_NAME')),
-                    ('user.email', ('GIT_AUTHOR_EMAIL', 'GIT_COMMITTER_EMAIL'))):
-                if cfg not in self._store \
-                        and not any(e in os.environ for e in envs):
-                    lgr.warning(
-                        "It is highly recommended to configure Git before using "
-                        "DataLad. Set both 'user.name' and 'user.email' "
-                        "configuration variables."
-                    )
-            ConfigManager._checked_git_identity = True
+        # update mtimes of config files, they have just been discovered
+        # and should still exist
+        store['mtimes'] = {c: c.stat().st_mtime for c in store['files']}
+        return store
 
     @_where_reload
     def obtain(self, var, default=None, dialog_type=None, valtype=None,
@@ -469,44 +461,87 @@ class ConfigManager(object):
             self.add(var, '{}'.format(_value), where=where, reload=reload)
         return value
 
+    def __repr__(self):
+        # give full list of all tracked config files, plus overrides
+        return "ConfigManager({}{})".format(
+            [str(p) for p in self._stores['dataset']['files'].union(
+                self._stores['git']['files'])],
+            ', overrides={!r}'.format(self.overrides) if self.overrides else '',
+        )
+
     def __str__(self):
+        # give path of dataset, if there is any, plus overrides
         return "ConfigManager({}{})".format(
-            self._cfgfiles,
-            '+ overrides' if self.overrides else '',
+            self._repo.path if self._repo else '',
+            'with overrides' if self.overrides else '',
         )
 
     #
     # Compatibility with dict API
     #
     def __len__(self):
-        return len(self._store)
+        return len(self._merged_store)
 
     def __getitem__(self, key):
-        return self._store.__getitem__(key)
+        return self._merged_store.__getitem__(key)
 
     def __contains__(self, key):
-        return self._store.__contains__(key)
+        return self._merged_store.__contains__(key)
 
     def keys(self):
         """Returns list of configuration item names"""
-        return self._store.keys()
+        return self._merged_store.keys()
 
     # XXX should this be *args?
     def get(self, key, default=None):
         """D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None."""
-        return self._store.get(key, default)
+        return self._merged_store.get(key, default)
+
+    def get_from_source(self, source, key, default=None):
+        """Like get(), but a source can be specific.
+
+        If `source` is 'dataset', only the commited configuration is queried,
+        overrides are applied. In the case of 'local', the committed
+        configuration is ignored, but overrides and configuration from
+        environment variables are applied as usual.
+        """
+        if source not in ('dataset', 'local'):
+            raise ValueError("source must be 'dataset' or 'local'")
+        if source == 'dataset':
+            return self.overrides.get(
+                key,
+                self._stores['dataset']['cfg'].get(
+                    key,
+                    default))
+        else:
+            if key not in self._stores['dataset']['cfg']:
+                # the key is not in the committed config, hence we can
+                # just report based on the merged representation
+                return self.get(key, default)
+            else:
+                # expensive case, rebuild a config without the committed
+                # dataset config contributing
+                env = {}
+                _update_from_env(env)
+                return env.get(
+                    key,
+                    self.overrides.get(
+                        key,
+                        self._stores['local']['cfg'].get(
+                            key,
+                            default)))
 
     #
     # Compatibility with ConfigParser API
     #
     def sections(self):
         """Returns a list of the sections available"""
-        return list(set([cfg_section_regex.match(k).group(1) for k in self._store]))
+        return list(set([cfg_section_regex.match(k).group(1) for k in self._merged_store]))
 
     def options(self, section):
         """Returns a list of options available in the specified section."""
         opts = []
-        for k in self._store:
+        for k in self._merged_store:
             sec, opt = cfg_sectionoption_regex.match(k).groups()
             if sec == section:
                 opts.append(opt)
@@ -514,14 +549,14 @@ class ConfigManager(object):
 
     def has_section(self, section):
         """Indicates whether a section is present in the configuration"""
-        for k in self._store:
+        for k in self._merged_store:
             if k.startswith(section):
                 return True
         return False
 
     def has_option(self, section, option):
         """If the given section exists, and contains the given option"""
-        for k in self._store:
+        for k in self._merged_store:
             sec, opt = cfg_sectionoption_regex.match(k).groups()
             if sec == section and opt == option:
                 return True
@@ -555,8 +590,8 @@ class ConfigManager(object):
         Optionally limited to a given section.
         """
         if section is None:
-            return self._store.items()
-        return [(k, v) for k, v in self._store.items()
+            return self._merged_store.items()
+        return [(k, v) for k, v in self._merged_store.items()
                 if cfg_section_regex.match(k).group(1) == section]
 
     #
@@ -595,7 +630,27 @@ class ConfigManager(object):
         """
         if where:
             args = self._get_location_args(where) + args
-        out = self._runner.run(self._config_cmd + args, **kwargs)
+        if '-l' in args:
+            # we are just reading, no need to reload, no need to lock
+            return self._runner.run(self._config_cmd + args, **kwargs)
+
+        # all other calls are modifications
+        if '--file' in args:
+            # all paths we are passing are absolute
+            custom_file = Path(args[args.index('--file') + 1])
+            custom_file.parent.mkdir(exist_ok=True)
+        lockfile = None
+        if self._repo and ('--local' in args or '--file' in args):
+            # modification of config in a dataset
+            lockfile = self._repo.dot_git / 'config.dataladlock'
+        else:
+            # follow pattern in downloaders for lockfile location
+            lockfile = Path(self.obtain('datalad.locations.cache')) \
+                / 'locks' / 'gitconfig.lck'
+
+        with InterProcessLock(lockfile, logger=lgr):
+            out = self._runner.run(self._config_cmd + args, **kwargs)
+
         if reload:
             self.reload()
         return out
@@ -609,18 +664,12 @@ class ConfigManager(object):
                 "unknown configuration label '{}' (not in {})".format(
                     where, cfg_labels))
         if where == 'dataset':
-            if not self._dataset_cfgfname:
+            if not self._repo:
                 raise ValueError(
                     'ConfigManager cannot store configuration to dataset, '
                     'none specified')
-            # create an empty config file if none exists, `git config` will
-            # fail otherwise
-            dscfg_dirname = opj(self._dataset_path, DATALAD_DOTDIR)
-            if not exists(dscfg_dirname):
-                os.makedirs(dscfg_dirname)
-            if not exists(self._dataset_cfgfname):
-                open(self._dataset_cfgfname, 'w').close()
-            args.extend(['--file', self._dataset_cfgfname])
+            dataset_cfgfile = self._repo.pathobj / DATASET_CONFIG_FILE
+            args.extend(['--file', str(dataset_cfgfile)])
         elif where == 'global':
             args.append('--global')
         elif where == 'local':
diff --git a/datalad/core/local/resulthooks.py b/datalad/core/local/resulthooks.py
index 044103e87..166bbdd69 100644
--- a/datalad/core/local/resulthooks.py
+++ b/datalad/core/local/resulthooks.py
@@ -38,7 +38,15 @@ def get_jsonhooks_from_config(cfg):
             continue
         hook_basevar = h[:-11]
         hook_name = hook_basevar[20:]
-        call = cfg.get('{}.call-json'.format(hook_basevar), None)
+        # do not use a normal `get()` here, because it reads the committed dataset
+        # config too. That means a datalad update can silently bring in new
+        # procedure definitions from the outside, and in some sense enable
+        # remote code execution by a 3rd-party
+        call = cfg.get_from_source(
+            'local',
+            '{}.call-json'.format(hook_basevar),
+            None
+        )
         if not call:
             lgr.warning(
                 'Incomplete result hook configuration %s in %s' % (
diff --git a/datalad/interface/run_procedure.py b/datalad/interface/run_procedure.py
index db06c9793..d33db0cff 100644
--- a/datalad/interface/run_procedure.py
+++ b/datalad/interface/run_procedure.py
@@ -454,6 +454,16 @@ class RunProcedure(Interface):
         ):
             yield r
 
+        if ds:
+            # the procedure ran and we have to anticipate that it might have
+            # changed the dataset config, so we need to trigger an unforced
+            # reload.
+            # we have to do this despite "being done here", because
+            # run_procedure() runs in the same process and reuses dataset (config
+            # manager) instances, and the next interaction with a dataset should
+            # be able to count on an up-to-date config
+            ds.config.reload()
+
     @staticmethod
     def custom_result_renderer(res, **kwargs):
         from datalad.ui import ui
diff --git a/datalad/interface/utils.py b/datalad/interface/utils.py
index 696e600b6..6291af79f 100644
--- a/datalad/interface/utils.py
+++ b/datalad/interface/utils.py
@@ -55,7 +55,6 @@ from datalad.interface.base import get_allargs_as_kwargs
 from datalad.interface.common_opts import eval_params
 from datalad.interface.common_opts import eval_defaults
 from .results import known_result_xfms
-from datalad.config import ConfigManager
 from datalad.core.local.resulthooks import (
     get_jsonhooks_from_config,
     match_jsonhook2result,
@@ -367,20 +366,8 @@ def eval_results(func):
         from datalad.distribution.dataset import Dataset
         ds = dataset_arg if isinstance(dataset_arg, Dataset) \
             else Dataset(dataset_arg) if dataset_arg else None
-        # do not reuse a dataset's existing config manager here
-        # they are configured to read the committed dataset configuration
-        # too. That means a datalad update can silently bring in new
-        # procedure definitions from the outside, and in some sense enable
-        # remote code execution by a 3rd-party
-        # To avoid that, create a new config manager that only reads local
-        # config (system and .git/config), plus any overrides given to this
-        # datalad session
-        proc_cfg = ConfigManager(
-            ds, source='local', overrides=dlcfg.overrides
-        ) if ds and ds.is_installed() else dlcfg
-
         # look for hooks
-        hooks = get_jsonhooks_from_config(proc_cfg)
+        hooks = get_jsonhooks_from_config(ds.config if ds else dlcfg)
 
         # this internal helper function actually drives the command
         # generator-style, it may generate an exception if desired,
diff --git a/datalad/support/gitrepo.py b/datalad/support/gitrepo.py
index 72041b9bf..500a4f9aa 100644
--- a/datalad/support/gitrepo.py
+++ b/datalad/support/gitrepo.py
@@ -2713,7 +2713,7 @@ class GitRepo(RepoInterface, metaclass=PathBasedFlyweight):
             '',
             ['git', 'config', '-z', '-l', '--file', '.gitmodules'])
         # abuse our config parser
-        db, _ = _parse_gitconfig_dump(out, {}, None, True, cwd=self.path)
+        db, _ = _parse_gitconfig_dump(out, cwd=self.path)
         mods = {}
         for k, v in db.items():
             if not k.startswith('submodule.'):


Test patch:
diff --git a/datalad/interface/tests/test_ls_webui.py b/datalad/interface/tests/test_ls_webui.py
index 2863da8ec..884d0dcd3 100644
--- a/datalad/interface/tests/test_ls_webui.py
+++ b/datalad/interface/tests/test_ls_webui.py
@@ -68,6 +68,8 @@ def test_ignored(topdir):
     assert_equal(ignored(opj(topdir, "annexdir"), only_hidden=True), False)
 
 
+# https://github.com/datalad/datalad/pull/4808#issuecomment-674381095
+@known_failure_windows
 @with_tree(
     tree={'dir': {'.fgit': {'ab.txt': '123'},
                   'subdir': {'file1.txt': '124', 'file2.txt': '123'},
diff --git a/datalad/tests/test_config.py b/datalad/tests/test_config.py
index 089b736eb..b0af66d91 100644
--- a/datalad/tests/test_config.py
+++ b/datalad/tests/test_config.py
@@ -73,7 +73,7 @@ def test_something(path, new_home):
     # will refuse to work on dataset without a dataset
     assert_raises(ValueError, ConfigManager, source='dataset')
     # now read the example config
-    cfg = ConfigManager(Dataset(opj(path, 'ds')), source='dataset')
+    cfg = ConfigManager(GitRepo(opj(path, 'ds'), create=True), source='dataset')
     assert_equal(len(cfg), 5)
     assert_in('something.user', cfg)
     # multi-value
@@ -226,7 +226,7 @@ def test_something(path, new_home):
     padry = !git paremotes | tr ' ' '\\n' | xargs -r -l1 git push --dry-run
 """}}})
 def test_crazy_cfg(path):
-    cfg = ConfigManager(Dataset(opj(path, 'ds')), source='dataset')
+    cfg = ConfigManager(GitRepo(opj(path, 'ds'), create=True), source='dataset')
     assert_in('crazy.padry', cfg)
     # make sure crazy config is not read when in local mode
     cfg = ConfigManager(Dataset(opj(path, 'ds')), source='local')
@@ -387,10 +387,8 @@ def test_overrides():
     from datalad.utils import Path
     assert_not_in(
         'ups.name', cfg,
-        (cfg._store,
+        (cfg._stores,
          cfg.overrides,
-         cfg._cfgfiles,
-         [Path(f).read_text() for f in cfg._cfgfiles if Path(f).exists()],
     ))
 
 
@@ -456,11 +454,15 @@ def test_no_leaks(path1, path2):
         assert_not_in('i.was.here', ds2.config.keys())
 
         # and that we do not track the wrong files
-        assert_not_in(opj(ds1.path, '.git', 'config'), ds2.config._cfgfiles)
-        assert_not_in(opj(ds1.path, '.datalad', 'config'), ds2.config._cfgfiles)
+        assert_not_in(ds1.pathobj / '.git' / 'config',
+                      ds2.config._stores['git']['files'])
+        assert_not_in(ds1.pathobj / '.datalad' / 'config',
+                      ds2.config._stores['dataset']['files'])
         # these are the right ones
-        assert_in(opj(ds2.path, '.git', 'config'), ds2.config._cfgfiles)
-        assert_in(opj(ds2.path, '.datalad', 'config'), ds2.config._cfgfiles)
+        assert_in(ds2.pathobj / '.git' / 'config',
+                  ds2.config._stores['git']['files'])
+        assert_in(ds2.pathobj / '.datalad' / 'config',
+                  ds2.config._stores['dataset']['files'])
 
 
 @with_tempfile()
@@ -512,7 +514,7 @@ def test_global_config():
     # from within tests, global config should be read from faked $HOME (see
     # setup_package)
     glb_cfg_file = Path(os.environ['HOME']) / '.gitconfig'
-    assert any(glb_cfg_file.samefile(Path(p)) for p in dl_cfg._cfgfiles)
+    assert any(glb_cfg_file.samefile(Path(p)) for p in dl_cfg._stores['git']['files'])
     assert_equal(dl_cfg.get("user.name"), "DataLad Tester")
     assert_equal(dl_cfg.get("user.email"), "test@example.com")
 
diff --git a/datalad/tests/utils.py b/datalad/tests/utils.py
index f2d1c5ef6..c861c83c7 100644
--- a/datalad/tests/utils.py
+++ b/datalad/tests/utils.py
@@ -1917,8 +1917,12 @@ def get_ssh_port(host):
 
 def patch_config(vars):
     """Patch our config with custom settings. Returns mock.patch cm
+
+    Only the merged configuration from all sources (global, local, dataset)
+    will be patched. Source-constrained patches (e.g. only committed dataset
+    configuration) are not supported.
     """
-    return patch.dict(dl_cfg._store, vars)
+    return patch.dict(dl_cfg._merged_store, vars)
 
 
 @contextmanager