Repository: datalad/datalad

Issue:
"Argument list too long: 'git' " while saving datalad dataset
I have a issue very similar to #4778. I wanted to datalad save all the files in a datalad dataset, and `datalad -l debug save .` returns the errors:
```
[DEBUG  ] Future for '/cbica/projects/RBC/RBC_RAWDATA/bidsdatasets/HCP_D' raised [Errno 7] Argument list too long: 'git'.  Re-raising to trigger graceful shutdown etc 
[WARNING] Received an exception [Errno 7] Argument list too long: 'git' [subprocess.py:_execute_child:1704].
| Canceling not-yet running jobs and waiting for completion of running.
| You can force earlier forceful exit by Ctrl-C. 
[DEBUG  ] Shutting down <concurrent.futures.thread.ThreadPoolExecutor object at 0x7fa0658d27c0> with 0 futures. Reason: [Errno 7] Argument list too long: 'git' 
[INFO   ] Canceled 0 out of 0 jobs. 0 left running. 
[DEBUG  ] Finished shutdown with force=False due to exception=OSError(7, 'Argument list too long') 
[DEBUG  ] Shutting down <concurrent.futures.thread.ThreadPoolExecutor object at 0x7fa0658d27c0> with 0 futures. Reason: [Errno 7] Argument list too long: 'git' 
[ERROR  ] [Errno 7] Argument list too long: 'git' [subprocess.py:_execute_child:1704] (OSError) 
[DEBUG  ] Shutting down None with 0 futures. Reason: None 
[DEBUG  ] Finished shutdown with force=True due to exception=None 
```

Checking the maximum argument length returns
```
In [1]: import os

In [2]: os.sysconf('SC_ARG_MAX')
Out[2]: 63963136
```
FYI, Datalad version: 0.14.6 and I am using it on a Linux based HPC system. Thanks so much for your time!


Discussion:

Patch:
diff --git a/datalad/runner/nonasyncrunner.py b/datalad/runner/nonasyncrunner.py
index fbb15ab75..0caf9efa2 100644
--- a/datalad/runner/nonasyncrunner.py
+++ b/datalad/runner/nonasyncrunner.py
@@ -26,6 +26,7 @@ from typing import (
     Union,
 )
 
+from datalad.utils import on_windows
 from .protocol import WitlessProtocol
 
 lgr = logging.getLogger("datalad.runner.nonasyncrunner")
@@ -198,7 +199,18 @@ def run_command(cmd: Union[str, List],
 
     protocol = protocol(**protocol_kwargs)
 
-    process = subprocess.Popen(cmd, **kwargs)
+    try:
+        process = subprocess.Popen(cmd, **kwargs)
+    except OSError as e:
+        if not on_windows and "argument list too long" in str(e).lower():
+            lgr.error(
+                "Caught exception suggesting too large stack size limits. "
+                "Hint: use 'ulimit -s' command to see current limit and "
+                "e.g. 'ulimit -s 8192' to reduce it to avoid this exception. "
+                "See https://github.com/datalad/datalad/issues/6106 for more "
+                "information."
+            )
+        raise
     process_stdin_fileno = process.stdin.fileno() if write_stdin else None
     process_stdout_fileno = process.stdout.fileno() if catch_stdout else None
     process_stderr_fileno = process.stderr.fileno() if catch_stderr else None


Test patch:
diff --git a/datalad/runner/tests/test_witless_runner.py b/datalad/runner/tests/test_witless_runner.py
index c404c4ffd..b8be06ec2 100644
--- a/datalad/runner/tests/test_witless_runner.py
+++ b/datalad/runner/tests/test_witless_runner.py
@@ -9,6 +9,7 @@
 """Test WitlessRunner
 """
 
+import logging
 import os
 import signal
 import sys
@@ -30,9 +31,12 @@ from datalad.tests.utils import (
     integration,
     ok_,
     ok_file_has_content,
+    skip_if_on_windows,
+    swallow_logs,
     with_tempfile,
 )
 from datalad.utils import (
+    CMD_MAX_ARG,
     Path,
     on_windows,
 )
@@ -294,3 +298,14 @@ def test_faulty_poll_detection():
     protocol = Protocol()
     protocol.process = PopenMock()
     assert_raises(CommandError, protocol._prepare_result)
+
+
+@skip_if_on_windows  # no "hint" on windows since no ulimit command there
+def test_too_long():
+    with swallow_logs(new_level=logging.ERROR) as cml:
+        with assert_raises(OSError):  # we still raise an exception if we exceed too much
+            Runner().run(
+                [sys.executable, '-c', 'import sys; print(len(sys.argv))'] + [str(i) for i in range(CMD_MAX_ARG)],
+                protocol=StdOutCapture
+            )
+        cml.assert_logged('.*use.*ulimit.*')
\ No newline at end of file