Repository: BobBuildTool/bob
Pull request #617: Make bob archive work remotely
Author: login: MarcKe
Created: 2025-02-21T13:53:21Z | Merged: 2025-03-20T19:31:22Z

relates to issue https://github.com/BobBuildTool/bob/issues/340



Diff:
diff --git a/doc/manpages/bob-archive.rst b/doc/manpages/bob-archive.rst
index 7119f9bfc..7c70b3338 100644
--- a/doc/manpages/bob-archive.rst
+++ b/doc/manpages/bob-archive.rst
@@ -17,7 +17,7 @@ Generic command format:
 
 ::
 
-    bob archive [-h] subcommand ...
+    bob archive [-h] [-l] [-a] [-b NAMES] subcommand ...
 
 Available sub-commands:
 
@@ -31,9 +31,11 @@ Available sub-commands:
 Description
 -----------
 
-The bob archive command can be used to manage local binary artifact archives.
-The command must be executed in the root of the archive and needs write access
-to create an index cache.
+The bob archive command can be used to manage binary artifact archives.
+The command works on the archives defined in the user configuration :ref:`archive <configuration-config-archive>` .
+The archives require the `managed` flag.
+The archives to work on need to be explicitly specified (`-l`, `-a`, `-b`). These arguments are mutually exclusive.
+Write access to the recipe root folder to create an index cache is required.
 
 Artifacts are managed by the information included in their :ref:`Audit Trail
 <audit-trail>`. See the Audit Trail documentation for a detailed description of
@@ -71,6 +73,14 @@ the included data. Currently the ``bob archive`` command has access to the
 Options
 -------
 
+``-l, --local``
+    Instead of working with the archives defined in the user configuration, the command will operate in the working directory.
+    This can be useful for automated tasks running on a server. Make sure to have write access to the working directory.
+``-a, --all``
+    Execute the command on all suitable archives defined in the user configuration.
+``-b, --backend NAME``
+    `NAME` is the name of the archive defined in the user configuration. Multiple archives may be provided by repeatedly using `-b`.
+    The command is executed on all of those archives.
 ``--dry-run``
     Do not actually delete any artifacts but show what would get removed.
 
diff --git a/doc/manual/configuration.rst b/doc/manual/configuration.rst
index 7318f436a..534d1e45c 100644
--- a/doc/manual/configuration.rst
+++ b/doc/manual/configuration.rst
@@ -2220,9 +2220,10 @@ Type: Dictionary or list of dictionaries
 The ``archive`` key configures the default binary artifact server(s) that
 should be used. It is either directly an archive backend entry or a list of
 archive backends. For each entry at least the ``backend`` key must be
-specified. Optionally there can be a ``flags`` key that receives a list of
-various flags, in particular for what operations the backend might be used. See
-the following list for possible flags. The default is ``[download, upload]``.
+specified. Optionally there can be a ``name`` key that is used in log output
+and a ``flags`` key that receives a list of various flags, in particular for
+what operations the backend might be used. See the following list for possible flags.
+The default is ``[download, upload]``.
 
 ``download``
     Use this archive to download artifacts. Note that you still have to
@@ -2232,7 +2233,9 @@ the following list for possible flags. The default is ``[download, upload]``.
 ``upload``
     Use this archive to upload artifacts. To actually upload to the archive the
     build must be performed with uploads enabled (``--upload``).
-
+``managed``
+    This archive is managed, meaning the files can be iterated and deleted.
+    This is required for the archive command to work.
 ``cache``
     Use this archive to cache downloaded artifacts from other archives. If a
     binary artifact was successfully downloaded from another archive it will
@@ -2316,10 +2319,12 @@ It is also possible to use separate methods for upload and download::
     archive:
         -
             backend: http
+            name: "http-archive"
             url: "http://localhost:8001/archive"
             flags: [download]
         -
             backend: shell
+            name: "ssh-archive"
             upload: "scp -q ${BOB_LOCAL_ARTIFACT} localhost:archive/${BOB_REMOTE_ARTIFACT}"
             download: "scp -q localhost:archive/${BOB_REMOTE_ARTIFACT} ${BOB_LOCAL_ARTIFACT}"
             flags: [upload]
diff --git a/pym/bob/archive.py b/pym/bob/archive.py
index 00e066c11..3d1a25c73 100644
--- a/pym/bob/archive.py
+++ b/pym/bob/archive.py
@@ -19,32 +19,28 @@
 concurrent uploads the artifact must appear atomically for unrelated readers.
 """
 
-from . import BOB_VERSION
+from .audit import Audit
 from .errors import BuildError
 from .tty import stepAction, stepMessage, \
     SKIPPED, EXECUTED, WARNING, INFO, TRACE, ERROR, IMPORTANT
-from .utils import asHexStr, removePath, isWindows, sslNoVerifyContext, \
-    getBashPath, tarfileOpen
-from shlex import quote
+from .utils import asHexStr, removePath, isWindows, getBashPath, tarfileOpen, binStat
+from .webdav import WebDav, HTTPException, HttpDownloadError, HttpUploadError, HttpNotFoundError, HttpAlreadyExistsError
 from tempfile import mkstemp, NamedTemporaryFile, TemporaryFile, gettempdir
-import argparse
 import asyncio
-import base64
 import concurrent.futures
 import concurrent.futures.process
 import gzip
-import hashlib
 import http.client
 import io
 import os
 import os.path
 import shutil
 import signal
-import ssl
+import struct
 import subprocess
 import tarfile
-import textwrap
 import urllib.parse
+import hashlib
 
 ARCHIVE_GENERATION = '-1'
 ARTIFACT_SUFFIX = ".tgz"
@@ -67,10 +63,15 @@ def writeFileOrHandle(name, fileobj, content):
     with open(name, "wb") as f:
         f.write(content)
 
+def namedErrorString(name, err):
+    return "Archive '{}': {}".format(name, err)
 
 class DummyArchive:
     """Archive that does nothing"""
 
+    def canManage(self):
+        return False
+
     def wantDownloadLocal(self, enable):
         pass
 
@@ -111,6 +112,9 @@ async def uploadLocalFingerprint(self, step, key, fingerprint, executor=None):
     async def downloadLocalFingerprint(self, step, key, executor=None):
         return None
 
+    def getArchiveName(self):
+       return "Dummy"
+
 class ArtifactNotFoundError(Exception):
     pass
 
@@ -162,6 +166,26 @@ def _extract(self, fileobj, audit, content):
             os.makedirs(content)
             self.__extractPackage(tar, audit, content)
 
+    def _extractAudit(self, filename=None, fileobj=None):
+        with tarfileOpen(name=filename, mode="r|*", fileobj=fileobj, errorlevel=1) as tar:
+            # validate
+            if tar.pax_headers.get('bob-archive-vsn') != "1":
+                return
+
+            # find audit trail
+            f = tar.next()
+            while f:
+                if f.name == "meta/audit.json.gz": break
+                f = tar.next()
+            else:
+                raise BuildError("Missing audit trail!")
+
+            # read audit trail
+            auditJsonGz = tar.extractfile(f)
+            auditJson = gzip.GzipFile(fileobj=auditJsonGz)
+
+            return Audit.fromByteStream(auditJson, filename)
+
     def _pack(self, name, fileobj, audit, content):
         pax = { 'bob-archive-vsn' : "1" }
         with gzip.open(name or fileobj, 'wb', 6) as gzf:
@@ -176,6 +200,7 @@ class JenkinsArchive(TarHelper):
 
     def __init__(self, spec):
         self.__xferArtifacts = spec.get("xfer", False)
+        self.__name = spec.get("name", "unknown name")
 
     def wantDownloadLocal(self, enable):
         pass
@@ -198,6 +223,9 @@ def canUpload(self):
     def canCache(self):
         return True
 
+    def canManage(self):
+        return False
+
     async def uploadPackage(self, step, buildId, audit, content, executor=None):
         if not audit:
             raise BuildError("Missing audit trail! Cannot proceed without one.")
@@ -206,7 +234,7 @@ async def uploadPackage(self, step, buildId, audit, content, executor=None):
             with open(self.buildIdName(step), "wb") as f:
                 f.write(buildId)
         except OSError as e:
-            raise BuildError("Cannot store artifact: " + str(e))
+            raise BuildError(namedErrorString(self.__name, "Cannot store artifact: " + str(e)))
 
         if self.__xferArtifacts:
             loop = asyncio.get_event_loop()
@@ -221,7 +249,7 @@ async def uploadPackage(self, step, buildId, audit, content, executor=None):
                             audit, content)
                         a.setResult(msg, kind)
                 except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
-                    raise BuildError("Packing of package interrupted.")
+                    raise BuildError(namedErrorString(self.__name, "Packing of package interrupted."))
 
     def _uploadPackage(self, name, buildId, audit, content):
         # Set default signal handler so that KeyboardInterrupt is raised.
@@ -230,7 +258,7 @@ def _uploadPackage(self, name, buildId, audit, content):
         try:
             self._pack(name, None, audit, content)
         except (tarfile.TarError, OSError) as e:
-            raise BuildError("Cannot pack artifact: " + str(e))
+            raise BuildError(namedErrorString(self.__name, "Cannot pack artifact: " + str(e)))
         finally:
             # Restore signals to default so that Ctrl+C kills process. Needed
             # to prevent ugly backtraces when user presses ctrl+c.
@@ -248,7 +276,7 @@ async def downloadPackage(self, step, buildId, audit, content, caches=[],
                 if not ret: a.fail(msg, WARNING)
                 return ret
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
-                raise BuildError("Extraction of package interrupted.")
+                raise BuildError(namedErrorString(self.__name, "Extraction of package interrupted."))
 
     def _downloadPackage(self, tgzName, buildIdName, buildId, audit, content):
         # Set default signal handler so that KeyboardInterrupt is raised.
@@ -265,7 +293,7 @@ def _downloadPackage(self, tgzName, buildIdName, buildId, audit, content):
                 self._extract(f, audit, content)
             return (True, None)
         except (OSError, tarfile.TarError) as e:
-            raise BuildError("Error extracting binary artifact: " + str(e))
+            raise BuildError(namedErrorString(self.__name, "Error extracting binary artifact: " + str(e)))
         finally:
             # Restore signals to default so that Ctrl+C kills process. Needed
             # to prevent ugly backtraces when user presses ctrl+c.
@@ -282,7 +310,7 @@ def cachePackage(self, buildId, workspace):
         except FileExistsError:
             return None
         except OSError as e:
-            raise BuildError("Cannot cache artifact: " + str(e))
+            raise BuildError(namedErrorString(self.__name, "Cannot cache artifact: " + str(e)))
 
     async def uploadLocalLiveBuildId(self, step, liveBuildId, buildId, executor=None):
         pass
@@ -312,6 +340,9 @@ def buildIdName(step):
     def _buildIdNameW(workspace):
         return workspace.replace('/', '_') + BUILDID_SUFFIX
 
+    def getArchiveName(self):
+       return self.__name
+
 class JenkinsCacheHelper:
     def __init__(self, f):
         self.__f = f
@@ -327,12 +358,14 @@ def __exit__(self, exc_type, exc_value, traceback):
 class BaseArchive(TarHelper):
     def __init__(self, spec):
         flags = spec.get("flags", ["upload", "download"])
+        self.__name = spec.get("name", "unknown name")
         self.__useDownload = "download" in flags
         self.__useUpload = "upload" in flags
         self.__ignoreErrors = "nofail" in flags
         self.__useLocal = "nolocal" not in flags
         self.__useJenkins = "nojenkins" not in flags
         self.__useCache = "cache" in flags
+        self.__managed = "managed" in flags
         self.__wantDownloadLocal = False
         self.__wantDownloadJenkins = False
         self.__wantUploadLocal = False
@@ -368,6 +401,12 @@ def canCache(self):
     def _openDownloadFile(self, buildId, suffix):
         raise ArtifactNotFoundError()
 
+    def canManage(self):
+        return self.__managed and self._canManage()
+
+    def _canManage(self):
+        return False
+
     async def downloadPackage(self, step, buildId, audit, content, caches=[],
                               executor=None):
         if not self.canDownload():
@@ -383,18 +422,18 @@ async def downloadPackage(self, step, buildId, audit, content, caches=[],
                 if not ret: a.fail(msg, kind)
                 return ret
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
-                raise BuildError("Download of package interrupted.")
+                raise BuildError(namedErrorString(self.__name, "Download of package interrupted."))
 
     def cachePackage(self, buildId, workspace):
         try:
             return self._openUploadFile(buildId, ARTIFACT_SUFFIX, False)
         except ArtifactExistsError:
             return None
-        except (ArtifactUploadError, OSError) as e:
+        except (ArtifactUploadError, HttpUploadError, OSError) as e:
             if self.__ignoreErrors:
                 return None
             else:
-                raise BuildError("Cannot cache artifact: " + str(e))
+                raise BuildError(namedErrorString(self.__name, "Cannot cache artifact: " + str(e)))
 
     def _downloadPackage(self, buildId, suffix, audit, content, caches, workspace):
         # Set default signal handler so that KeyboardInterrupt is raised.
@@ -406,16 +445,18 @@ def _downloadPackage(self, buildId, suffix, audit, content, caches, workspace):
                 with Tee(name, fileobj, buildId, caches, workspace) as fo:
                     self._extract(fo, audit, content)
             return (True, None, None)
-        except ArtifactNotFoundError:
-            return (False, "not found", WARNING)
-        except ArtifactDownloadError as e:
-            return (False, e.reason, WARNING)
+        except (ArtifactNotFoundError, HttpNotFoundError):
+            return (False, namedErrorString(self.__name, "not found"), WARNING)
+        except (ArtifactDownloadError, HttpDownloadError) as e:
+            return (False, namedErrorString(self.__name, e.reason), WARNING)
+        except ConnectionRefusedError:
+            return (False, namedErrorString(self.__name, "connection failed"), WARNING)
         except BuildError as e:
             raise
         except OSError as e:
-            raise BuildError("Cannot download artifact: " + str(e))
+            raise BuildError(namedErrorString(self.__name, "Cannot download artifact: " + str(e)))
         except tarfile.TarError as e:
-            raise BuildError("Error extracting binary artifact: " + str(e))
+            raise BuildError(namedErrorString(self.__name, "Error extracting binary artifact: " + str(e)))
         finally:
             # Restore signals to default so that Ctrl+C kills process. Needed
             # to prevent ugly backtraces when user presses ctrl+c.
@@ -433,7 +474,7 @@ async def downloadLocalLiveBuildId(self, step, liveBuildId, executor=None):
                 if ret is None: a.fail(msg, kind)
                 return ret
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
-                raise BuildError("Download of build-id interrupted.")
+                raise BuildError(namedErrorString(self.__name, "Download of build-id interrupted."))
 
     def _downloadLocalFile(self, key, suffix):
         # Set default signal handler so that KeyboardInterrupt is raised.
@@ -444,14 +485,16 @@ def _downloadLocalFile(self, key, suffix):
             with self._openDownloadFile(key, suffix) as (name, fileobj):
                 ret = readFileOrHandle(name, fileobj)
             return (ret, None, None)
-        except ArtifactNotFoundError:
-            return (None, "not found", WARNING)
-        except ArtifactDownloadError as e:
-            return (None, e.reason, WARNING)
+        except (ArtifactNotFoundError, HttpNotFoundError):
+            return (None, namedErrorString(self.__name, "not found"), WARNING)
+        except (ArtifactDownloadError, HttpDownloadError) as e:
+            return (None, namedErrorString(self.__name, e.reason), WARNING)
+        except ConnectionRefusedError:
+            return (None, namedErrorString(self.__name, "connection failed"), WARNING)
         except BuildError as e:
             raise
         except OSError as e:
-            raise BuildError("Cannot download file: " + str(e))
+            raise BuildError(namedErrorString(self.__name, "Cannot download file: " + str(e)))
         finally:
             # Restore signals to default so that Ctrl+C kills process. Needed
             # to prevent ugly backtraces when user presses ctrl+c.
@@ -477,7 +520,7 @@ async def uploadPackage(self, step, buildId, audit, content, executor=None):
                     self, buildId, suffix, audit, content)
                 a.setResult(msg, kind)
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
-                raise BuildError("Upload of package interrupted.")
+                raise BuildError(namedErrorString(self.__name, "Upload of package interrupted."))
 
     def _uploadPackage(self, buildId, suffix, audit, content):
         # Set default signal handler so that KeyboardInterrupt is raised.
@@ -487,13 +530,13 @@ def _uploadPackage(self, buildId, suffix, audit, content):
         try:
             with self._openUploadFile(buildId, suffix, False) as (name, fileobj):
                 self._pack(name, fileobj, audit, content)
-        except ArtifactExistsError:
-            return ("skipped ({} exists in archive)".format(content), SKIPPED)
-        except (ArtifactUploadError, tarfile.TarError, OSError) as e:
+        except (ArtifactExistsError, HttpAlreadyExistsError):
+            return (namedErrorString(self.__name, "skipped ({} exists in archive)".format(content)), SKIPPED)
+        except (ArtifactUploadError, HttpUploadError, tarfile.TarError, OSError) as e:
             if self.__ignoreErrors:
-                return ("error ("+str(e)+")", ERROR)
+                return (namedErrorString(self.__name, "error ("+str(e)+")"), ERROR)
             else:
-                raise BuildError("Cannot upload artifact: " + str(e))
+                raise BuildError(namedErrorString(self.__name, "Cannot upload artifact: " + str(e)))
         finally:
             # Restore signals to default so that Ctrl+C kills process. Needed
             # to prevent ugly backtraces when user presses ctrl+c.
@@ -510,7 +553,7 @@ async def uploadLocalLiveBuildId(self, step, liveBuildId, buildId, executor=None
                 msg, kind = await loop.run_in_executor(executor, BaseArchive._uploadLocalFile, self, liveBuildId, BUILDID_SUFFIX, buildId)
                 a.setResult(msg, kind)
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
-                raise BuildError("Upload of build-id interrupted.")
+                raise BuildError(namedErrorString(self.__name, "Upload of build-id interrupted."))
 
     def _uploadLocalFile(self, key, suffix, content):
         # Set default signal handler so that KeyboardInterrupt is raised.
@@ -522,11 +565,11 @@ def _uploadLocalFile(self, key, suffix, content):
             # never see an ArtifactExistsError.
             with self._openUploadFile(key, suffix, True) as (name, fileobj):
                 writeFileOrHandle(name, fileobj, content)
-        except (ArtifactUploadError, OSError) as e:
+        except (ArtifactUploadError, HttpUploadError, OSError) as e:
             if self.__ignoreErrors:
-                return ("error ("+str(e)+")", ERROR)
+                return (namedErrorString(self.__name, "error ("+str(e)+")"), ERROR)
             else:
-                raise BuildError("Cannot upload file: " + str(e))
+                raise BuildError(namedErrorString(self.__name, "Cannot upload file: " + str(e)))
         finally:
             # Restore signals to default so that Ctrl+C kills process. Needed
             # to prevent ugly backtraces when user presses ctrl+c.
@@ -543,7 +586,7 @@ async def uploadLocalFingerprint(self, step, key, fingerprint, executor=None):
                 msg, kind = await loop.run_in_executor(executor, BaseArchive._uploadLocalFile, self, key, FINGERPRINT_SUFFIX, fingerprint)
                 a.setResult(msg, kind)
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
-                raise BuildError("Upload of build-id interrupted.")
+                raise BuildError(namedErrorString(self.__name, "Upload of build-id interrupted."))
 
     async def downloadLocalFingerprint(self, step, key, executor=None):
         if not self.canDownload():
@@ -557,7 +600,70 @@ async def downloadLocalFingerprint(self, step, key, executor=None):
                 if ret is None: a.fail(msg, kind)
                 return ret
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
-                raise BuildError("Download of fingerprint interrupted.")
+                raise BuildError(namedErrorString(self.__name, "Download of fingerprint interrupted."))
+
+    def deleteFile(self, filepath):
+        try:
+            self._delete(filepath)
+        except (ArtifactDownloadError, OSError) as e:
+            if self.__ignoreErrors:
+                return (namedErrorString(self.__name, "error ("+str(e)+")"), ERROR)
+            else:
+                raise BuildError(namedErrorString(self.__name, "Could not delete file: " + str(e)))
+
+    def _delete(self, filepath):
+        raise ArtifactDownloadError("not implemented")
+
+    def listDir(self, path):
+        try:
+            return self._listDir(path)
+        except (ArtifactDownloadError, OSError) as e:
+            if self.__ignoreErrors:
+                return (namedErrorString(self.__name, "error (" + str(e) + ")"), ERROR)
+            else:
+                raise BuildError(namedErrorString(self.__name, "Could not list dir: " + str(e)))
+
+    def _listDir(self, path):
+        raise ArtifactDownloadError("not implemented")
+
+    def stat(self, filepath):
+        try:
+            return self._stat(filepath)
+        except (ArtifactDownloadError, OSError) as e:
+            if self.__ignoreErrors:
+                return (namedErrorString(self.__name, "error (" + str(e) + ")"), ERROR)
+            else:
+                raise BuildError(namedErrorString(self.__name, "Could not stat file: " + str(e)))
+
+    def _stat(self, filepath):
+        raise ArtifactDownloadError("not implemented")
+
+    def getAudit(self, filepath):
+        try:
+            return self._getAudit(filepath)
+        except (ArtifactDownloadError, OSError) as e:
+            if self.__ignoreErrors:
+                return (namedErrorString(self.__name, "error (" + str(e) + ")"), ERROR)
+            else:
+                raise BuildError(namedErrorString(self.__name, "Could not get audit from file: " + str(e)))
+
+    def _getAudit(self, filepath):
+        raise ArtifactDownloadError("not implemented")
+
+    def getArchiveUri(self):
+        try:
+            return self._getArchiveUri()
+        except (ArtifactDownloadError, OSError) as e:
+            if self.__ignoreErrors:
+                return (namedErrorString(self.__name, "error (" + str(e) + ")"), ERROR)
+            else:
+                raise BuildError(namedErrorString(self.__name, "Could not get archive hash: " + str(e)))
+
+    def _getArchiveUri(self):
+        raise ArtifactDownloadError("not implemented")
+
+    def getArchiveName(self):
+       return self.__name
 
 
 class Tee:
@@ -652,6 +758,9 @@ def __init__(self, spec):
         self.__fileMode = spec.get("fileMode")
         self.__dirMode = spec.get("directoryMode")
 
+    def _canManage(self):
+        return True
+
     def _getPath(self, buildId, suffix):
         packageResultId = buildIdToName(buildId)
         packageResultPath = os.path.join(self.__basePath, packageResultId[0:2],
@@ -688,6 +797,27 @@ def _openUploadFile(self, buildId, suffix, overwrite):
             NamedTemporaryFile(dir=packageResultPath, delete=False),
             self.__fileMode, packageResultFile, overwrite)
 
+    def _delete(self, filename):
+        try:
+            os.unlink(os.path.join(self.__basePath, filename))
+        except FileNotFoundError:
+            pass
+        except OSError as e:
+            raise BuildError(namedErrorString(self.__name, "Cannot remove {}: {}".format(filename, str(e))))
+
+    def _listDir(self, path):
+        return os.listdir(os.path.join(self.__basePath, path))
+
+    def _stat(self, filename):
+        return binStat(os.path.join(self.__basePath, filename))
+
+    def _getAudit(self, filename):
+        return self._extractAudit(filename=os.path.join(self.__basePath, filename))
+
+    def _getArchiveUri(self):
+        return self.__basePath
+
+
 class LocalArchiveDownloader:
     def __init__(self, name):
         try:
@@ -737,118 +867,69 @@ def __exit__(self, exc_type, exc_value, traceback):
         return False
 
 
-class SimpleHttpArchive(BaseArchive):
+class HttpArchive(BaseArchive):
     def __init__(self, spec):
         super().__init__(spec)
         self.__url = urllib.parse.urlparse(spec["url"])
-        self.__connection = None
-        self.__sslVerify = spec.get("sslVerify", True)
+        self._webdav = WebDav(self.__url, spec.get("sslVerify", True))
 
     def __retry(self, request):
         retry = True
         while True:
             try:
                 return (True, request())
-            except (http.client.HTTPException, OSError) as e:
-                self._resetConnection()
+            except (HTTPException, OSError) as e:
+                self._webdav._resetConnection()
                 if not retry: return (False, e)
                 retry = False
 
-    def _makeUrl(self, buildId, suffix):
+    def _canManage(self):
+        return True
+
+    def _resetConnection(self):
+        self._webdav._resetConnection()
+
+    def _makePath(self, buildId, suffix):
         packageResultId = buildIdToName(buildId)
         return "/".join([self.__url.path, packageResultId[0:2], packageResultId[2:4],
             packageResultId[4:] + suffix])
 
-    def _remoteName(self, buildId, suffix):
-        url = self.__url
-        return urllib.parse.urlunparse((url.scheme, url.netloc, self._makeUrl(buildId, suffix), '', '', ''))
-
-    def _getConnection(self):
-        if self.__connection is not None:
-            return self.__connection
-
-        url = self.__url
-        if url.scheme == 'http':
-            connection = http.client.HTTPConnection(url.hostname, url.port)
-        elif url.scheme == 'https':
-            ctx = None if self.__sslVerify else sslNoVerifyContext()
-            connection = http.client.HTTPSConnection(url.hostname, url.port,
-                                                     context=ctx)
+    def _makeParentDirs(self, path):
+        (dirs, _, _) = path.rpartition("/")
+        (ok, result) = self.__retry(lambda: self._webdav.mkdir(dirs, dirs.count("/")))
+        if ok:
+            return result
         else:
-            raise BuildError("Unsupported URL scheme: '{}'".format(url.schema))
+            raise result
 
-        self.__connection = connection
-        return connection
-
-    def _resetConnection(self):
-        if self.__connection is not None:
-            self.__connection.close()
-            self.__connection = None
-
-    def _getHeaders(self):
-        headers = { 'User-Agent' : 'BobBuildTool/{}'.format(BOB_VERSION) }
-        if self.__url.username is not None:
-            username = urllib.parse.unquote(self.__url.username)
-            passwd = urllib.parse.unquote(self.__url.password)
-            userPass = username + ":" + passwd
-            headers['Authorization'] = 'Basic ' + base64.b64encode(
-                userPass.encode("utf-8")).decode("ascii")
-        return headers
+    def _remoteName(self, buildId, suffix):
+        url = self.__url
+        return urllib.parse.urlunparse((url.scheme, url.netloc, self._makePath(buildId, suffix), '', '', ''))
 
-    def _openDownloadFile(self, buildId, suffix):
-        (ok, result) = self.__retry(lambda: self.__openDownloadFile(buildId, suffix))
+    def _exists(self, path):
+        (ok, result) = self.__retry(lambda: self._webdav.exists(path))
         if ok:
             return result
         else:
-            raise ArtifactDownloadError(str(result))
-
-    def __openDownloadFile(self, buildId, suffix):
-        connection = self._getConnection()
-        url = self._makeUrl(buildId, suffix)
-        connection.request("GET", url, headers=self._getHeaders())
-        response = connection.getresponse()
-        if response.status == 200:
-            return SimpleHttpDownloader(self, response)
-        else:
-            response.read()
-            if response.status == 404:
-                raise ArtifactNotFoundError()
-            else:
-                raise ArtifactDownloadError("{} {}".format(response.status,
-                                                           response.reason))
+            raise result
 
-    def _openUploadFile(self, buildId, suffix, overwrite):
-        (ok, result) = self.__retry(lambda: self.__openUploadFile(buildId, suffix, overwrite))
+    def _openDownloadFile(self, buildId, suffix):
+        path = self._makePath(buildId, suffix)
+        (ok, result) = self.__retry(lambda: self.__openDownloadFile(path))
         if ok:
             return result
         else:
-            raise ArtifactUploadError(str(result))
+            raise result
 
-    def __openUploadFile(self, buildId, suffix, overwrite):
-        connection = self._getConnection()
-        url = self._makeUrl(buildId, suffix)
+    def __openDownloadFile(self, path):
+        return HttpDownloader(self, self._webdav.download(path))
 
+    def _openUploadFile(self, buildId, suffix, overwrite):
+        path = self._makePath(buildId, suffix)
         # check if already there
         if not overwrite:
-            connection.request("HEAD", url, headers=self._getHeaders())
-            response = connection.getresponse()
-            response.read()
-            if response.status == 200:
+            if self._exists(path):
                 raise ArtifactExistsError()
-            elif response.status != 404:
-                raise ArtifactUploadError("HEAD {} {}".format(response.status, response.reason))
-
-        # create temporary file
-        return SimpleHttpUploader(self, url, overwrite)
-
-    def _putUploadFile(self, url, tmp, overwrite):
-        (ok, result) = self.__retry(lambda: self.__putUploadFile(url, tmp, overwrite))
-        if ok:
-            return result
-        else:
-            raise ArtifactUploadError(str(result))
-
-    def __putUploadFile(self, url, tmp, overwrite):
         # Quoting RFC 4918:
         #
         #   A PUT that would result in the creation of a resource without an
@@ -857,62 +938,52 @@ def __putUploadFile(self, url, tmp, overwrite):
         #
         # We don't want to waste bandwith by uploading big artifacts into a
         # missing directory. Thus make sure the directory always exists.
-        self.__makeParentDirs(url)
-
-        # Determine file length outself and add a "Content-Length" header. This
-        # used to work in Python 3.5 automatically but was removed later.
-        tmp.seek(0, os.SEEK_END)
-        length = str(tmp.tell())
-        tmp.seek(0)
-        headers = self._getHeaders()
-        headers.update({ 'Content-Length' : length })
-        if not overwrite:
-            headers.update({ 'If-None-Match' : '*' })
-        connection = self._getConnection()
-        connection.request("PUT", url, tmp, headers=headers)
-        response = connection.getresponse()
-        response.read()
-        if response.status == 412:
-            # precondition failed -> lost race with other upload
-            raise ArtifactExistsError()
-        elif response.status not in [200, 201, 204]:
-            raise ArtifactUploadError("PUT {} {}".format(response.status, response.reason))
+        self._makeParentDirs(path)
+        return HttpUploader(self, path, overwrite)
+
+    def _putUploadFile(self, path, tmp, overwrite):
+        (ok, result) = self.__retry(lambda: self._webdav.upload(path, tmp, overwrite))
+        if ok:
+            return result
+        else:
+            raise result
+
+    def _listDir(self, path):
+        path_info = self._webdav.listdir(path)
+        entries = []
+        for info in path_info:
+            if info["path"]:
+                entries.append(info["path"].removeprefix(path + "/"))
+        return entries
+
+    def _delete(self, filename):
+        self._webdav.delete(filename)
+
+    def _stat(self, filename):
+        stats = self._webdav.stat(filename)
+        if stats['etag'] is not None and not stats['etag'].startswith('W/'):
+            return stats['etag']
+        if not stats['mdate'] or not stats['len']:
+            raise ArtifactDownloadError("Missing stats for file " + filename)
+        from email.utils import parsedate_to_datetime
+        return struct.pack('=dL', parsedate_to_datetime(stats['mdate']).timestamp(), stats['len'])
+
+    def _getAudit(self, filename):
+        downloader = self._webdav.getPartialDownloader("/".join([self.__url.path, filename]))
+        while True:
+            try:
+                file = io.BytesIO(downloader.get())
+                return self._extractAudit(fileobj=file)
+            except (EOFError, tarfile.ReadError):
+                # partial downloader reached EOF or could not extract the audit from the tarfile, so we get more data
+                downloader.more()
+                pass
 
-    def __makeParentDirs(self, url, depth=0):
-        """Create parent directories.
+    def _getArchiveUri(self):
+        return self.__url.netloc + self.__url.path
 
-        Our artifacts are stored directory levels deep. Only do the MKCOL up
-        to the base level.
-        """
-        (dirs, _, _) = url.rpartition("/")
-        if not dirs:
-            return
 
-        response = self.__mkcol(dirs)
-        if response.status == 409 and depth < 2:
-            # 409 Conflict - Parent collection does not exist.
-            self.__makeParentDirs(dirs, depth + 1)
-            response = self.__mkcol(dirs)
-
-        # We expect to create the directory (201) or it already existed (405).
-        # If the server does not support MKCOL we'd expect a 405 too and hope
-        # for the best...
-        if response.status not in [201, 405]:
-            raise ArtifactUploadError("MKCOL {} {}".format(response.status, response.reason))
-
-    def __mkcol(self, url):
-        # MKCOL resources must have a trailing slash because they are
-        # directories. Otherwise Apache might send a HTTP 301. Nginx refuses to
-        # create the directory with a 409 which looks odd.
-        if not url.endswith("/"):
-            url += "/"
-        connection = self._getConnection()
-        connection.request("MKCOL", url, headers=self._getHeaders())
-        response = connection.getresponse()
-        response.read()
-        return response
-
-class SimpleHttpDownloader:
+class HttpDownloader:
     def __init__(self, archiver, response):
         self.archiver = archiver
         self.response = response
@@ -924,7 +995,8 @@ def __exit__(self, exc_type, exc_value, traceback):
             self.archiver._resetConnection()
         return False
 
-class SimpleHttpUploader:
+
+class HttpUploader:
     def __init__(self, archiver, url, overwrite):
         self.archiver = archiver
         self.tmp = TemporaryFile()
@@ -1218,7 +1290,7 @@ def getSingleArchiver(recipes, archiveSpec):
     if archiveBackend == "file":
         return LocalArchive(archiveSpec)
     elif archiveBackend == "http":
-        return SimpleHttpArchive(archiveSpec)
+        return HttpArchive(archiveSpec)
     elif archiveBackend == "shell":
         return CustomArchive(archiveSpec, recipes.envWhiteList())
     elif archiveBackend == "azure":
diff --git a/pym/bob/cmds/archive.py b/pym/bob/cmds/archive.py
index 29a092bbc..a5dfc6833 100644
--- a/pym/bob/cmds/archive.py
+++ b/pym/bob/cmds/archive.py
@@ -3,12 +3,11 @@
 #
 # SPDX-License-Identifier: GPL-3.0-or-later
 
-from ..audit import Audit
 from ..errors import BobError
 from ..utils import binStat, asHexStr, infixBinaryOp, tarfileOpen
+from ..archive import getSingleArchiver
+from ..input import RecipeSet
 import argparse
-import gzip
-import json
 import os, os.path
 import pickle
 import pyparsing
@@ -21,17 +20,21 @@
 pyparsing.ParserElement.enablePackrat()
 
 class ArchiveScanner:
-    CUR_VERSION = 2
+    CUR_VERSION = 3
 
-    def __init__(self):
+    def __init__(self, archiver):
         self.__dirSchema = re.compile(r'[0-9a-zA-Z]{2}')
         self.__archiveSchema = re.compile(r'[0-9a-zA-Z]{36,}-1.tgz')
         self.__db = None
         self.__cleanup = False
+        self.__archiver = archiver
+        self.__dbName = ".bob-archive.sqlite3"
+        self.__uri = self.__archiver.getArchiveUri()
+        self.__archiveKey = None
 
     def __enter__(self):
         try:
-            self.__con = sqlite3.connect(".bob-archive.sqlite3", isolation_level=None)
+            self.__con = sqlite3.connect(self.__dbName, isolation_level=None)
             self.__db = self.__con.cursor()
             self.__db.execute("""\
                 CREATE TABLE IF NOT EXISTS meta(
@@ -42,21 +45,34 @@ def __enter__(self):
             vsn = self.__db.fetchone()
             if vsn is None:
                 self.__db.executescript("""
+                    CREATE TABLE archives(
+                        key INTEGER PRIMARY KEY NOT NULL,
+                        uri TEXT
+                    );
                     CREATE TABLE files(
-                        bid BLOB PRIMARY KEY NOT NULL,
+                        bid BLOB NOT NULL,
                         stat BLOB,
-                        vars BLOB
+                        vars BLOB,
+                        arch INTEGER NOT NULL,
+                        PRIMARY KEY(bid, arch)
+                        FOREIGN KEY(arch) REFERENCES archives(key)
                     );
                     CREATE TABLE refs(
                         bid BLOB NOT NULL,
                         ref BLOB NOT NULL,
-                        PRIMARY KEY (bid, ref)
+                        arch INTEGER NOT NULL,
+                        PRIMARY KEY (bid, ref, arch),
+                        FOREIGN KEY(arch) REFERENCES archives(key)
                     );
                     """)
                 self.__db.execute("INSERT INTO meta VALUES ('vsn', ?)", (self.CUR_VERSION,))
             elif vsn[0] != self.CUR_VERSION:
                 raise BobError("Archive database was created by an incompatible version of Bob!",
-                    help="Delete '.bob-archive.sqlite3' and run again to re-index.")
+                    help="Delete '{}' and run again to re-index.".format(self.__dbName))
+            # get archive key/id for archiver uri
+            self.__db.execute("INSERT OR IGNORE INTO archives VALUES (NULL, ?)", (self.__uri,))
+            self.__db.execute("SELECT key FROM archives WHERE uri=?", (self.__uri,))
+            self.__archiveKey = self.__db.fetchone()[0]
         except sqlite3.Error as e:
             raise BobError("Cannot open cache: " + str(e))
         return self
@@ -80,12 +96,12 @@ def scan(self, verbose):
         found = False
         try:
             self.__db.execute("BEGIN")
-            for l1 in os.listdir("."):
+            for l1 in self.__archiver.listDir("."):
                 if not self.__dirSchema.fullmatch(l1): continue
-                for l2 in os.listdir(l1):
+                for l2 in self.__archiver.listDir(l1):
                     if not self.__dirSchema.fullmatch(l2): continue
                     l2 = os.path.join(l1, l2)
-                    for l3 in os.listdir(l2):
+                    for l3 in self.__archiver.listDir(l2):
                         m = self.__archiveSchema.fullmatch(l3)
                         if not m: continue
                         found = True
@@ -95,47 +111,32 @@ def scan(self, verbose):
         finally:
             self.__db.execute("END")
             if verbose and not found:
-                print("Your archive seems to be empty. "
-                      "Are you running 'bob archive' from within the correct directory?",
+                print("Your archive seems to be empty. Nothing to be done here.",
                       file=sys.stderr)
         return found
 
     def __scan(self, fileName, verbose):
         try:
-            st = binStat(fileName)
+            st = self.__archiver.stat(fileName)
             bidHex, sep, suffix = fileName.partition("-")
             bid = bytes.fromhex(bidHex[0:2] + bidHex[3:5] + bidHex[6:])
 
             # Validate entry in caching db. Delete entry if stat has changed.
             # The database will clean the 'refs' table automatically.
-            self.__db.execute("SELECT stat FROM files WHERE bid=?",
-                                (bid,))
+            self.__db.execute("SELECT stat FROM files WHERE bid=? AND arch=?",
+                                (bid, self.__archiveKey))
             cachedStat = self.__db.fetchone()
             if cachedStat is not None:
                 if cachedStat[0] == st: return
-                self.__db.execute("DELETE FROM files WHERE bid=?",
-                    (bid,))
+                self.__db.execute("DELETE FROM files WHERE bid=? AND arch=?",
+                    (bid, self.__archiveKey))
 
             # read audit trail
-            if verbose: print("scan", fileName)
-            with tarfileOpen(fileName, errorlevel=1) as tar:
-                # validate
-                if tar.pax_headers.get('bob-archive-vsn') != "1":
-                    print("Not a Bob archive:", fileName, "Ignored!")
-                    return
-
-                # find audit trail
-                f = tar.next()
-                while f:
-                    if f.name == "meta/audit.json.gz": break
-                    f = tar.next()
-                else:
-                    raise Error("Missing audit trail!")
-
-                # read audit trail
-                auditJsonGz = tar.extractfile(f)
-                auditJson = gzip.GzipFile(fileobj=auditJsonGz)
-                audit = Audit.fromByteStream(auditJson, fileName)
+            if verbose: print("\tscan", fileName)
+            audit = self.__archiver.getAudit(fileName)
+            if audit is None:
+                print("\tCould not get audit for ", fileName)
+                return
 
             # import data
             artifact = audit.getArtifact()
@@ -144,10 +145,10 @@ def __scan(self, fileName, verbose):
                 'build' : artifact.getBuildInfo(),
                 'metaEnv' : artifact.getMetaEnv(),
             })
-            self.__db.execute("INSERT INTO files VALUES (?, ?, ?)",
-                (bid, st, vrs))
-            self.__db.executemany("INSERT OR IGNORE INTO refs VALUES (?, ?)",
-                [ (bid, r) for r in audit.getReferencedBuildIds() ])
+            self.__db.execute("INSERT INTO files VALUES (?, ?, ?, ?)",
+                (bid, st, vrs, self.__archiveKey))
+            self.__db.executemany("INSERT OR IGNORE INTO refs VALUES (?, ?, ?)",
+                [ (bid, r, self.__archiveKey) for r in audit.getReferencedBuildIds() ])
         except tarfile.TarError as e:
             raise BobError("Cannot read {}: {}".format(fileName, str(e)))
         except OSError as e:
@@ -155,21 +156,24 @@ def __scan(self, fileName, verbose):
 
     def remove(self, bid):
         self.__cleanup = True
-        self.__db.execute("DELETE FROM files WHERE bid=?",
-            (bid,))
+        self.__db.execute("DELETE FROM files WHERE bid=? AND arch=?",
+            (bid, self.__archiveKey))
+
+    def deleteFile(self, filename):
+        self.__archiver.deleteFile(filename)
 
     def getBuildIds(self):
-        self.__db.execute("SELECT bid FROM files")
+        self.__db.execute("SELECT bid FROM files WHERE arch=?", (self.__archiveKey,))
         return [ r[0] for r in self.__db.fetchall() ]
 
     def getReferencedBuildIds(self, bid):
-        self.__db.execute("SELECT ref FROM refs WHERE bid=?",
-            (bid,))
+        self.__db.execute("SELECT ref FROM refs WHERE bid=? AND arch=?",
+            (bid, self.__archiveKey))
         return [ r[0] for r in self.__db.fetchall() ]
 
     def getVars(self, bid):
-        self.__db.execute("SELECT vars FROM files WHERE bid=?",
-            (bid,))
+        self.__db.execute("SELECT vars FROM files WHERE bid=? AND arch=?",
+            (bid, self.__archiveKey))
         v = self.__db.fetchone()
         if v:
             return pickle.loads(v[0])
@@ -386,22 +390,24 @@ def query(scanner, expressions):
     return retained
 
 
-def doArchiveScan(argv):
+def doArchiveScan(archivers, argv):
     parser = argparse.ArgumentParser(prog="bob archive scan")
     parser.add_argument("-v", "--verbose", action='store_true',
         help="Verbose operation")
     parser.add_argument("-f", "--fail", action='store_true',
         help="Return a non-zero error code in case of errors")
     args = parser.parse_args(argv)
-
-    scanner = ArchiveScanner()
-    with scanner:
-        if not scanner.scan(args.verbose) and args.fail:
-            sys.exit(1)
+    for archiver in archivers:
+        if args.verbose:
+            print("archive '{}':".format(archiver.getArchiveName()))
+        scanner = ArchiveScanner(archiver)
+        with scanner:
+            if not scanner.scan(args.verbose) and args.fail:
+                sys.exit(1)
 
 
 # meta.package == "root" && build.date > "2017-06-19" LIMIT 5 ORDER BY build.date ASC
-def doArchiveClean(argv):
+def doArchiveClean(archivers, argv):
     parser = argparse.ArgumentParser(prog="bob archive clean")
     parser.add_argument('expression', nargs='+',
         help="Expression of artifacts that shall be kept")
@@ -415,44 +421,42 @@ def doArchiveClean(argv):
         help="Return a non-zero error code in case of errors")
     args = parser.parse_args(argv)
 
-    scanner = ArchiveScanner()
-    with scanner:
-        if not args.noscan:
-            if not scanner.scan(args.verbose) and args.fail:
-                sys.exit(1)
-
-        # First pass: determine all directly retained artifacts
-        retained = query(scanner, args.expression)
-
-        # Second pass: determine all transitively retained artifacts
-        todo = set()
-        for bid in retained:
-            todo.update(scanner.getReferencedBuildIds(bid))
-        while todo:
-            n = todo.pop()
-            if n in retained: continue
-            retained.add(n)
-            todo.update(scanner.getReferencedBuildIds(n))
-
-        # Third pass: remove everything that is *not* retained
-        for bid in scanner.getBuildIds():
-            if bid in retained: continue
-            victim = asHexStr(bid)
-            victim = os.path.join(victim[0:2], victim[2:4], victim[4:] + "-1.tgz")
-            if args.dry_run:
-                print(victim)
-            else:
-                try:
+    for archiver in archivers:
+        if args.verbose:
+            print("archive '{}':".format(archiver.getArchiveName()))
+        scanner = ArchiveScanner(archiver)
+        with scanner:
+            if not args.noscan:
+                if not scanner.scan(args.verbose) and args.fail:
+                    sys.exit(1)
+
+            # First pass: determine all directly retained artifacts
+            retained = query(scanner, args.expression)
+
+            # Second pass: determine all transitively retained artifacts
+            todo = set()
+            for bid in retained:
+                todo.update(scanner.getReferencedBuildIds(bid))
+            while todo:
+                n = todo.pop()
+                if n in retained: continue
+                retained.add(n)
+                todo.update(scanner.getReferencedBuildIds(n))
+
+            # Third pass: remove everything that is *not* retained
+            for bid in scanner.getBuildIds():
+                if bid in retained: continue
+                victim = asHexStr(bid)
+                victim = os.path.join(victim[0:2], victim[2:4], victim[4:] + "-1.tgz")
+                if args.dry_run:
+                    print(victim)
+                else:
                     if args.verbose:
-                        print("rm", victim)
-                    os.unlink(victim)
-                except FileNotFoundError:
-                    pass
-                except OSError as e:
-                    raise BobError("Cannot remove {}: {}".format(victim, str(e)))
-                scanner.remove(bid)
-
-def doArchiveFind(argv):
+                        print("\trm", victim)
+                    scanner.deleteFile(victim)
+                    scanner.remove(bid)
+
+def doArchiveFind(archivers, argv):
     parser = argparse.ArgumentParser(prog="bob archive find")
     parser.add_argument('expression', nargs='+',
         help="Expression that artifacts need to match")
@@ -464,18 +468,20 @@ def doArchiveFind(argv):
         help="Return a non-zero error code in case of errors")
     args = parser.parse_args(argv)
 
-    scanner = ArchiveScanner()
-    with scanner:
-        if not args.noscan:
-            if not scanner.scan(args.verbose) and args.fail:
-                sys.exit(1)
+    for archiver in archivers:
+        print("archive '{}':".format(archiver.getArchiveName()))
+        scanner = ArchiveScanner(archiver)
+        with scanner:
+            if not args.noscan:
+                if not scanner.scan(args.verbose) and args.fail:
+                    sys.exit(1)
 
-        # First pass: determine all directly retained artifacts
-        retained = query(scanner, args.expression)
+            # First pass: determine all directly retained artifacts
+            retained = query(scanner, args.expression)
 
-    for bid in sorted(retained):
-        bid = asHexStr(bid)
-        print(os.path.join(bid[0:2], bid[2:4], bid[4:] + "-1.tgz"))
+        for bid in sorted(retained):
+            bid = asHexStr(bid)
+            print("\t" + os.path.join(bid[0:2], bid[2:4], bid[4:] + "-1.tgz"))
 
 availableArchiveCmds = {
     "scan" : (doArchiveScan, "Scan archive for new artifacts"),
@@ -492,14 +498,62 @@ def doArchive(argv, bobRoot):
 
   bob archive {}
 """.format(subHelp))
+    arg_group = parser.add_mutually_exclusive_group(required=False)
+    arg_group.add_argument("-l", "--local", action='store_true',
+                        help="ignore the archive configuration and work on the current directory")
+    arg_group.add_argument("-a", "--all", action='store_true',
+                        help="execute the command on all suitable archives in the user configuration")
+    arg_group.add_argument("-b", "--backend", type=str, metavar="name", action='append',
+                        help="execute the command on the archive backend provided. the name corresponds "
+                             "to the archive name from the user configuration. multiple backends may be"
+                             "provided")
     parser.add_argument('subcommand', help="Subcommand")
     parser.add_argument('args', nargs=argparse.REMAINDER,
                         help="Arguments for subcommand")
 
     args = parser.parse_args(argv)
 
+    # get archiver
+    archivers = []
+    if args.local:
+        # provide local directory as backend
+        archivespec = [{"name" : "local", "backend": "file", "path": "{}".format(os.getcwd()), "flags" : "[download, upload, managed]"}]
+        archiver = getSingleArchiver(None, archivespec[0])
+        archivers.append(archiver)
+    else:
+        # use the configuration to find suitable archives
+        recipes = RecipeSet()
+        recipes.parse()
+        archivespec = recipes.archiveSpec()
+        if isinstance(archivespec, list):
+            if len(archivespec) == 0:
+                raise BobError("No archiver defined")
+            elif len(archivespec) == 1:
+                archiver = getSingleArchiver(recipes, archivespec[0])
+                if archiver.canManage():
+                    archivers.append(archiver)
+                else:
+                    raise BobError("Archiver does not support the archive command")
+            else:
+                backends = []
+                if args.backend is not None:
+                    backends = args.backend
+                for i in archivespec:
+                    archiver = getSingleArchiver(recipes, i)
+                    if (args.all or archiver.getArchiveName() in backends) and archiver.canManage():
+                        archivers.append(archiver)
+                if len(archivers) == 0:
+                    raise BobError("None of the archivers supports the archive command. Maybe you did not select any (-b/--backend)?")
+
+        else:
+            archiver = getSingleArchiver(recipes, archivespec)
+            if archiver.canManage():
+                archivers.append(archiver)
+            else:
+                raise BobError("Archiver does not support the archive command")
+
     if args.subcommand in availableArchiveCmds:
-        availableArchiveCmds[args.subcommand][0](args.args)
+        availableArchiveCmds[args.subcommand][0](archivers, args.args)
     else:
         parser.error("Unknown subcommand '{}'".format(args.subcommand))
 
diff --git a/pym/bob/input.py b/pym/bob/input.py
index 1c35e52a6..0d310b5b3 100644
--- a/pym/bob/input.py
+++ b/pym/bob/input.py
@@ -2983,7 +2983,8 @@ def __init__(self):
             ignore_extra_keys=True)
         baseArchive = {
             'backend' : str,
-            schema.Optional('flags') : schema.Schema(["download", "upload",
+            schema.Optional('name') : str,
+            schema.Optional('flags') : schema.Schema(["download", "upload", "managed",
                 "nofail", "nolocal", "nojenkins", "cache"])
         }
         fileArchive = baseArchive.copy()
diff --git a/pym/bob/webdav.py b/pym/bob/webdav.py
new file mode 100644
index 000000000..9c289bb40
--- /dev/null
+++ b/pym/bob/webdav.py
@@ -0,0 +1,231 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit
+from xml.etree.ElementTree import fromstring
+
+class HTTPException(Exception):
+    pass
+
+class HttpDownloadError(HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpUploadError(HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpNotFoundError(HTTPException):
+    pass
+
+class HttpAlreadyExistsError(HTTPException):
+    pass
+
+class WebDav:
+
+    class PartialDownloader:
+        def __init__(self, webdav, path, length=512*1024):
+            self.__webdev = webdav
+            self.__path = path
+            self.__data = bytearray(self.__webdev.download(self.__path, 0, length).read())
+            self.__offset = length
+
+        def get(self):
+            return self.__data
+
+        def more(self, length=512*1024):
+            new_data = self.__webdev.download(self.__path, self.__offset, length)
+            self.__offset += length
+            self.__data.extend(new_data.read())
+            return self.__data
+
+    def __init__(self, url, sslVerify=True):
+        self.__url = url
+        self.__connection = None
+        self.__sslVerify = sslVerify
+
+    def getPartialDownloader(self, path, length=512*1024):
+        return self.PartialDownloader(self, path, length)
+
+    def _getConnection(self):
+        if self.__connection is not None:
+            return self.__connection
+
+        url = self.__url
+        if url.scheme == 'http':
+            connection = http.client.HTTPConnection(url.hostname, url.port)
+        elif url.scheme == 'https':
+            ctx = None if self.__sslVerify else sslNoVerifyContext()
+            connection = http.client.HTTPSConnection(url.hostname, url.port,
+                                                     context=ctx)
+        else:
+            raise BuildError("Unsupported URL scheme: '{}'".format(url.scheme))
+
+        self.__connection = connection
+        return connection
+
+    def _resetConnection(self):
+        if self.__connection is not None:
+            self.__connection.close()
+            self.__connection = None
+
+    def _getHeaders(self):
+        headers = {'User-Agent': 'BobBuildTool/{}'.format(BOB_VERSION)}
+        if self.__url.username is not None:
+            username = unquote(self.__url.username)
+            passwd = unquote(self.__url.password)
+            userPass = username + ":" + passwd
+            headers['Authorization'] = 'Basic ' + base64.b64encode(
+                userPass.encode("utf-8")).decode("ascii")
+        return headers
+
+    def exists(self, path):
+        connection = self._getConnection()
+        connection.request("HEAD", path, headers=self._getHeaders())
+        response = connection.getresponse()
+        response.read()
+        if response.status == 200:
+            return True
+        elif response.status == 404:
+            return False
+        else:
+            raise HttpUploadError("HEAD {} {}".format(response.status, response.reason))
+
+    def download(self, path, offset=None, length=None):
+        connection = self._getConnection()
+        headers = self._getHeaders()
+        if offset is not None and length is not None:
+            headers.update({'Range': 'bytes={}-{}'.format(offset, offset + length - 1)})
+        connection.request("GET", path, headers=headers)
+        response = connection.getresponse()
+        if response.status in [200, 206]:
+            return response
+        else:
+            response.read()
+            if response.status == 404:
+                raise HttpNotFoundError()
+            else:
+                raise HttpDownloadError("{} {}".format(response.status,
+                                                           response.reason))
+
+    def upload(self, path, buf, overwrite):
+        # Determine file length ourselves and add a "Content-Length" header. This
+        # used to work in Python 3.5 automatically but was removed later.
+        buf.seek(0, os.SEEK_END)
+        length = str(buf.tell())
+        buf.seek(0)
+        headers = self._getHeaders()
+        headers.update({'Content-Length': length})
+        if not overwrite:
+            headers.update({'If-None-Match': '*'})
+        connection = self._getConnection()
+        connection.request("PUT", path, buf, headers=headers)
+        response = connection.getresponse()
+        response.read()
+        if response.status == 412:
+            # precondition failed -> lost race with other upload
+            raise HttpAlreadyExistsError()
+        elif response.status not in [200, 201, 204]:
+            raise HttpUploadError("PUT {} {}".format(response.status, response.reason))
+
+    def _mkdir(self, path):
+        # MKCOL resources must have a trailing slash because they are
+        # directories. Otherwise Apache might send a HTTP 301. Nginx refuses to
+        # create the directory with a 409 which looks odd.
+        if not path.endswith("/"):
+            path += "/"
+        connection = self._getConnection()
+        connection.request("MKCOL", path, headers=self._getHeaders())
+        response = connection.getresponse()
+        response.read()
+        return response
+
+    def mkdir(self, path, depth=1):
+        if depth > 0:
+            response = self._mkdir(path)
+            if response.status == 409:
+                (_path, _, _) = path.rpartition("/")
+                self.mkdir(_path, depth - 1)
+                response = self._mkdir(path)
+            # We expect to create the directory (201) or it already existed (405).
+            # If the server does not support MKCOL we'd expect a 405 too and hope
+            # for the best...
+            if response.status not in [201, 405]:
+                raise HttpUploadError("MKCOL {} {}".format(response.status, response.reason))
+
+    def listdir(self, path):
+        base_path = self.__url.path
+        # create a full path ending with trailing / (should prevent http 301 - moved permanently)
+        path = '/'.join([base_path, path.strip('/'), ''])
+        if self.exists(path):
+            headers = self._getHeaders()
+            # Depth: 1 - applies to the resource and the immediate children (infinity usually prohibited by server)
+            headers.update({'Depth': '1'})
+            connection = self._getConnection()
+            connection.request("PROPFIND", path, headers=headers)
+            response = connection.getresponse()
+            if response.status not in [207]:
+                raise HttpDownloadError("PROPFIND {} {}".format(response.status, response.reason))
+            content = response.read()
+            # get all dav responses from multistatusresponse
+            tree = fromstring(content)
+            dir_infos = []
+            for resp in tree.findall(".//{DAV:}response"):
+                # only need the path in case the full URL is included
+                href = unquote(urlsplit(resp.findtext(".//{DAV:}href")).path)
+                # exclude base path
+                if href.strip('/') == path.strip('/'):
+                    continue
+                dir_info = dict()
+                # collect if it is a dir, the href and self defined path (href without base path)
+                dir_info['is_dir'] = resp.find(".//{DAV:}collection") is not None
+                dir_info['href'] = href
+                dir_info['path'] = href[len(base_path):].strip('/')
+                dir_infos.append(dir_info)
+            return dir_infos
+
+    def delete(self, filename):
+        base_path = self.__url.path
+        # create a full path
+        filepath = '/'.join([base_path, filename.strip('/')])
+        headers = self._getHeaders()
+        connection = self._getConnection()
+        connection.request("DELETE", filepath, headers=headers)
+        response = connection.getresponse()
+        response.read()
+        if response.status not in [200, 204]:
+            raise HttpDownloadError("DELETE {} {}".format(response.status, response.reason))
+
+    def stat(self, file):
+        base_path = self.__url.path
+        # create a full path
+        filepath = '/'.join([base_path, file.strip('/')])
+        if self.exists(filepath):
+            headers = self._getHeaders()
+            # Depth: 0 - applies to the resource itself
+            headers.update({'Depth': '0'})
+            connection = self._getConnection()
+            connection.request("PROPFIND", filepath, headers=headers)
+            response = connection.getresponse()
+            if response.status not in [207]:
+                raise HttpDownloadError("PROPFIND {} {}".format(response.status, response.reason))
+            # get response
+            content = response.read()
+            # parse tree from content
+            tree = fromstring(content)
+            # get response tag tree
+            resp = tree.find(".//{DAV:}response")
+            stats = dict()
+            stats['mdate'] = resp.find(".//{DAV:}getlastmodified")
+            if stats['mdate'] is not None:
+                stats['mdate'] = stats['mdate'].text
+            stats['len'] = resp.find(".//{DAV:}getcontentlength")
+            if stats['len'] is not None:
+                stats['len'] = int(stats['len'].text)
+            stats['etag'] = resp.find(".//{DAV:}getetag")
+            if stats['etag'] is not None:
+                stats['etag'] = stats['etag'].text
+            return stats
diff --git a/test/black-box/archive-mirror/run.sh b/test/black-box/archive-mirror/run.sh
index 648b423e9..e6338b758 100755
--- a/test/black-box/archive-mirror/run.sh
+++ b/test/black-box/archive-mirror/run.sh
@@ -12,6 +12,7 @@ upload="$archiveDir/uploads"
 cat >"${upload}.yaml" <<EOF
 archive:
   -
+    name: "local"
     backend: file
     path: "$(mangle_path "$archiveDir/source")"
 EOF
@@ -21,19 +22,23 @@ mirror="$archiveDir/mirror"
 cat >"${mirror}.yaml" <<EOF
 archive:
   -
+    name: "source"
     flags: [download]
     backend: file
     path: "$(mangle_path "$archiveDir/source")"
   -
+    name: "mirror1"
     flags: [download, cache]
     backend: file
     path: "$(mangle_path "$archiveDir/mirror1")"
   -
+    name: "shell1"
     flags: [download, cache]
     backend: shell
     download: "cp \\"$archiveDir/mirror2/\$BOB_REMOTE_ARTIFACT\\" \\"\$BOB_LOCAL_ARTIFACT\\""
     upload: "mkdir -p \\"$archiveDir/mirror2/\${BOB_REMOTE_ARTIFACT%/*}\\" && cp -n \\"\$BOB_LOCAL_ARTIFACT\\" \\"$archiveDir/mirror2/\$BOB_REMOTE_ARTIFACT\\""
   -
+    name: "shell2"
     flags: [download, cache, nofail]
     backend: shell
     upload: "false"
diff --git a/test/black-box/archive/run.sh b/test/black-box/archive/run.sh
index 68a8613bf..7a9f22a50 100755
--- a/test/black-box/archive/run.sh
+++ b/test/black-box/archive/run.sh
@@ -5,105 +5,237 @@ cleanup
 rm -rf default.yaml
 
 # setup local archive
-trap 'rm -rf "${archiveDir}"' EXIT
+trap 'rm -rf "${archiveDir}" "${archiveDir2}"' EXIT
 archiveDir=$(mktemp -d)
-cat >default.yaml <<EOF
+archiveDir2=$(mktemp -d)
+
+
+function create_config () {
+  cat > default.yaml <<EOF
 archive:
   -
+    name: "local"
     backend: file
     path: "$(mangle_path "$archiveDir")"
+    flags: [download, upload, managed]
 EOF
-
-# fill archive
-run_bob build --download=no --upload root-alpha root-bravo
-FINGERPRINT=Alice run_bob build --force --download=no --upload root-alpha root-bravo
-
-pushd $archiveDir
-run_bob archive scan
-
-# first try to keep everything by using multiple expressions
-oldNum=$(/usr/bin/find -name '*.tgz' | wc -l)
-run_bob archive clean -v 'metaEnv.TYPE == "alpha"' 'meta.package == "root-bravo"'
-newNum=$(/usr/bin/find -name '*.tgz' | wc -l)
-test $oldNum -eq $newNum
-
-# Find one of the artifacts
-found=$(run_bob archive find 'meta.package == "root-bravo"')
-expect_exist "$found"
-
-# selectively keep one half
-run_bob archive clean --dry-run 'metaEnv.TYPE == "alpha"'
-run_bob archive clean -v 'metaEnv.TYPE == "alpha"'
-popd
-
-# alpha can be downloaded, bravo must fail
-rm -rf work
-run_bob build --download=forced root-alpha
-expect_fail run_bob build --download=forced root-bravo
-
-# Update archive and rescan. Add some invalid files too.
-rm -rf work $archiveDir/*
-run_bob build --force --download=no --upload root-alpha root-bravo
-pushd $archiveDir
-mkdir -p 64/ad
-/usr/bin/tar zcf 64/ad/6386bae45ebd6788e404758a247e26e5c778-1.tgz /dev/zero
-touch 64/ad/aabbcc-too-short.tgz
-run_bob archive scan
-popd
-
-# Test that -v doesn't catch fire
-run_bob archive scan -v
-run_bob archive clean -v 'metaEnv.TYPE == "alpha"'
-
-# Test for --fail option
-expect_fail run_bob archive scan --fail -v
-expect_fail run_bob archive clean --fail -v 'metaEnv.TYPE == "alpha"'
-pushd $archiveDir
-run_bob archive scan --fail -v
-run_bob archive clean --fail -v 'metaEnv.TYPE == "alpha"'
-popd
-
-# Test the "LIMIT" feature. Build a number or artifacts and keep only a portion
-# of them. By default the artifacts are sorted by build date and the most
-# recent is kept. Verify that the correct subset was retained.
-rm -rf "$archiveDir/"*
-run_bob build --download no --upload -q 'many-*'
-pushd $archiveDir
-run_bob archive clean --fail -v 'meta.recipe == "many" LIMIT 3'
-popd
-test $(/usr/bin/find $archiveDir -name '*.tgz' | wc -l) -eq 3
-run_bob build --download forced --force many-07 many-06 many-05
-
-# Do the same again with ascending sorting and a different ordering key.  The
-# tricky part is that metaEnv.FUZZ is not set in all packages and such packages
-# must not be counted.
-rm -rf "$archiveDir/"*
-run_bob build --download no --force --upload -q 'many-*'
-pushd $archiveDir
-run_bob archive clean --fail -v 'meta.recipe == "many" LIMIT 2 OrDeR By metaEnv.FUZZ ASC'
-popd
-test $(/usr/bin/find $archiveDir -name '*.tgz' | wc -l) -eq 2
-run_bob build --download forced --force many-01 many-03
-
-# Must fail if LIMIT is zero, invalid or negative
-pushd $archiveDir
-expect_fail run_bob archive clean 'meta.recipe == "many" LIMIT 0'
-expect_fail run_bob archive clean 'meta.recipe == "many" LIMIT -3'
-expect_fail run_bob archive clean 'meta.recipe == "many" LIMIT foobar'
-popd
-
-# Build artifacts with special audit meta keys. Try to find them later.
-rm -rf "$archiveDir/"* work
-run_bob build --upload -M my-key=one root-alpha
-run_bob build --upload -M my-key=two root-bravo
-pushd $archiveDir
-run_bob archive scan --fail
-found1=$(run_bob archive find -n 'meta.recipe == "root" && meta.my-key == "one"')
-expect_exist "$found1"
-found2=$(run_bob archive find -n 'meta.recipe == "root" && meta.my-key == "two"')
-expect_exist "$found2"
-test "$found1" != "$found2"
-popd
-
-# Make sure invalid audit meta keys are rejected
-expect_fail run_bob build -M "!nv@l1d=key" root-alpha
+  if [[ "$1" == "multi" ]]; then
+    cat >> default.yaml << EOF
+  -
+    name: "local2"
+    backend: file
+    path: "$(mangle_path "$archiveDir2")"
+    flags: [download, upload, managed]
+EOF
+  fi
+}
+
+function clean_output () {
+  echo $(echo "$@" | sed '/^archive/d' | sed "s/^[ \t]*//")
+}
+
+function run_single_tests () {
+  # cleanup
+  rm -rf work $archiveDir/*
+  if [[ "$1" == "local" ]]; then
+    LOCAL=true
+    BOB_ARGS="-l"
+  fi
+  # fill archive
+  run_bob build --download=no --upload root-alpha root-bravo
+  FINGERPRINT=Alice run_bob build --force --download=no --upload root-alpha root-bravo
+
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+  run_bob archive ${BOB_ARGS} scan
+
+  # first try to keep everything by using multiple expressions
+  oldNum=$(/usr/bin/find -name '*.tgz' | wc -l)
+  run_bob archive ${BOB_ARGS} clean -v 'metaEnv.TYPE == "alpha"' 'meta.package == "root-bravo"'
+  newNum=$(/usr/bin/find -name '*.tgz' | wc -l)
+  test $oldNum -eq $newNum
+
+  # Find one of the artifacts
+  found=$(clean_output "$(run_bob archive ${BOB_ARGS} find 'meta.package == "root-bravo"')")
+  if [[ ! $LOCAL ]]; then pushd $archiveDir; fi
+  expect_exist "$found"
+  if [[ ! $LOCAL ]]; then popd; fi
+
+  # selectively keep one half
+  run_bob archive ${BOB_ARGS} clean --dry-run 'metaEnv.TYPE == "alpha"'
+  run_bob archive ${BOB_ARGS} clean -v 'metaEnv.TYPE == "alpha"'
+  if [[ $LOCAL ]]; then popd; fi
+
+  # alpha can be downloaded, bravo must fail
+  rm -rf work
+  run_bob build --download=forced root-alpha
+  expect_fail run_bob build --download=forced root-bravo
+
+  # Update archive and rescan. Add some invalid files too.
+  rm -rf work $archiveDir/*
+  run_bob build --force --download=no --upload root-alpha root-bravo
+  pushd $archiveDir
+  mkdir -p 64/ad
+  /usr/bin/tar zcf 64/ad/6386bae45ebd6788e404758a247e26e5c778-1.tgz /dev/zero
+  touch 64/ad/aabbcc-too-short.tgz
+  if [[ $LOCAL ]]; then run_bob archive ${BOB_ARGS} scan; fi
+  popd
+  if [[ ! $LOCAL ]]; then run_bob archive ${BOB_ARGS} scan; fi
+
+  # Test that -v doesn't catch fire
+  run_bob archive ${BOB_ARGS} scan -v
+  run_bob archive ${BOB_ARGS} clean -v 'metaEnv.TYPE == "alpha"'
+
+  # Test for --fail option (--fail only reports -1 if there are no files matching the pattern xx/xx/hash-1.tgz)
+  if [[ ! $LOCAL ]]; then
+    run_bob archive ${BOB_ARGS} scan --fail -v
+    run_bob archive ${BOB_ARGS} clean --fail -v 'metaEnv.TYPE == "alpha"'
+    rm -rf work $archiveDir/*
+    expect_fail run_bob archive ${BOB_ARGS} scan --fail -v
+    expect_fail run_bob archive ${BOB_ARGS} clean --fail -v 'metaEnv.TYPE == "alpha"'
+  else
+    expect_fail run_bob archive ${BOB_ARGS} scan --fail -v
+    expect_fail run_bob archive ${BOB_ARGS} clean --fail -v 'metaEnv.TYPE == "alpha"'
+    pushd $archiveDir
+    run_bob archive ${BOB_ARGS} scan --fail -v
+    run_bob archive ${BOB_ARGS} clean --fail -v 'metaEnv.TYPE == "alpha"'
+    popd
+  fi
+
+  # Test the "LIMIT" feature. Build a number or artifacts and keep only a portion
+  # of them. By default the artifacts are sorted by build date and the most
+  # recent is kept. Verify that the correct subset was retained.
+  rm -rf "$archiveDir/"*
+  run_bob build --download no --upload -q 'many-*'
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+  run_bob archive ${BOB_ARGS} clean --fail -v 'meta.recipe == "many" LIMIT 3'
+  if [[ $LOCAL ]]; then popd; fi
+  test $(/usr/bin/find $archiveDir -name '*.tgz' | wc -l) -eq 3
+  run_bob build --download forced --force many-07 many-06 many-05
+
+  # Do the same again with ascending sorting and a different ordering key.  The
+  # tricky part is that metaEnv.FUZZ is not set in all packages and such packages
+  # must not be counted.
+  rm -rf "$archiveDir/"*
+  run_bob build --download no --force --upload -q 'many-*'
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+  run_bob archive ${BOB_ARGS} clean --fail -v 'meta.recipe == "many" LIMIT 2 OrDeR By metaEnv.FUZZ ASC'
+  if [[ $LOCAL ]]; then popd; fi
+  test $(/usr/bin/find $archiveDir -name '*.tgz' | wc -l) -eq 2
+  run_bob build --download forced --force many-01 many-03
+
+  # Must fail if LIMIT is zero, invalid or negative
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+  expect_fail run_bob archive ${BOB_ARGS} clean 'meta.recipe == "many" LIMIT 0'
+  expect_fail run_bob archive ${BOB_ARGS} clean 'meta.recipe == "many" LIMIT -3'
+  expect_fail run_bob archive ${BOB_ARGS} clean 'meta.recipe == "many" LIMIT foobar'
+  if [[ $LOCAL ]]; then popd; fi
+
+  # Build artifacts with special audit meta keys. Try to find them later.
+  rm -rf "$archiveDir/"* work
+  run_bob build --upload -M my-key=one root-alpha
+  run_bob build --upload -M my-key=two root-bravo
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+    run_bob archive ${BOB_ARGS} scan --fail
+    found1=$(clean_output "$(run_bob archive ${BOB_ARGS} find -n 'meta.recipe == "root" && meta.my-key == "one"')")
+    found2=$(clean_output "$(run_bob archive ${BOB_ARGS} find -n 'meta.recipe == "root" && meta.my-key == "two"')")
+  if [[ ! $LOCAL ]]; then pushd $archiveDir; fi
+  expect_exist "$found1"
+  expect_exist "$found2"
+  popd
+  test "$found1" != "$found2"
+
+  # Make sure invalid audit meta keys are rejected
+  expect_fail run_bob build -M "!nv@l1d=key" root-alpha
+}
+
+function run_multi_tests () {
+  # cleanup
+  rm -rf work $archiveDir/*  $archiveDir2/*
+
+  # fill archive
+  run_bob build --download=no --upload root-alpha root-bravo
+  FINGERPRINT=Alice run_bob build --force --download=no --upload root-alpha root-bravo
+
+  # scan should fail as we did not specify one of the archives
+  expect_fail run_bob archive scan
+  # provide -a to scan all archives
+  run_bob archive -a scan
+  # clean up archive databses to allow proper rescanning
+  rm .bob-archive*
+  # scan single archive "local"
+  run_bob archive -b "local" scan
+  rm .bob-archive*
+  # scan single archive "local2"
+  run_bob archive -b "local2" scan
+  rm .bob-archive*
+
+  # test find
+  run_bob archive -a scan
+  # find also fails without specifying an archive
+  expect_fail run_bob archive find 'meta.package == "root-bravo"'
+  # check if file is found in archive 1 and archive 2
+  found1=$(clean_output "$(run_bob archive -b "local" find 'meta.package == "root-bravo"')")
+  found2=$(clean_output "$(run_bob archive -b "local2" find 'meta.package == "root-bravo"')")
+  pushd $archiveDir
+  expect_exist "$found1"
+  pushd $archiveDir2
+  expect_exist "$found2"
+  popd
+  popd
+  # should find the same artifact twice
+  found=$(clean_output "$(run_bob archive -a find 'meta.package == "root-bravo"')")
+  foundarr=($found)
+  test "${foundarr[0]}" == "${foundarr[1]}"
+  pushd $archiveDir
+  expect_exist "${foundarr[0]}"
+  pushd $archiveDir2
+  expect_exist "${foundarr[0]}"
+  popd
+  popd
+
+  # test clean
+  # artifact should only be removed in archive 1
+  run_bob archive -b "local" clean 'meta.package != "root-bravo"'
+  pushd $archiveDir
+  expect_not_exist "${foundarr[0]}"
+  popd
+  pushd $archiveDir2
+  expect_exist "${foundarr[0]}"
+  popd
+  # artifact should now be remove in archive 2, too
+  run_bob archive -b "local2" clean 'meta.package != "root-bravo"'
+  pushd $archiveDir
+  expect_not_exist "${foundarr[0]}"
+  popd
+  pushd $archiveDir2
+  expect_not_exist "${foundarr[0]}"
+  popd
+  # got to "reupload" them
+  rm -rf work $archiveDir/*  $archiveDir2/*
+  run_bob build --download=no --upload root-alpha root-bravo
+  FINGERPRINT=Alice run_bob build --force --download=no --upload root-alpha root-bravo
+  run_bob archive -a scan
+  pushd $archiveDir
+  expect_exist "${foundarr[0]}"
+  popd
+  pushd $archiveDir2
+  expect_exist "${foundarr[0]}"
+  popd
+  # clean from all archives
+  run_bob archive -a clean 'meta.package != "root-bravo"'
+  pushd $archiveDir
+  expect_not_exist "${foundarr[0]}"
+  popd
+  pushd $archiveDir2
+  expect_not_exist "${foundarr[0]}"
+  popd
+}
+
+create_config
+# run tests with configuration with file backend
+run_single_tests
+# run tests with local directory
+run_single_tests "local"
+# tests with two archive backends
+create_config "multi"
+run_multi_tests
diff --git a/test/black-box/audit/run.sh b/test/black-box/audit/run.sh
index d2a545d95..9f75a6b35 100755
--- a/test/black-box/audit/run.sh
+++ b/test/black-box/audit/run.sh
@@ -14,6 +14,7 @@ trap 'rm -rf "${archiveDir}"' EXIT
 cat >default.yaml <<EOF
 archive:
   -
+    name: "local"
     backend: file
     path: "$(mangle_path "$archiveDir")"
 EOF
diff --git a/test/black-box/buildid-change/run.sh b/test/black-box/buildid-change/run.sh
index afa220711..a1c316db7 100755
--- a/test/black-box/buildid-change/run.sh
+++ b/test/black-box/buildid-change/run.sh
@@ -28,6 +28,7 @@ rm -rf "$D"
 # create binary archive config
 cat >repo.yaml <<EOF
 archive:
+    name: "local"
     backend: file
     path: "$(mangle_path "$REPO/archive")"
 EOF
diff --git a/test/black-box/fingerprintVars/default.yaml b/test/black-box/fingerprintVars/default.yaml
index 80f2939b1..e3804b743 100644
--- a/test/black-box/fingerprintVars/default.yaml
+++ b/test/black-box/fingerprintVars/default.yaml
@@ -1,4 +1,5 @@
 archive:
     -
+        name: "local"
         backend: file
         path: "output"
diff --git a/test/black-box/fingerprints/default.yaml b/test/black-box/fingerprints/default.yaml
index 694a4717f..1ecf53ff6 100644
--- a/test/black-box/fingerprints/default.yaml
+++ b/test/black-box/fingerprints/default.yaml
@@ -1,5 +1,6 @@
 archive:
     -
+        name: "local"
         backend: file
         path: "output"
 
diff --git a/test/black-box/layer-download/run.sh b/test/black-box/layer-download/run.sh
index 290e07b6c..c2479973c 100755
--- a/test/black-box/layer-download/run.sh
+++ b/test/black-box/layer-download/run.sh
@@ -10,6 +10,7 @@ archiveDir=$(mktemp -d)
 cat >default.yaml <<EOF
 archive:
   -
+    name: "local"
     backend: file
     path: "$(mangle_path "$archiveDir")"
 EOF
diff --git a/test/black-box/non-relocatable/run.sh b/test/black-box/non-relocatable/run.sh
index 827e290bd..a76976aaa 100755
--- a/test/black-box/non-relocatable/run.sh
+++ b/test/black-box/non-relocatable/run.sh
@@ -22,6 +22,7 @@ writeCfg()
 cat >"${cfg}.yaml" <<EOF
 archive:
    -
+    name: "local"
     backend: file
     path: "$(mangle_path "$uploadDir")"
 EOF
diff --git a/test/black-box/parallel/default.yaml b/test/black-box/parallel/default.yaml
index b45ab914e..543240d71 100644
--- a/test/black-box/parallel/default.yaml
+++ b/test/black-box/parallel/default.yaml
@@ -1,3 +1,4 @@
 archive:
+    name : "local"
     backend: file
     path: /does/not/exist
diff --git a/test/black-box/sandbox-fingerprints/run.sh b/test/black-box/sandbox-fingerprints/run.sh
index 529f82cc9..0752204a7 100755
--- a/test/black-box/sandbox-fingerprints/run.sh
+++ b/test/black-box/sandbox-fingerprints/run.sh
@@ -16,6 +16,7 @@ ARCHIVE="$(mktemp -d)"
 trap 'rm -rf $ARCHIVE' EXIT
 cat >default.yaml <<EOF
 archive:
+    name: "local"
     backend: file
     path: "$ARCHIVE"
 whitelist:
diff --git a/test/black-box/shared-pkg/run.sh b/test/black-box/shared-pkg/run.sh
index 912fc4d9c..1c8d091ac 100755
--- a/test/black-box/shared-pkg/run.sh
+++ b/test/black-box/shared-pkg/run.sh
@@ -14,6 +14,7 @@ cat >"${cfg}.yaml" <<EOF
 share:
     path: "$(mangle_path "$shareDir")"
 archive:
+    name: "local"
     backend: file
     path: "$(mangle_path "$archiveDir")"
 EOF
diff --git a/test/unit/mocks/http_server.py b/test/unit/mocks/http_server.py
index a2f3d2229..95b62b6c4 100644
--- a/test/unit/mocks/http_server.py
+++ b/test/unit/mocks/http_server.py
@@ -1,17 +1,50 @@
 from http import HTTPStatus
 from http.server import BaseHTTPRequestHandler, HTTPServer
 from threading import Thread, Lock
-import os,sys
+from xml.etree.ElementTree import Element, SubElement, tostring
+import os, time
 
 def createHttpHandler(repoPath, args):
 
     class Handler(BaseHTTPRequestHandler):
-
-        def getCommon(self):
+        def addProps(self, path, response):
+            # trying to create basic props
+            href = SubElement(response, 'D:href')
+            href_path = os.path.relpath(path, repoPath)
+            if href_path == '.':
+                href.text = '/'
+            else:
+                href.text = '/' + str(href_path).replace('\\', '/')
+            propstat = SubElement(response, 'D:propstat')
+            prop = SubElement(propstat, 'D:prop')
+            stats = os.stat(path)
+            length = SubElement(prop, 'D:getcontentlength')
+            length.text = str(stats[6])
+            last_mod = SubElement(prop, 'D:getlastmodified')
+            last_mod.text = time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime(stats[8]))
+            etag = SubElement(prop, 'D:getetag')
+            # etag inspired by apache
+            etag.text = f'{stats[6]:x}-{stats[1]:x}-{stats[8]:x}'
+            resource_type = SubElement(prop, 'D:resourcetype')
+            if os.path.isdir(path):
+               collection = SubElement(resource_type, 'D:collection')
+            status = SubElement(propstat, 'D:status')
+            status.text = 'HTTP/1.1 200 OK'
+
+        def getCommon(self, start=None, end=None):
+            f = None
             path = repoPath + self.path
             try:
-                f = open(path, "rb")
-                fs = os.fstat(f.fileno())
+                if not os.path.exists(path):
+                    raise FileNotFoundError
+                if os.path.isfile(path):
+                    f = open(path, "rb")
+                    fs = os.fstat(f.fileno())
+                    mtype = "application/octet-stream"
+                    length = str(fs[6])
+                else:
+                    mtype = None
+                    length = 0
             except FileNotFoundError:
                 self.send_error(404, "not found")
                 return None
@@ -26,13 +59,22 @@ def getCommon(self):
                 return None
 
             self.send_response(200)
-            self.send_header("Content-type", "application/octet-stream")
-            self.send_header("Content-Length", str(fs[6]))
+            self.send_header("Content-type", mtype)
+            # length is not the full size when a Range is requested
+            if end is not None:
+                if end >= int(length):
+                    end = int(length) - 1
+                length = str(end - start + 1)
+            self.send_header("Content-Length", length)
             self.end_headers()
             return f
 
         def do_HEAD(self):
             self.stats.headRequests += 1
+            if args.get('retryHead') and args.get('retries') > 0:
+                self.send_error(500, "internal error (retries={})".format(args['retries']))
+                args['retries'] = args.get('retries') - 1
+                return
             f = self.getCommon()
             if f: f.close()
 
@@ -46,10 +88,19 @@ def do_GET(self):
                 self.send_error(500, "internal error (retries={})".format(args['retries']))
                 args['retries'] = args.get('retries') - 1
                 return
-
-            f = self.getCommon()
+            # check if a Range was defined
+            start = None
+            end = None
+            if self.headers['Range']:
+                start, end = (int(x) for x in self.headers.get('Range').strip().strip('bytes=').split('-'))
+            f = self.getCommon(start, end)
             if f:
-                self.wfile.write(f.read())
+                if end is not None:
+                    f.seek(start)
+                    data = f.read(end - start + 1)
+                    self.wfile.write(data)
+                else:
+                    self.wfile.write(f.read())
                 f.close()
 
         def do_PUT(self):
@@ -62,7 +113,6 @@ def do_PUT(self):
                 self.send_error(500, "internal error")
                 args['retries'] = args.get('retries') - 1
                 return
-
             path = repoPath + self.path
             if os.path.exists(path) and ("If-None-Match" in self.headers):
                 self.send_response(412)
@@ -76,6 +126,81 @@ def do_PUT(self):
             self.send_response(200)
             self.end_headers()
 
+        def do_DELETE(self):
+            self.stats.deleteRequests += 1
+            if args.get('noResponse'):
+                self.log_error("PUT: noResponse: drop connection!")
+                self.close_connection = True
+                return
+            if args.get('retries') > 0:
+                self.send_error(500, "internal error")
+                args['retries'] = args.get('retries') - 1
+                return
+            path = repoPath + self.path
+            if os.path.exists(path):
+                os.unlink(path)
+                self.send_response(200)
+                self.end_headers()
+            else:
+                self.send_response(404)
+                self.end_headers()
+
+        def do_MKCOL(self):
+            self.stats.mkcolRequests += 1
+            if args.get('noResponse'):
+                self.log_error("PUT: noResponse: drop connection!")
+                self.close_connection = True
+                return
+            if args.get('retries') > 0:
+                self.send_error(500, "internal error")
+                args['retries'] = args.get('retries') - 1
+                return
+            path = repoPath + self.path
+            # remove trailing slash
+            if path[-1] == '/': path = path[:-1]
+            # check if already exists, return 405 if it does
+            if os.path.exists(path):
+                self.send_response(405)
+                self.end_headers()
+                return
+            # check if parent exists, return 409 if not
+            h, t = os.path.split(path)
+            if not os.path.exists(h):
+                self.send_response(409)
+                self.end_headers()
+                return
+            os.mkdir(path)
+            self.send_response(201)
+            self.end_headers()
+
+        def do_PROPFIND(self):
+            self.stats.propfindRequests += 1
+            if args.get('noResponse'):
+                self.log_error("PUT: noResponse: drop connection!")
+                self.close_connection = True
+                return
+            if args.get('retries') > 0:
+                self.send_error(500, "internal error")
+                args['retries'] = args.get('retries') - 1
+                return
+            path = repoPath + self.path
+            depth = str(self.headers.get('Depth', 'infinity'))
+            et = Element('D:multistatus', {'xmlns:D' : 'DAV:'})
+            if depth == '0' or depth == '1':
+                response = SubElement(et, 'D:response')
+                self.addProps(path, response)
+                if depth == '1':
+                    for x in os.listdir(path):
+                        response = SubElement(et, 'D:response')
+                        self.addProps(os.path.join(path, x), response)
+                self.send_response(207)
+                self.end_headers()
+                self.wfile.write(tostring(et))
+            else:
+                # infinity or something else not allowed
+                self.send_response(405)
+                self.end_headers()
+
     return Handler
 
 class HttpServerStats:
@@ -84,11 +209,16 @@ def __init__(self, port):
         self.headRequests = 0
         self.getRequests = 0
         self.putRequests = 0
+        self.deleteRequests = 0
+        self.mkcolRequests = 0
+        self.propfindRequests = 0
 
 class HttpServerMock():
-    def __init__(self, repoPath, noResponse=False, retries=0):
+    def __init__(self, repoPath, noResponse=False, retries=0, retryHead=False):
+        if not repoPath.endswith(os.sep):
+            repoPath += os.sep
         self.handler = createHttpHandler(repoPath,
-            {'noResponse' : noResponse, 'retries' : retries })
+            {'noResponse' : noResponse, 'retries' : retries, 'retryHead' : retryHead})
         self.server = HTTPServer(('localhost', 0), self.handler)
 
     def __enter__(self):
diff --git a/test/unit/mocks/jenkins_tests.py b/test/unit/mocks/jenkins_tests.py
index 6efdcbfca..84e4116b1 100644
--- a/test/unit/mocks/jenkins_tests.py
+++ b/test/unit/mocks/jenkins_tests.py
@@ -54,7 +54,8 @@ def setUp(self):
         self.cwd = tempfile.mkdtemp()
         os.chdir(self.cwd)
         self.archive = tempfile.TemporaryDirectory()
-        self.writeDefault({ "archive" : { "backend" : "file",
+        self.writeDefault({ "archive" : { "name" : "local",
+                                          "backend" : "file",
                                           "path" : self.archive.name }})
         self.writeConfig({ "bobMinimumVersion" : "0.20" })
         self.executeBobJenkinsCmd("add test http://localhost:{} -r root"
diff --git a/test/unit/test_archive.py b/test/unit/test_archive.py
index d94e121f5..7af7e0030 100644
--- a/test/unit/test_archive.py
+++ b/test/unit/test_archive.py
@@ -19,7 +19,7 @@
 import urllib.parse
 import sys
 
-from bob.archive import DummyArchive, SimpleHttpArchive, getArchiver
+from bob.archive import DummyArchive, HttpArchive, getArchiver
 from bob.errors import BuildError
 from bob.utils import runInEventLoop, getProcessPoolExecutor
 
@@ -479,8 +479,9 @@ def getCommon(self):
                 challenge = 'Basic ' + base64.b64encode(
                     (username+":"+password).encode("utf-8")).decode("ascii")
                 if self.headers.get('Authorization') != challenge:
+                    self.send_response(401, "Unauthorized")
                     self.send_header("WWW-Authenticate", 'Basic realm="default"')
-                    self.send_error(401, "Unauthorized")
+                    self.end_headers()
                     return None
 
             path = repoPath + self.path
@@ -499,14 +500,36 @@ def getCommon(self):
             return f
 
         def do_HEAD(self):
-            f = self.getCommon()
-            if f: f.close()
+            path = repoPath + self.path
+            # handle folder
+            if path.endswith('/'):
+                if os.path.exists(path):
+                    self.send_response(200)
+                    self.end_headers()
+                else:
+                    self.send_error(404, "Not found")
+            # handle file
+            else:
+                f = self.getCommon()
+                if f: f.close()
 
         def do_GET(self):
-            f = self.getCommon()
-            if f:
-                self.wfile.write(f.read())
-                f.close()
+            path = repoPath + self.path
+            # handle folder
+            if path.endswith('/'):
+                if os.path.exists(path):
+                    self.send_response(200)
+                    # just a random answer for directory
+                    self.wfile.write(path + " is doing fine")
+                    self.end_headers()
+                else:
+                    self.send_error(404, "Not found")
+            # handle file
+            else:
+                f = self.getCommon()
+                if f:
+                    self.wfile.write(f.read())
+                    f.close()
 
         def do_PUT(self):
             length = int(self.headers['Content-Length'])
@@ -558,6 +581,12 @@ def do_MKCOL(self):
                 self.send_response(403)
             self.end_headers()
 
+        def do_OPTIONS(self):
+            self.send_response(200)
+            self.send_header('DAV', '1,2')
+            self.send_header('Allow', "OPTIONS, GET, HEAD, PUT")
+            self.end_headers()
+
     return Handler
 
 class TestLocalArchive(BaseTester, TestCase):
@@ -583,6 +612,7 @@ def tearDown(self):
         super().tearDown()
 
     def _setArchiveSpec(self, spec):
+        spec['name'] = "remote"
         spec['backend'] = "http"
         spec["url"] = "http://{}:{}".format(self.ip, self.port)
 
@@ -590,7 +620,7 @@ def testInvalidServer(self):
         """Test download on non-existent server"""
 
         spec = { 'url' : "https://127.1.2.3:7257" }
-        archive = SimpleHttpArchive(spec)
+        archive = HttpArchive(spec)
         archive.wantDownloadLocal(True)
         archive.wantUploadLocal(True)
 
@@ -627,7 +657,7 @@ def testUnauthorized(self):
 
         spec = { }
         self._setArchiveSpec(spec, "wrong_password")
-        archive = SimpleHttpArchive(spec)
+        archive = HttpArchive(spec)
         archive.wantDownloadLocal(True)
         archive.wantUploadLocal(True)
 
diff --git a/test/unit/test_input_recipeset.py b/test/unit/test_input_recipeset.py
index 0e1ed61f4..5f0aa7c03 100644
--- a/test/unit/test_input_recipeset.py
+++ b/test/unit/test_input_recipeset.py
@@ -251,19 +251,23 @@ def testArchiveAppendPrepend(self):
             {
                 "archivePrepend" : [
                     {
+                        "name" : "local",
                         "backend" : "file",
                         "path" : "/foo/bar",
                     },
                     {
+                        "name" : "remote_prepend",
                         "backend" : "http",
                         "url" : "http://bob.test/prepend",
                     },
                 ],
                 "archive" : {
+                    "name" : "remote",
                     "backend" : "http",
                     "url" : "http://bob.test/main",
                 },
                 "archiveAppend" : {
+                    "name" : "remote_append",
                     "backend" : "http",
                     "url" : "http://bob.test/append",
                 },
@@ -274,18 +278,22 @@ def testArchiveAppendPrepend(self):
         self.assertEqual(
             [
                 {
+                    "name" : "local",
                     "backend" : "file",
                     "path" : "/foo/bar",
                 },
                 {
+                    "name" : "remote_prepend",
                     "backend" : "http",
                     "url" : "http://bob.test/prepend",
                 },
                 {
+                    "name" : "remote",
                     "backend" : "http",
                     "url" : "http://bob.test/main",
                 },
                 {
+                    "name" : "remote_append",
                     "backend" : "http",
                     "url" : "http://bob.test/append",
                 },
diff --git a/test/unit/test_webdav.py b/test/unit/test_webdav.py
new file mode 100644
index 000000000..3c98483bf
--- /dev/null
+++ b/test/unit/test_webdav.py
@@ -0,0 +1,234 @@
+from bob.webdav import WebDav, HttpAlreadyExistsError, HttpDownloadError, HttpNotFoundError, HttpUploadError
+from mocks.http_server import HttpServerMock
+from tempfile import TemporaryDirectory, NamedTemporaryFile
+from unittest import TestCase
+from urllib.parse import urlparse
+import os
+import time
+
+TEST_FILE="test.txt"
+TEST_FILE2="test2.txt"
+TEST_OUTPUT="testoutput"
+TEST_OUTPUT_SIZE=10
+TEST_PATH1="dir1"
+TEST_PATH2="dir2"
+
+def GetWebdav(port):
+    return WebDav(url=urlparse("http://localhost:{}/".format(port)), sslVerify=False)
+
+class TestWebdav(TestCase):
+
+    def setUp(self):
+        self.__repodir = TemporaryDirectory()
+        self.dir = self.__repodir.name
+        super().setUp()
+
+    def tearDown(self):
+        self.__repodir.cleanup()
+        super().tearDown()
+
+    def testUpload(self):
+        with HttpServerMock(repoPath=self.dir) as srv:
+            webdav = GetWebdav(srv.port)
+            with NamedTemporaryFile() as file:
+                file.write(TEST_OUTPUT.encode('utf-8'))
+                webdav.upload(TEST_FILE, file, False)
+                # check if file got uploaded and check the content
+                self.assertTrue(os.path.exists(os.path.join(self.dir, TEST_FILE)))
+                with open(os.path.join(self.dir, TEST_FILE)) as f:
+                    content = f.readline()
+                    self.assertEqual(TEST_OUTPUT, content)
+                # try to upload again should fail with HttpAlreadyExistsError
+                with self.assertRaises(HttpAlreadyExistsError) as cm:
+                    webdav.upload(TEST_FILE, file, False)
+                # append same content to temp file
+                file.write(TEST_OUTPUT.encode('utf-8'))
+                webdav.upload(TEST_FILE, file, True)
+                # content should be doubled in file now
+                with open(os.path.join(self.dir, TEST_FILE)) as f:
+                    content = f.readline()
+                    self.assertEqual(TEST_OUTPUT + TEST_OUTPUT, content)
+        # cleanup server directory
+        self.__repodir.cleanup()
+        # with 1 retry, the upload will fail initially
+        with HttpServerMock(repoPath=self.dir, retries=1) as srv:
+            webdav = GetWebdav(srv.port)
+            with NamedTemporaryFile() as file:
+                file.write(TEST_OUTPUT.encode('utf-8'))
+                with self.assertRaises(HttpUploadError) as cm:
+                    webdav.upload(TEST_FILE, file, False)
+                self.assertFalse(os.path.exists(os.path.join(self.dir, TEST_FILE)))
+                # second upload will succeed
+                webdav.upload(TEST_FILE, file, False)
+                self.assertTrue(os.path.exists(os.path.join(self.dir, TEST_FILE)))
+
+    def testDownload(self):
+        with HttpServerMock(self.dir) as srv:
+            # create file for server
+            fpath = os.path.join(self.dir, TEST_FILE)
+            with open(fpath, 'w') as f:
+                f.write(TEST_OUTPUT)
+            webdav = GetWebdav(srv.port)
+            file = webdav.download(TEST_FILE)
+            res = file.read()
+            self.assertEqual(res.decode('utf-8'), TEST_OUTPUT)
+            # add more data to file
+            with open(fpath, 'a') as f:
+                f.write(TEST_OUTPUT)
+            file = webdav.download(TEST_FILE)
+            res = file.read()
+            self.assertEqual(res.decode('utf-8'), TEST_OUTPUT+TEST_OUTPUT)
+            # test offset and length
+            file = webdav.download(TEST_FILE, offset=0, length=TEST_OUTPUT_SIZE)
+            res = file.read()
+            self.assertEqual(res.decode('utf-8'), TEST_OUTPUT)
+            # data is testoutputtestoutput, with offset 4 and length 10 we should get outputtest
+            file = webdav.download(TEST_FILE, offset=4, length=TEST_OUTPUT_SIZE)
+            res = file.read()
+            self.assertEqual(res.decode('utf-8'), 'outputtest')
+            # remove the file
+            os.unlink(os.path.join(self.dir, TEST_FILE))
+            # missing file will raise HttpNotFoundError
+            with self.assertRaises(HttpNotFoundError) as cm:
+                webdav.download(TEST_FILE)
+        # with 1 retry, the download will fail initially
+        with HttpServerMock(self.dir, retries=1) as srv:
+            webdav = GetWebdav(srv.port)
+            with self.assertRaises(HttpDownloadError) as cm:
+                webdav.download(TEST_FILE)
+
+    def testExists(self):
+        with HttpServerMock(self.dir) as srv:
+            # create file for server
+            with open(os.path.join(self.dir, TEST_FILE), 'w') as f:
+                f.write(TEST_OUTPUT)
+            webdav = GetWebdav(srv.port)
+            # file exists
+            self.assertTrue(webdav.exists(TEST_FILE))
+            # delete the file and exists shall return false
+            os.unlink(os.path.join(self.dir, TEST_FILE))
+            self.assertFalse(webdav.exists(TEST_FILE))
+        # with 1 retry, exists will fail initially
+        with HttpServerMock(self.dir, retries=1, retryHead=True) as srv:
+            webdav = GetWebdav(srv.port)
+            # file exists
+            with self.assertRaises(HttpUploadError) as cm:
+                webdav.exists(TEST_FILE)
+            self.assertFalse(webdav.exists(TEST_FILE))
+
+    def testDelete(self):
+        with HttpServerMock(self.dir) as srv:
+            # create file for server
+            path = os.path.join(self.dir, TEST_FILE)
+            with open(path, 'w') as f:
+                f.write(TEST_OUTPUT)
+            self.assertTrue(os.path.exists(path))
+            webdav = GetWebdav(srv.port)
+            webdav.delete(TEST_FILE)
+            # file should be deleted
+            self.assertFalse(os.path.exists(path))
+            with self.assertRaises(HttpDownloadError) as cm:
+                webdav.delete(TEST_FILE)
+
+    def testMkdir(self):
+        with HttpServerMock(self.dir) as srv:
+            ## depth 1
+            path = os.path.join(self.dir, TEST_PATH1)
+            webdav = GetWebdav(srv.port)
+            webdav.mkdir(TEST_PATH1)
+            self.assertTrue(os.path.exists(path))
+            ## path already exists, no error occurs
+            webdav.mkdir(TEST_PATH1)
+            self.assertTrue(os.path.exists(path))
+            os.rmdir(path)
+            ## depth 2
+            remote_path = TEST_PATH1 + '/' + TEST_PATH2
+            path = os.path.join(self.dir, TEST_PATH1, TEST_PATH2)
+            webdav = GetWebdav(srv.port)
+            # should raise exception with error code 409, because we have a depth of 2, but only request 1
+            with self.assertRaises(HttpUploadError) as cm:
+                webdav.mkdir(remote_path)
+            self.assertEqual("MKCOL 409 Conflict", str(cm.exception))
+            self.assertFalse(os.path.exists(path))
+            # depth 2, so it will create the parent dir here, too
+            webdav.mkdir(remote_path, 2)
+            self.assertTrue(os.path.exists(path))
+            os.rmdir(path)
+
+    def testListdir(self):
+        def checkEntries(entries, is_dir, paths):
+            def checkyEntry(entry, is_dir, paths):
+                return entry["is_dir"] == is_dir and entry["path"] in paths
+            self.assertEqual(len(paths), len(entries))
+            [self.assertTrue(checkyEntry(x, is_dir, paths)) for x in entries]
+
+        with HttpServerMock(self.dir) as srv:
+            # create some simple directory structure
+            path1 = os.path.join(self.dir, TEST_PATH1, TEST_PATH2)
+            path2 = os.path.join(self.dir, TEST_PATH2, TEST_PATH1)
+            os.makedirs(path1)
+            os.makedirs(path2)
+            open(os.path.join(path1, TEST_FILE), 'a')
+            open(os.path.join(path1, TEST_FILE2), 'a')
+            open(os.path.join(path2, TEST_FILE), 'a')
+            # check if listdir returns the proper content
+            webdav = GetWebdav(srv.port)
+            res = webdav.listdir('/')
+            checkEntries(res, True, [TEST_PATH1, TEST_PATH2])
+            res = webdav.listdir('/' + TEST_PATH1)
+            checkEntries(res, True, [TEST_PATH1 + '/' + TEST_PATH2])
+            base_path = TEST_PATH1 + '/' + TEST_PATH2 + '/'
+            res = webdav.listdir('/' + base_path)
+            checkEntries(res, False, [base_path + TEST_FILE, base_path + TEST_FILE2 ])
+            base_path = TEST_PATH2 + '/' + TEST_PATH1 + '/'
+            res = webdav.listdir('/' + base_path)
+            checkEntries(res, False, [base_path + TEST_FILE])
+
+            self.__repodir.cleanup()
+            # listdir on non-existing dir should return None
+            self.assertIsNone(webdav.listdir('/'))
+
+    def testStat(self):
+        with HttpServerMock(self.dir) as srv:
+            webdav = GetWebdav(srv.port)
+            path = os.path.join(self.dir, TEST_FILE)
+            # create file
+            with open(os.path.join(path), 'a') as f:
+                f.write(TEST_OUTPUT)
+            stats = os.stat(path)
+            res = webdav.stat(TEST_FILE)
+            # compare file stats with returned webdav stats
+            self.assertEqual(res['mdate'], time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime(stats[8])))
+            self.assertEqual(res['len'], stats[6])
+            self.assertEqual(res['etag'], f'{stats[6]:x}-{stats[1]:x}-{stats[8]:x}')
+            # modify file at a later time point
+            time.sleep(1)
+            with open(os.path.join(path), 'a') as f:
+                f.write(TEST_OUTPUT)
+            res2 = webdav.stat(TEST_FILE)
+            # new webdav stats should be different
+            self.assertNotEqual(res['mdate'], res2['mdate'])
+            self.assertNotEqual(res['len'], res2['len'])
+            self.assertNotEqual(res['etag'], res2['etag'])
+            # stat on a non-existing file should return None
+            os.unlink(path)
+            self.assertIsNone(webdav.stat(TEST_FILE))
+
+    def testPartialDownloader(self):
+        with HttpServerMock(self.dir) as srv:
+            webdav = GetWebdav(srv.port)
+            # create file for server
+            fpath = os.path.join(self.dir, TEST_FILE)
+            with open(fpath, 'w') as f:
+                f.write(TEST_OUTPUT + TEST_OUTPUT)
+            pd = webdav.getPartialDownloader(TEST_FILE, 4)
+            # first 4 bytes should be 'test'
+            self.assertEqual(pd.get().decode('utf-8'), 'test')
+            # getting 2 more bytes and we should get 'testou'
+            pd.more(2)
+            self.assertEqual(pd.get().decode('utf-8'), 'testou')
+            pd.more(4)
+            self.assertEqual(pd.get().decode('utf-8'), TEST_OUTPUT)
+            # requesting more than what we should have left will only give us the leftover
+            pd.more(11)
+            self.assertEqual(pd.get().decode('utf-8'), TEST_OUTPUT + TEST_OUTPUT)


Reviews:
user: login: jkloetzke
body: This turned out to be quite a lot of comments. I certainly did not catch everything yet but I hope you bear with me...
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T21:19:45Z
user: login: MarcKe
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T10:48:36Z
user: login: jkloetzke
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T19:33:05Z
user: login: jkloetzke
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T19:35:25Z
user: login: jkloetzke
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T19:44:01Z
user: login: jkloetzke
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T19:47:03Z
user: login: jkloetzke
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T19:49:36Z
user: login: MarcKe
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T08:53:26Z
user: login: MarcKe
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T08:57:37Z
user: login: jkloetzke
body: That looks quite good already. Would it be possible to add some testing for the Webdav parts? Maybe (re)use the HTTP mock server from the unit tests to have some minimally responsive Webdav server? Otherwise this is basically all untested and I fear it will break unnoticed whenever we touch the archive stuff...
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-15T20:56:37Z

Review comments:
diff_hunk: @@ -559,6 +582,119 @@ async def downloadLocalFingerprint(self, step, key, executor=None):
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
                 raise BuildError("Download of fingerprint interrupted.")
 
+    def deleteFile(self, filepath):
+        # Set default signal handler so that KeyboardInterrupt is raised.
path: pym/bob/archive.py
user: login: MarcKe
body: is that even necessary here?
created_at: 2025-02-21T13:54:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
path: pym/bob/webdav.py
user: login: jkloetzke
body: Do we really need to derive from `http.client.HTTPException`? These error look like new classes rather than amending the existing http exceptions.
created_at: 2025-03-03T19:43:52Z
author_association: MEMBER
diff_hunk: @@ -737,182 +734,51 @@ def __exit__(self, exc_type, exc_value, traceback):
         return False
 
 
-class SimpleHttpArchive(BaseArchive):
+class HttpArchive(BaseArchive):
     def __init__(self, spec):
         super().__init__(spec)
         self.__url = urllib.parse.urlparse(spec["url"])
-        self.__connection = None
-        self.__sslVerify = spec.get("sslVerify", True)
+        self.webdav = WebDav(spec)
path: pym/bob/archive.py
user: login: jkloetzke
body: The `WebDav` class should not get a `spec`. This is something `archive` specific. Instead, pass it the arguments explicitly. We want to keep the coupling as loose as possible.
created_at: 2025-03-03T19:48:08Z
author_association: MEMBER
diff_hunk: @@ -737,182 +734,51 @@ def __exit__(self, exc_type, exc_value, traceback):
         return False
 
 
-class SimpleHttpArchive(BaseArchive):
+class HttpArchive(BaseArchive):
     def __init__(self, spec):
         super().__init__(spec)
         self.__url = urllib.parse.urlparse(spec["url"])
-        self.__connection = None
-        self.__sslVerify = spec.get("sslVerify", True)
+        self.webdav = WebDav(spec)
 
-    def __retry(self, request):
path: pym/bob/archive.py
user: login: jkloetzke
body: Could we keep the retry logic in this module? Retries are not a concept of Webdav but rather the Bob archive handling.
created_at: 2025-03-03T19:50:20Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpUploadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpNotFoundError(http.client.HTTPException):
+    pass
+
+class HttpAlreadyExistsError(http.client.HTTPException):
+    pass
+
+class WebDav:
+
+    class PartialDownloader:
path: pym/bob/webdav.py
user: login: jkloetzke
body: This nested class, and the `getPartialDownloader()` factory method do not seem to be used any more.
created_at: 2025-03-03T19:57:39Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpUploadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpNotFoundError(http.client.HTTPException):
+    pass
+
+class HttpAlreadyExistsError(http.client.HTTPException):
+    pass
+
+class WebDav:
+
+    class PartialDownloader:
+        def __init__(self, webdav, path, length=512*1024):
+            self.__webdev = webdav
+            self.__path = path
+            self.__data = bytearray(self.__webdev.download(self.__path, 0, length).read())
+            self.__offset = length
+
+        def get(self):
+            return self.__data
+
+        def more(self, length=512*1024):
+            new_data = self.__webdev.download(self.__path, self.__offset, length)
+            self.__offset += length
+            self.__data.extend(new_data.read())
+            return self.__data
+
+    def __init__(self, spec):
+        self.__url = urlparse(spec["url"])
+        self.__connection = None
+        self.__sslVerify = spec.get("sslVerify", True)
+
+    def __retry(self, request):
+        retry = True
+        while True:
+            try:
+                return (True, request())
+            except (http.client.HTTPException, OSError) as e:
+                self._resetConnection()
+                if not retry: return (False, e)
+                retry = False
+
+    def getPartialDownloader(self, path):
+        return self.PartialDownloader(self, path)
+
+    def _getConnection(self):
+        if self.__connection is not None:
+            return self.__connection
+
+        url = self.__url
+        if url.scheme == 'http':
+            connection = http.client.HTTPConnection(url.hostname, url.port)
+        elif url.scheme == 'https':
+            ctx = None if self.__sslVerify else sslNoVerifyContext()
+            connection = http.client.HTTPSConnection(url.hostname, url.port,
+                                                     context=ctx)
+        else:
+            raise BuildError("Unsupported URL scheme: '{}'".format(url.scheme))
+
+        # simple check if webdav is supported
+        connection.request("OPTIONS", '/'.join([self.__url.path, '']), headers=self._getHeaders())
+
+        response = connection.getresponse()
+        response.read()
+        dav_supported = response.getheader("DAV") is not None
+        if not dav_supported:
+            raise BuildError("WEBDAV not supported by server")
path: pym/bob/webdav.py
user: login: jkloetzke
body: Do we really need to check that? At least for downloads it's harmful because Webdav support is not needed. So if this is required, it really must be checked only for `MKCOL` or similar requirests. But even then I would assume we get a proper failure if the server does not support it.

Keep in mind this is in the hot path of artifact downloads.
created_at: 2025-03-03T20:00:11Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpUploadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpNotFoundError(http.client.HTTPException):
+    pass
+
+class HttpAlreadyExistsError(http.client.HTTPException):
+    pass
+
+class WebDav:
+
+    class PartialDownloader:
+        def __init__(self, webdav, path, length=512*1024):
+            self.__webdev = webdav
+            self.__path = path
+            self.__data = bytearray(self.__webdev.download(self.__path, 0, length).read())
+            self.__offset = length
+
+        def get(self):
+            return self.__data
+
+        def more(self, length=512*1024):
+            new_data = self.__webdev.download(self.__path, self.__offset, length)
+            self.__offset += length
+            self.__data.extend(new_data.read())
+            return self.__data
+
+    def __init__(self, spec):
+        self.__url = urlparse(spec["url"])
+        self.__connection = None
+        self.__sslVerify = spec.get("sslVerify", True)
+
+    def __retry(self, request):
+        retry = True
+        while True:
+            try:
+                return (True, request())
+            except (http.client.HTTPException, OSError) as e:
+                self._resetConnection()
+                if not retry: return (False, e)
+                retry = False
+
+    def getPartialDownloader(self, path):
+        return self.PartialDownloader(self, path)
+
+    def _getConnection(self):
+        if self.__connection is not None:
+            return self.__connection
+
+        url = self.__url
+        if url.scheme == 'http':
+            connection = http.client.HTTPConnection(url.hostname, url.port)
+        elif url.scheme == 'https':
+            ctx = None if self.__sslVerify else sslNoVerifyContext()
+            connection = http.client.HTTPSConnection(url.hostname, url.port,
+                                                     context=ctx)
+        else:
+            raise BuildError("Unsupported URL scheme: '{}'".format(url.scheme))
+
+        # simple check if webdav is supported
+        connection.request("OPTIONS", '/'.join([self.__url.path, '']), headers=self._getHeaders())
+
+        response = connection.getresponse()
+        response.read()
+        dav_supported = response.getheader("DAV") is not None
+        if not dav_supported:
+            raise BuildError("WEBDAV not supported by server")
+
+        self.__connection = connection
+        return connection
+
+    def _resetConnection(self):
+        if self.__connection is not None:
+            self.__connection.close()
+            self.__connection = None
+
+    def _getHeaders(self):
+        headers = {'User-Agent': 'BobBuildTool/{}'.format(BOB_VERSION)}
+        if self.__url.username is not None:
+            username = unquote(self.__url.username)
+            passwd = unquote(self.__url.password)
+            userPass = username + ":" + passwd
+            headers['Authorization'] = 'Basic ' + base64.b64encode(
+                userPass.encode("utf-8")).decode("ascii")
+        return headers
+
+    def _check(self, path):
+        connection = self._getConnection()
+        connection.request("HEAD", path, headers=self._getHeaders())
+        response = connection.getresponse()
+        response.read()
+        if response.status == 200:
+            return True
+        elif response.status == 404:
+            return False
+        else:
+            raise HttpUploadError("HEAD {} {}".format(response.status, response.reason))
+
+    def check(self, path):
path: pym/bob/webdav.py
user: login: jkloetzke
body: How about naming this `exists`? To mirror the `os.path` style...
created_at: 2025-03-03T20:03:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpUploadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpNotFoundError(http.client.HTTPException):
+    pass
+
+class HttpAlreadyExistsError(http.client.HTTPException):
+    pass
+
+class WebDav:
+
+    class PartialDownloader:
+        def __init__(self, webdav, path, length=512*1024):
+            self.__webdev = webdav
+            self.__path = path
+            self.__data = bytearray(self.__webdev.download(self.__path, 0, length).read())
+            self.__offset = length
+
+        def get(self):
+            return self.__data
+
+        def more(self, length=512*1024):
+            new_data = self.__webdev.download(self.__path, self.__offset, length)
+            self.__offset += length
+            self.__data.extend(new_data.read())
+            return self.__data
+
+    def __init__(self, spec):
+        self.__url = urlparse(spec["url"])
+        self.__connection = None
+        self.__sslVerify = spec.get("sslVerify", True)
+
+    def __retry(self, request):
+        retry = True
+        while True:
+            try:
+                return (True, request())
+            except (http.client.HTTPException, OSError) as e:
+                self._resetConnection()
+                if not retry: return (False, e)
+                retry = False
+
+    def getPartialDownloader(self, path):
+        return self.PartialDownloader(self, path)
+
+    def _getConnection(self):
+        if self.__connection is not None:
+            return self.__connection
+
+        url = self.__url
+        if url.scheme == 'http':
+            connection = http.client.HTTPConnection(url.hostname, url.port)
+        elif url.scheme == 'https':
+            ctx = None if self.__sslVerify else sslNoVerifyContext()
+            connection = http.client.HTTPSConnection(url.hostname, url.port,
+                                                     context=ctx)
+        else:
+            raise BuildError("Unsupported URL scheme: '{}'".format(url.scheme))
+
+        # simple check if webdav is supported
+        connection.request("OPTIONS", '/'.join([self.__url.path, '']), headers=self._getHeaders())
+
+        response = connection.getresponse()
+        response.read()
+        dav_supported = response.getheader("DAV") is not None
+        if not dav_supported:
+            raise BuildError("WEBDAV not supported by server")
+
+        self.__connection = connection
+        return connection
+
+    def _resetConnection(self):
+        if self.__connection is not None:
+            self.__connection.close()
+            self.__connection = None
+
+    def _getHeaders(self):
+        headers = {'User-Agent': 'BobBuildTool/{}'.format(BOB_VERSION)}
+        if self.__url.username is not None:
+            username = unquote(self.__url.username)
+            passwd = unquote(self.__url.password)
+            userPass = username + ":" + passwd
+            headers['Authorization'] = 'Basic ' + base64.b64encode(
+                userPass.encode("utf-8")).decode("ascii")
+        return headers
+
+    def _check(self, path):
+        connection = self._getConnection()
+        connection.request("HEAD", path, headers=self._getHeaders())
+        response = connection.getresponse()
+        response.read()
+        if response.status == 200:
+            return True
+        elif response.status == 404:
+            return False
+        else:
+            raise HttpUploadError("HEAD {} {}".format(response.status, response.reason))
+
+    def check(self, path):
+        (ok, result) = self.__retry(lambda: self._check(path))
+        if ok:
+            return result
+        else:
+            raise result
+
+    def _download(self, path, offset=None, length=None):
+        connection = self._getConnection()
+        headers = self._getHeaders()
+        if offset is not None and length is not None:
+            headers.update({'Range': 'bytes={}-{}'.format(offset, offset + length - 1)})
+        connection.request("GET", path, headers=headers)
+        response = connection.getresponse()
+        if response.status in [200, 206]:
+            return response
+        else:
+            response.read()
+            if response.status == 404:
+                raise HttpNotFoundError()
+            else:
+                raise HttpDownloadError("{} {}".format(response.status,
+                                                           response.reason))
+
+    def download(self, path, offset=None, length=None):
+        (ok, result) = self.__retry(lambda: self._download(path, offset, length))
+        if ok:
+            return result
+        else:
+            raise result
+
+    def _upload(self, path, buf, overwrite):
+        # Determine file length ourselves and add a "Content-Length" header. This
+        # used to work in Python 3.5 automatically but was removed later.
+        buf.seek(0, os.SEEK_END)
+        length = str(buf.tell())
+        buf.seek(0)
+        headers = self._getHeaders()
+        headers.update({'Content-Length': length})
+        if not overwrite:
+            headers.update({'If-None-Match': '*'})
+        connection = self._getConnection()
+        connection.request("PUT", path, buf, headers=headers)
+        response = connection.getresponse()
+        response.read()
+        if response.status == 412:
+            # precondition failed -> lost race with other upload
+            raise HttpAlreadyExistsError()
+        elif response.status not in [200, 201, 204]:
+            raise HttpUploadError("PUT {} {}".format(response.status, response.reason))
+
+    def upload(self, path, buf, overwrite):
+        (ok, result) = self.__retry(lambda: self._upload(path, buf, overwrite))
+        if ok:
+            return result
+        else:
+            raise result
+
+    def _mkdir(self, path):
+        if not self.check(path):
+            headers = self._getHeaders()
+            connection = self._getConnection()
+            connection.request("MKCOL", path, headers=headers)
+            response = connection.getresponse()
+            response.read()
+            if response.status not in [200, 201]:
+                raise HttpUploadError("MKCOL {} {}".format(response.status, response.reason))
+
+    def mkdir(self, path):
+        (ok, result) = self.__retry(lambda: self._mkdir(path))
+        if ok:
+            return result
+        else:
+            raise result
+
+    def _list(self, path):
+        base_path = self.__url.path
+        # create a full path ending with trailing / (should prevent http 301 - moved permanently)
+        path = '/'.join([base_path, path.strip('/'), ''])
+        if self.check(path):
+            headers = self._getHeaders()
+            # Depth: 1 - applies to the resource and the immediate children (infinity usually prohibited by server)
+            headers.update({'Depth': '1'})
+            connection = self._getConnection()
+            connection.request("PROPFIND", path, headers=headers)
+            response = connection.getresponse()
+            if response.status not in [207]:
+                raise HttpDownloadError("PROPFIND {} {}".format(response.status, response.reason))
+            content = response.read()
+            # get all dav responses from multistatusresponse
+            tree = fromstring(content)
+            dir_infos = []
+            for resp in tree.findall(".//{DAV:}response"):
+                # only need the path in case the full URL is included
+                href = unquote(urlsplit(resp.findtext(".//{DAV:}href")).path)
+                # exclude base path
+                if href.strip('/') == path.strip('/'):
+                    continue
+                dir_info = dict()
+                # collect if it is a dir, the href and self defined path (href without base path)
+                dir_info['is_dir'] = resp.find(".//{DAV:}collection") is not None
+                dir_info['href'] = href
+                dir_info['path'] = href[len(base_path):].strip('/')
+                dir_infos.append(dir_info)
+            return dir_infos
+
+    def list(self, path):
path: pym/bob/webdav.py
user: login: jkloetzke
body: Maybe `listdir`?
created_at: 2025-03-03T20:05:13Z
author_association: MEMBER
diff_hunk: @@ -737,182 +734,51 @@ def __exit__(self, exc_type, exc_value, traceback):
         return False
 
 
-class SimpleHttpArchive(BaseArchive):
+class HttpArchive(BaseArchive):
     def __init__(self, spec):
         super().__init__(spec)
         self.__url = urllib.parse.urlparse(spec["url"])
-        self.__connection = None
-        self.__sslVerify = spec.get("sslVerify", True)
+        self.webdav = WebDav(spec)
 
-    def __retry(self, request):
-        retry = True
-        while True:
-            try:
-                return (True, request())
-            except (http.client.HTTPException, OSError) as e:
-                self._resetConnection()
-                if not retry: return (False, e)
-                retry = False
+    def _resetConnection(self):
+        self.webdav._resetConnection()
 
-    def _makeUrl(self, buildId, suffix):
+    def _makePath(self, buildId, suffix):
         packageResultId = buildIdToName(buildId)
         return "/".join([self.__url.path, packageResultId[0:2], packageResultId[2:4],
             packageResultId[4:] + suffix])
 
-    def _remoteName(self, buildId, suffix):
-        url = self.__url
-        return urllib.parse.urlunparse((url.scheme, url.netloc, self._makeUrl(buildId, suffix), '', '', ''))
-
-    def _getConnection(self):
-        if self.__connection is not None:
-            return self.__connection
+    def _create_dirs(self, buildId):
+        packageResultId = buildIdToName(buildId)
+        path = "/".join([self.__url.path, packageResultId[0:2], ''])
path: pym/bob/archive.py
user: login: jkloetzke
body: No. Adding the `''` to force a trailing `/` is not the right approach. Also, it's way to subtle and could cause breakage in the future.
created_at: 2025-03-03T20:09:17Z
author_association: MEMBER
diff_hunk: @@ -737,182 +734,51 @@ def __exit__(self, exc_type, exc_value, traceback):
         return False
 
 
-class SimpleHttpArchive(BaseArchive):
+class HttpArchive(BaseArchive):
     def __init__(self, spec):
         super().__init__(spec)
         self.__url = urllib.parse.urlparse(spec["url"])
-        self.__connection = None
-        self.__sslVerify = spec.get("sslVerify", True)
+        self.webdav = WebDav(spec)
 
-    def __retry(self, request):
-        retry = True
-        while True:
-            try:
-                return (True, request())
-            except (http.client.HTTPException, OSError) as e:
-                self._resetConnection()
-                if not retry: return (False, e)
-                retry = False
+    def _resetConnection(self):
+        self.webdav._resetConnection()
 
-    def _makeUrl(self, buildId, suffix):
+    def _makePath(self, buildId, suffix):
         packageResultId = buildIdToName(buildId)
         return "/".join([self.__url.path, packageResultId[0:2], packageResultId[2:4],
             packageResultId[4:] + suffix])
 
-    def _remoteName(self, buildId, suffix):
-        url = self.__url
-        return urllib.parse.urlunparse((url.scheme, url.netloc, self._makeUrl(buildId, suffix), '', '', ''))
-
-    def _getConnection(self):
-        if self.__connection is not None:
-            return self.__connection
+    def _create_dirs(self, buildId):
+        packageResultId = buildIdToName(buildId)
+        path = "/".join([self.__url.path, packageResultId[0:2], ''])
+        self.webdav.mkdir(path)
+        path = "/".join([self.__url.path, packageResultId[0:2], packageResultId[2:4], ''])
+        self.webdav.mkdir(path)
 
+    def _remoteName(self, buildId, suffix):
         url = self.__url
-        if url.scheme == 'http':
-            connection = http.client.HTTPConnection(url.hostname, url.port)
-        elif url.scheme == 'https':
-            ctx = None if self.__sslVerify else sslNoVerifyContext()
-            connection = http.client.HTTPSConnection(url.hostname, url.port,
-                                                     context=ctx)
-        else:
-            raise BuildError("Unsupported URL scheme: '{}'".format(url.schema))
-
-        self.__connection = connection
-        return connection
-
-    def _resetConnection(self):
-        if self.__connection is not None:
-            self.__connection.close()
-            self.__connection = None
-
-    def _getHeaders(self):
-        headers = { 'User-Agent' : 'BobBuildTool/{}'.format(BOB_VERSION) }
-        if self.__url.username is not None:
-            username = urllib.parse.unquote(self.__url.username)
-            passwd = urllib.parse.unquote(self.__url.password)
-            userPass = username + ":" + passwd
-            headers['Authorization'] = 'Basic ' + base64.b64encode(
-                userPass.encode("utf-8")).decode("ascii")
-        return headers
+        return urllib.parse.urlunparse((url.scheme, url.netloc, self._makePath(buildId, suffix), '', '', ''))
 
     def _openDownloadFile(self, buildId, suffix):
-        (ok, result) = self.__retry(lambda: self.__openDownloadFile(buildId, suffix))
-        if ok:
-            return result
-        else:
-            raise ArtifactDownloadError(str(result))
-
-    def __openDownloadFile(self, buildId, suffix):
-        connection = self._getConnection()
-        url = self._makeUrl(buildId, suffix)
-        connection.request("GET", url, headers=self._getHeaders())
-        response = connection.getresponse()
-        if response.status == 200:
-            return SimpleHttpDownloader(self, response)
-        else:
-            response.read()
-            if response.status == 404:
-                raise ArtifactNotFoundError()
-            else:
-                raise ArtifactDownloadError("{} {}".format(response.status,
-                                                           response.reason))
+        url = self._makePath(buildId, suffix)
+        return HttpDownloader(self, self.webdav.download(url))
 
     def _openUploadFile(self, buildId, suffix, overwrite):
-        (ok, result) = self.__retry(lambda: self.__openUploadFile(buildId, suffix, overwrite))
-        if ok:
-            return result
-        else:
-            raise ArtifactUploadError(str(result))
-
-    def __openUploadFile(self, buildId, suffix, overwrite):
-        connection = self._getConnection()
-        url = self._makeUrl(buildId, suffix)
-
+        url = self._makePath(buildId, suffix)
         # check if already there
         if not overwrite:
-            connection.request("HEAD", url, headers=self._getHeaders())
-            response = connection.getresponse()
-            response.read()
-            if response.status == 200:
+            if self.webdav.check(url):
                 raise ArtifactExistsError()
-            elif response.status != 404:
-                raise ArtifactUploadError("HEAD {} {}".format(response.status, response.reason))
 
-        # create temporary file
-        return SimpleHttpUploader(self, url, overwrite)
+        # create dirs based on buildId to make 100% sure they are available
+        self._create_dirs(buildId)
+        return HttpUploader(self, url, overwrite)
 
     def _putUploadFile(self, url, tmp, overwrite):
-        (ok, result) = self.__retry(lambda: self.__putUploadFile(url, tmp, overwrite))
-        if ok:
-            return result
-        else:
-            raise ArtifactUploadError(str(result))
-
-    def __putUploadFile(self, url, tmp, overwrite):
-        # Quoting RFC 4918:
-        #
-        #   A PUT that would result in the creation of a resource without an
-        #   appropriately scoped parent collection MUST fail with a 409
-        #   (Conflict).
-        #
-        # We don't want to waste bandwith by uploading big artifacts into a
-        # missing directory. Thus make sure the directory always exists.
path: pym/bob/archive.py
user: login: jkloetzke
body: Could we please keep such comments that explain *why* this and that is done? Unless they are obsolete or wrong, I would prefer to keep them.
created_at: 2025-03-03T20:11:17Z
author_association: MEMBER
diff_hunk: @@ -737,182 +734,51 @@ def __exit__(self, exc_type, exc_value, traceback):
         return False
 
 
-class SimpleHttpArchive(BaseArchive):
+class HttpArchive(BaseArchive):
     def __init__(self, spec):
         super().__init__(spec)
         self.__url = urllib.parse.urlparse(spec["url"])
-        self.__connection = None
-        self.__sslVerify = spec.get("sslVerify", True)
+        self.webdav = WebDav(spec)
 
-    def __retry(self, request):
-        retry = True
-        while True:
-            try:
-                return (True, request())
-            except (http.client.HTTPException, OSError) as e:
-                self._resetConnection()
-                if not retry: return (False, e)
-                retry = False
+    def _resetConnection(self):
+        self.webdav._resetConnection()
 
-    def _makeUrl(self, buildId, suffix):
+    def _makePath(self, buildId, suffix):
         packageResultId = buildIdToName(buildId)
         return "/".join([self.__url.path, packageResultId[0:2], packageResultId[2:4],
             packageResultId[4:] + suffix])
 
-    def _remoteName(self, buildId, suffix):
-        url = self.__url
-        return urllib.parse.urlunparse((url.scheme, url.netloc, self._makeUrl(buildId, suffix), '', '', ''))
-
-    def _getConnection(self):
-        if self.__connection is not None:
-            return self.__connection
+    def _create_dirs(self, buildId):
+        packageResultId = buildIdToName(buildId)
+        path = "/".join([self.__url.path, packageResultId[0:2], ''])
+        self.webdav.mkdir(path)
+        path = "/".join([self.__url.path, packageResultId[0:2], packageResultId[2:4], ''])
+        self.webdav.mkdir(path)
 
+    def _remoteName(self, buildId, suffix):
         url = self.__url
-        if url.scheme == 'http':
-            connection = http.client.HTTPConnection(url.hostname, url.port)
-        elif url.scheme == 'https':
-            ctx = None if self.__sslVerify else sslNoVerifyContext()
-            connection = http.client.HTTPSConnection(url.hostname, url.port,
-                                                     context=ctx)
-        else:
-            raise BuildError("Unsupported URL scheme: '{}'".format(url.schema))
-
-        self.__connection = connection
-        return connection
-
-    def _resetConnection(self):
-        if self.__connection is not None:
-            self.__connection.close()
-            self.__connection = None
-
-    def _getHeaders(self):
-        headers = { 'User-Agent' : 'BobBuildTool/{}'.format(BOB_VERSION) }
-        if self.__url.username is not None:
-            username = urllib.parse.unquote(self.__url.username)
-            passwd = urllib.parse.unquote(self.__url.password)
-            userPass = username + ":" + passwd
-            headers['Authorization'] = 'Basic ' + base64.b64encode(
-                userPass.encode("utf-8")).decode("ascii")
-        return headers
+        return urllib.parse.urlunparse((url.scheme, url.netloc, self._makePath(buildId, suffix), '', '', ''))
 
     def _openDownloadFile(self, buildId, suffix):
-        (ok, result) = self.__retry(lambda: self.__openDownloadFile(buildId, suffix))
-        if ok:
-            return result
-        else:
-            raise ArtifactDownloadError(str(result))
-
-    def __openDownloadFile(self, buildId, suffix):
-        connection = self._getConnection()
-        url = self._makeUrl(buildId, suffix)
-        connection.request("GET", url, headers=self._getHeaders())
-        response = connection.getresponse()
-        if response.status == 200:
-            return SimpleHttpDownloader(self, response)
-        else:
-            response.read()
-            if response.status == 404:
-                raise ArtifactNotFoundError()
-            else:
-                raise ArtifactDownloadError("{} {}".format(response.status,
-                                                           response.reason))
+        url = self._makePath(buildId, suffix)
+        return HttpDownloader(self, self.webdav.download(url))
 
     def _openUploadFile(self, buildId, suffix, overwrite):
-        (ok, result) = self.__retry(lambda: self.__openUploadFile(buildId, suffix, overwrite))
-        if ok:
-            return result
-        else:
-            raise ArtifactUploadError(str(result))
-
-    def __openUploadFile(self, buildId, suffix, overwrite):
-        connection = self._getConnection()
-        url = self._makeUrl(buildId, suffix)
-
+        url = self._makePath(buildId, suffix)
         # check if already there
         if not overwrite:
-            connection.request("HEAD", url, headers=self._getHeaders())
-            response = connection.getresponse()
-            response.read()
-            if response.status == 200:
+            if self.webdav.check(url):
                 raise ArtifactExistsError()
-            elif response.status != 404:
-                raise ArtifactUploadError("HEAD {} {}".format(response.status, response.reason))
 
-        # create temporary file
-        return SimpleHttpUploader(self, url, overwrite)
+        # create dirs based on buildId to make 100% sure they are available
+        self._create_dirs(buildId)
+        return HttpUploader(self, url, overwrite)
 
     def _putUploadFile(self, url, tmp, overwrite):
-        (ok, result) = self.__retry(lambda: self.__putUploadFile(url, tmp, overwrite))
-        if ok:
-            return result
-        else:
-            raise ArtifactUploadError(str(result))
-
-    def __putUploadFile(self, url, tmp, overwrite):
-        # Quoting RFC 4918:
-        #
-        #   A PUT that would result in the creation of a resource without an
-        #   appropriately scoped parent collection MUST fail with a 409
-        #   (Conflict).
-        #
-        # We don't want to waste bandwith by uploading big artifacts into a
-        # missing directory. Thus make sure the directory always exists.
-        self.__makeParentDirs(url)
-
-        # Determine file length outself and add a "Content-Length" header. This
-        # used to work in Python 3.5 automatically but was removed later.
-        tmp.seek(0, os.SEEK_END)
-        length = str(tmp.tell())
-        tmp.seek(0)
-        headers = self._getHeaders()
-        headers.update({ 'Content-Length' : length })
-        if not overwrite:
-            headers.update({ 'If-None-Match' : '*' })
-        connection = self._getConnection()
-        connection.request("PUT", url, tmp, headers=headers)
-        response = connection.getresponse()
-        response.read()
-        if response.status == 412:
-            # precondition failed -> lost race with other upload
-            raise ArtifactExistsError()
-        elif response.status not in [200, 201, 204]:
-            raise ArtifactUploadError("PUT {} {}".format(response.status, response.reason))
+        return self.webdav.upload(url, tmp, overwrite)
 
-    def __makeParentDirs(self, url, depth=0):
path: pym/bob/archive.py
user: login: jkloetzke
body: Previously, we tried to make only one request because parent directory usually exists. I would like to keep it that way and also with all the comments.
created_at: 2025-03-03T20:15:39Z
author_association: MEMBER
diff_hunk: @@ -393,7 +425,6 @@ def _downloadPackage(self, buildId, suffix, audit, content, caches, workspace):
         # Set default signal handler so that KeyboardInterrupt is raised.
         # Needed to gracefully handle ctrl+c.
         signal.signal(signal.SIGINT, signal.default_int_handler)
-
path: pym/bob/archive.py
user: login: jkloetzke
body: Unrelated change
created_at: 2025-03-03T20:17:48Z
author_association: MEMBER
diff_hunk: @@ -556,6 +587,119 @@ async def downloadLocalFingerprint(self, step, key, executor=None):
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
                 raise BuildError("Download of fingerprint interrupted.")
 
+    def deleteFile(self, filepath):
+        # Set default signal handler so that KeyboardInterrupt is raised.
+        # Needed to gracefully handle ctrl+c.
+        signal.signal(signal.SIGINT, signal.default_int_handler)
path: pym/bob/archive.py
user: login: jkloetzke
body: This whole `signal` dance is only required for the functions above because they are executed in an `asnycio` executor. (notice the `async def` of the public functions). The new functions are blocking and don't need this.
created_at: 2025-03-03T20:24:51Z
author_association: MEMBER
diff_hunk: @@ -361,6 +390,9 @@ def canCache(self):
     def _openDownloadFile(self, buildId, suffix):
         raise ArtifactNotFoundError()
 
+    def archiveCmdSupported(self):
path: pym/bob/archive.py
user: login: jkloetzke
body: How about `canManage`?
created_at: 2025-03-03T20:30:20Z
author_association: MEMBER
diff_hunk: @@ -155,6 +160,27 @@ def _extract(self, fileobj, audit, content):
             os.makedirs(content)
             self.__extractPackage(tar, audit, content)
 
+    def _extractAudit(self, filename=None, fileobj=None):
+        with tarfileOpen(name=filename, mode="r|*", fileobj=fileobj, errorlevel=1) as tar:
+            # validate
+            if tar.pax_headers.get('bob-archive-vsn') != "1":
+                print("\tNot a valid bob tar archive: ", filename)
path: pym/bob/archive.py
user: login: jkloetzke
body: Generic code should not print. Only command related code should do that.
created_at: 2025-03-03T20:35:13Z
author_association: MEMBER
diff_hunk: @@ -777,6 +950,41 @@ def _openUploadFile(self, buildId, suffix, overwrite):
     def _putUploadFile(self, url, tmp, overwrite):
         return self.webdav.upload(url, tmp, overwrite)
 
+    def _listDir(self, path):
+        path_info = self.webdav.list(path)
+        entries = []
+        for info in path_info:
+            if info["path"]:
+                entries.append(info["path"].removeprefix(path + "/"))
+        return entries
+
+    def _delete(self, filename):
+        self.webdav.delete(filename)
+
+    def _stat(self, filename):
+        stats = self.webdav.stat(filename)
+        if not stats['cdate'] or not stats['mdate'] or not stats['etag'] or not stats['len']:
+            print("error")
path: pym/bob/archive.py
user: login: jkloetzke
body: Debug left over? Also, the code will crash below in this case, doesn't it?
created_at: 2025-03-03T20:39:06Z
author_association: MEMBER
diff_hunk: @@ -777,6 +950,41 @@ def _openUploadFile(self, buildId, suffix, overwrite):
     def _putUploadFile(self, url, tmp, overwrite):
         return self.webdav.upload(url, tmp, overwrite)
 
+    def _listDir(self, path):
+        path_info = self.webdav.list(path)
+        entries = []
+        for info in path_info:
+            if info["path"]:
+                entries.append(info["path"].removeprefix(path + "/"))
+        return entries
+
+    def _delete(self, filename):
+        self.webdav.delete(filename)
+
+    def _stat(self, filename):
+        stats = self.webdav.stat(filename)
+        if not stats['cdate'] or not stats['mdate'] or not stats['etag'] or not stats['len']:
+            print("error")
+        from datetime import datetime
+        from email.utils import parsedate_to_datetime
+        import binascii
+        return struct.pack('=ddLL', datetime.fromisoformat(stats['cdate']).timestamp(), parsedate_to_datetime(stats['mdate']).timestamp(), binascii.crc32(stats['etag'].encode('utf8')), stats['len'])
path: pym/bob/archive.py
user: login: jkloetzke
body: Shouldn't we use the `ETag` right away if it's available? Note that we should probably discard weak ETags (starting with `W/`. Only if there is no Etag I would fall back to date and length.

I would consider the `crc32` hashing too weak. It's really prone to collisions and seems unnecessary here.
created_at: 2025-03-03T20:45:47Z
author_association: MEMBER
diff_hunk: @@ -3,12 +3,11 @@
 #
 # SPDX-License-Identifier: GPL-3.0-or-later
 
-from ..audit import Audit
 from ..errors import BobError
 from ..utils import binStat, asHexStr, infixBinaryOp, tarfileOpen
+from ..archive import *
path: pym/bob/cmds/archive.py
user: login: jkloetzke
body: Please don't use `*` imports. That is fine for a small script but I would consider this an anti-pattern for large code bases.
created_at: 2025-03-03T20:47:05Z
author_association: MEMBER
diff_hunk: @@ -23,15 +22,17 @@
 class ArchiveScanner:
     CUR_VERSION = 2
 
-    def __init__(self):
+    def __init__(self, archiver):
         self.__dirSchema = re.compile(r'[0-9a-zA-Z]{2}')
         self.__archiveSchema = re.compile(r'[0-9a-zA-Z]{36,}-1.tgz')
         self.__db = None
         self.__cleanup = False
+        self.__archiver = archiver
+        self.__dbName = ".bob-archive_{}.sqlite3".format(self.__archiver.getArchiveHash())
path: pym/bob/cmds/archive.py
user: login: jkloetzke
body: And what happens on hash collisions? The CRC32 is not very resilient. How about storing everything in one database with full URL? There could be a separate "archives" table for the archive URLs. All rows in the "files" could have a foreign key to the "archives" primary key.
created_at: 2025-03-03T20:55:58Z
author_association: MEMBER
diff_hunk: @@ -149,15 +134,20 @@ def __scan(self, fileName, verbose):
             self.__db.executemany("INSERT OR IGNORE INTO refs VALUES (?, ?)",
                 [ (bid, r) for r in audit.getReferencedBuildIds() ])
         except tarfile.TarError as e:
+            print("tar")
path: pym/bob/cmds/archive.py
user: login: jkloetzke
body: Debug left over I guess. Same below...
created_at: 2025-03-03T20:56:24Z
author_association: MEMBER
diff_hunk: @@ -154,6 +154,15 @@ expect_fail()
 	return 1
 }
 
+expect_success()
path: test/test-lib.sh
user: login: jkloetzke
body: All tests run with `set -e`. So implicitly every command is prefixed by `expect_success`.
created_at: 2025-03-03T21:04:38Z
author_association: MEMBER
diff_hunk: @@ -581,6 +581,12 @@ def do_MKCOL(self):
                 self.send_response(403)
             self.end_headers()
 
+        def do_OPTIONS(self):
path: test/unit/test_archive.py
user: login: jkloetzke
body: That could go to a preparatory commit. Otherwise, the test adaptations, should be in the same commit that changes the code. The goal is that is must be possible to `git bisect` and run the tests at any stage...
created_at: 2025-03-03T21:06:08Z
author_association: MEMBER
diff_hunk: @@ -484,40 +484,52 @@ def doArchive(argv, bobRoot):
 
   bob archive {}
 """.format(subHelp))
+    parser.add_argument("-l", "--local", action='store_true',
+                        help="ignore the archive configuration and work on the current directory")
     parser.add_argument('subcommand', help="Subcommand")
     parser.add_argument('args', nargs=argparse.REMAINDER,
                         help="Arguments for subcommand")
 
     args = parser.parse_args(argv)
 
     # get archiver
-    recipes = RecipeSet()
-    recipes.parse()
-    archivespec = recipes.archiveSpec()
     archivers = []
-    if isinstance(archivespec, list):
-        if len(archivespec) == 0:
-            raise BobError("No archiver defined")
-        elif len(archivespec) == 1:
-            archiver = getSingleArchiver(recipes, archivespec[0])
-            if archiver.archiveCmdSupported():
-                archivers.append(archiver)
-            else:
-                raise BobError("Archiver does not support the archive command")
+    if args.local:
+        # provide local directory as backend
+        archivespec = [{"backend": "file", "path": "{}".format(os.getcwd())}]
+        archiver = getSingleArchiver(None, archivespec[0])
+        if archiver.archiveCmdSupported():
path: pym/bob/cmds/archive.py
user: login: jkloetzke
body: Can this ever be false? Maybe `assert` instead...
created_at: 2025-03-03T21:09:05Z
author_association: MEMBER
diff_hunk: @@ -18,89 +18,120 @@ function clean_output () {
   echo $(echo "$@" | sed 1d | sed "s/^[ \t]*//")
 }
 
-# fill archive
-run_bob build --download=no --upload root-alpha root-bravo
-FINGERPRINT=Alice run_bob build --force --download=no --upload root-alpha root-bravo
-
-run_bob archive scan
-
-# first try to keep everything by using multiple expressions
-oldNum=$(/usr/bin/find -name '*.tgz' | wc -l)
-run_bob archive clean -v 'metaEnv.TYPE == "alpha"' 'meta.package == "root-bravo"'
-newNum=$(/usr/bin/find -name '*.tgz' | wc -l)
-test $oldNum -eq $newNum
-
-# Find one of the artifacts
-found=$(clean_output "$(run_bob archive find 'meta.package == "root-bravo"')")
-pushd $archiveDir
-expect_exist "$found"
-popd
-
-# selectively keep one half
-run_bob archive clean --dry-run 'metaEnv.TYPE == "alpha"'
-run_bob archive clean -v 'metaEnv.TYPE == "alpha"'
-
-# alpha can be downloaded, bravo must fail
-rm -rf work
-run_bob build --download=forced root-alpha
-expect_fail run_bob build --download=forced root-bravo
-
-# Update archive and rescan. Add some invalid files too.
-rm -rf work $archiveDir/*
-run_bob build --force --download=no --upload root-alpha root-bravo
-pushd $archiveDir
-mkdir -p 64/ad
-/usr/bin/tar zcf 64/ad/6386bae45ebd6788e404758a247e26e5c778-1.tgz /dev/zero
-touch 64/ad/aabbcc-too-short.tgz
-popd
-
-run_bob archive scan
-# Test that -v doesn't catch fire
-run_bob archive scan -v
-run_bob archive clean -v 'metaEnv.TYPE == "alpha"'
-
-# Test for --fail option (--fail only reports -1 if there are no files matching the pattern xx/xx/hash-1.tgz)
-expect_success run_bob archive scan --fail -v
-expect_success run_bob archive clean --fail -v 'metaEnv.TYPE == "alpha"'
-rm -rf work $archiveDir/*
-expect_fail run_bob archive scan --fail -v
-expect_fail run_bob archive clean --fail -v 'metaEnv.TYPE == "alpha"'
-
-# Test the "LIMIT" feature. Build a number or artifacts and keep only a portion
-# of them. By default the artifacts are sorted by build date and the most
-# recent is kept. Verify that the correct subset was retained.
-rm -rf "$archiveDir/"*
-run_bob build --download no --upload -q 'many-*'
-run_bob archive clean --fail -v 'meta.recipe == "many" LIMIT 3'
-test $(/usr/bin/find $archiveDir -name '*.tgz' | wc -l) -eq 3
-run_bob build --download forced --force many-07 many-06 many-05
-
-# Do the same again with ascending sorting and a different ordering key.  The
-# tricky part is that metaEnv.FUZZ is not set in all packages and such packages
-# must not be counted.
-rm -rf "$archiveDir/"*
-run_bob build --download no --force --upload -q 'many-*'
-run_bob archive clean --fail -v 'meta.recipe == "many" LIMIT 2 OrDeR By metaEnv.FUZZ ASC'
-test $(/usr/bin/find $archiveDir -name '*.tgz' | wc -l) -eq 2
-run_bob build --download forced --force many-01 many-03
-
-# Must fail if LIMIT is zero, invalid or negative
-expect_fail run_bob archive clean 'meta.recipe == "many" LIMIT 0'
-expect_fail run_bob archive clean 'meta.recipe == "many" LIMIT -3'
-expect_fail run_bob archive clean 'meta.recipe == "many" LIMIT foobar'
-
-# Build artifacts with special audit meta keys. Try to find them later.
-rm -rf "$archiveDir/"* work
-run_bob build --upload -M my-key=one root-alpha
-run_bob build --upload -M my-key=two root-bravo
-run_bob archive scan --fail
-found1=$(clean_output "$(run_bob archive find -n 'meta.recipe == "root" && meta.my-key == "one"')")
-found2=$(clean_output "$(run_bob archive find -n 'meta.recipe == "root" && meta.my-key == "two"')")
-pushd $archiveDir
-expect_exist "$found1"
-expect_exist "$found2"
-popd
-test "$found1" != "$found2"
-
-# Make sure invalid audit meta keys are rejected
-expect_fail run_bob build -M "!nv@l1d=key" root-alpha
+function run_tests () {
+  # cleanup
+  rm -rf work $archiveDir/*
+  if [[ "$1" == "local" ]]; then
+    LOCAL=true
+    BOB_ARGS="-l"
+  fi
+  # fill archive
+  run_bob build --download=no --upload root-alpha root-bravo
+  FINGERPRINT=Alice run_bob build --force --download=no --upload root-alpha root-bravo
+
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+  run_bob archive ${BOB_ARGS} scan
+
+  # first try to keep everything by using multiple expressions
+  oldNum=$(/usr/bin/find -name '*.tgz' | wc -l)
+  run_bob archive ${BOB_ARGS} clean -v 'metaEnv.TYPE == "alpha"' 'meta.package == "root-bravo"'
+  newNum=$(/usr/bin/find -name '*.tgz' | wc -l)
+  test $oldNum -eq $newNum
+
+  # Find one of the artifacts
+  found=$(clean_output "$(run_bob archive ${BOB_ARGS} find 'meta.package == "root-bravo"')")
+  if [[ ! $LOCAL ]]; then pushd $archiveDir; fi
+  expect_exist "$found"
+  if [[ ! $LOCAL ]]; then popd; fi
+
+  # selectively keep one half
+  run_bob archive ${BOB_ARGS} clean --dry-run 'metaEnv.TYPE == "alpha"'
+  run_bob archive ${BOB_ARGS} clean -v 'metaEnv.TYPE == "alpha"'
+  if [[ $LOCAL ]]; then popd; fi
+
+  # alpha can be downloaded, bravo must fail
+  rm -rf work
+  run_bob build --download=forced root-alpha
+  expect_fail run_bob build --download=forced root-bravo
+
+  # Update archive and rescan. Add some invalid files too.
+  rm -rf work $archiveDir/*
+  run_bob build --force --download=no --upload root-alpha root-bravo
+  pushd $archiveDir
+  mkdir -p 64/ad
+  /usr/bin/tar zcf 64/ad/6386bae45ebd6788e404758a247e26e5c778-1.tgz /dev/zero
+  touch 64/ad/aabbcc-too-short.tgz
+  if [[ $LOCAL ]]; then run_bob archive ${BOB_ARGS} scan; fi
+  popd
+  if [[ ! $LOCAL ]]; then run_bob archive ${BOB_ARGS} scan; fi
+
+  # Test that -v doesn't catch fire
+  run_bob archive ${BOB_ARGS} scan -v
+  run_bob archive ${BOB_ARGS} clean -v 'metaEnv.TYPE == "alpha"'
+
+  # Test for --fail option (--fail only reports -1 if there are no files matching the pattern xx/xx/hash-1.tgz)
+  if [[ ! $LOCAL ]]; then
+    expect_success run_bob archive ${BOB_ARGS} scan --fail -v
+    expect_success run_bob archive ${BOB_ARGS} clean --fail -v 'metaEnv.TYPE == "alpha"'
+    rm -rf work $archiveDir/*
+    expect_fail run_bob archive ${BOB_ARGS} scan --fail -v
+    expect_fail run_bob archive ${BOB_ARGS} clean --fail -v 'metaEnv.TYPE == "alpha"'
+  else
+    expect_fail run_bob archive ${BOB_ARGS} scan --fail -v
+    expect_fail run_bob archive ${BOB_ARGS} clean --fail -v 'metaEnv.TYPE == "alpha"'
+    pushd $archiveDir
+    run_bob archive ${BOB_ARGS} scan --fail -v
+    run_bob archive ${BOB_ARGS} clean --fail -v 'metaEnv.TYPE == "alpha"'
+    popd
+  fi
+
+  # Test the "LIMIT" feature. Build a number or artifacts and keep only a portion
+  # of them. By default the artifacts are sorted by build date and the most
+  # recent is kept. Verify that the correct subset was retained.
+  rm -rf "$archiveDir/"*
+  run_bob build --download no --upload -q 'many-*'
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+  run_bob archive ${BOB_ARGS} clean --fail -v 'meta.recipe == "many" LIMIT 3'
+  if [[ $LOCAL ]]; then popd; fi
+  test $(/usr/bin/find $archiveDir -name '*.tgz' | wc -l) -eq 3
+  run_bob build --download forced --force many-07 many-06 many-05
+
+  # Do the same again with ascending sorting and a different ordering key.  The
+  # tricky part is that metaEnv.FUZZ is not set in all packages and such packages
+  # must not be counted.
+  rm -rf "$archiveDir/"*
+  run_bob build --download no --force --upload -q 'many-*'
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+  run_bob archive ${BOB_ARGS} clean --fail -v 'meta.recipe == "many" LIMIT 2 OrDeR By metaEnv.FUZZ ASC'
+  if [[ $LOCAL ]]; then popd; fi
+  test $(/usr/bin/find $archiveDir -name '*.tgz' | wc -l) -eq 2
+  run_bob build --download forced --force many-01 many-03
+
+  # Must fail if LIMIT is zero, invalid or negative
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+  expect_fail run_bob archive ${BOB_ARGS} clean 'meta.recipe == "many" LIMIT 0'
+  expect_fail run_bob archive ${BOB_ARGS} clean 'meta.recipe == "many" LIMIT -3'
+  expect_fail run_bob archive ${BOB_ARGS} clean 'meta.recipe == "many" LIMIT foobar'
+  if [[ $LOCAL ]]; then popd; fi
+
+  # Build artifacts with special audit meta keys. Try to find them later.
+  rm -rf "$archiveDir/"* work
+  run_bob build --upload -M my-key=one root-alpha
+  run_bob build --upload -M my-key=two root-bravo
+  if [[ $LOCAL ]]; then pushd $archiveDir; fi
+    run_bob archive ${BOB_ARGS} scan --fail
+    found1=$(clean_output "$(run_bob archive ${BOB_ARGS} find -n 'meta.recipe == "root" && meta.my-key == "one"')")
+    found2=$(clean_output "$(run_bob archive ${BOB_ARGS} find -n 'meta.recipe == "root" && meta.my-key == "two"')")
+  if [[ ! $LOCAL ]]; then pushd $archiveDir; fi
+  expect_exist "$found1"
+  expect_exist "$found2"
+  popd
+  test "$found1" != "$found2"
+
+  # Make sure invalid audit meta keys are rejected
+  expect_fail run_bob build -M "!nv@l1d=key" root-alpha
+}
+# run tests with configuration with file backend
+run_tests
+# run tests with local directory
+run_tests "local"
path: test/black-box/archive/run.sh
user: login: jkloetzke
body: Missing newline at end of file.
created_at: 2025-03-03T21:09:38Z
author_association: MEMBER
diff_hunk: @@ -2982,6 +2982,7 @@ def __init__(self):
         self.__validTypes = schema.Schema({'backend': schema.Or('none', 'file', 'http', 'shell', 'azure')},
             ignore_extra_keys=True)
         baseArchive = {
+            'name' : str,
path: pym/bob/input.py
user: login: jkloetzke
body: The `name` attribute must be optional. Otherwise we'll break a lot of existing projects. That will also save you the test adaptions in large part I guess.
created_at: 2025-03-03T21:12:56Z
author_association: MEMBER
diff_hunk: @@ -486,6 +486,12 @@ def doArchive(argv, bobRoot):
 """.format(subHelp))
     parser.add_argument("-l", "--local", action='store_true',
                         help="ignore the archive configuration and work on the current directory")
+    parser.add_argument("-a", "--all", action='store_true',
+                        help="execute the command on all suitable archives in the user configuration")
+    parser.add_argument("-b", "--backends", type=str, metavar="names",
path: pym/bob/cmds/archive.py
user: login: jkloetzke
body: There is an `action='append'` mode of argparse. With that, the user could just give `-b` multiple times, one for each archive.
created_at: 2025-03-03T21:17:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
path: pym/bob/webdav.py
user: login: MarcKe
body: I thought that is what you meant with `the webdav module should throw HTTP errors`.
Are you ok with extracting the ArtifactErrors to a separate file?
created_at: 2025-03-04T07:24:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -737,182 +734,51 @@ def __exit__(self, exc_type, exc_value, traceback):
         return False
 
 
-class SimpleHttpArchive(BaseArchive):
+class HttpArchive(BaseArchive):
     def __init__(self, spec):
         super().__init__(spec)
         self.__url = urllib.parse.urlparse(spec["url"])
-        self.__connection = None
-        self.__sslVerify = spec.get("sslVerify", True)
+        self.webdav = WebDav(spec)
 
-    def __retry(self, request):
path: pym/bob/archive.py
user: login: MarcKe
body: I will change it.
created_at: 2025-03-04T07:25:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpUploadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpNotFoundError(http.client.HTTPException):
+    pass
+
+class HttpAlreadyExistsError(http.client.HTTPException):
+    pass
+
+class WebDav:
+
+    class PartialDownloader:
+        def __init__(self, webdav, path, length=512*1024):
+            self.__webdev = webdav
+            self.__path = path
+            self.__data = bytearray(self.__webdev.download(self.__path, 0, length).read())
+            self.__offset = length
+
+        def get(self):
+            return self.__data
+
+        def more(self, length=512*1024):
+            new_data = self.__webdev.download(self.__path, self.__offset, length)
+            self.__offset += length
+            self.__data.extend(new_data.read())
+            return self.__data
+
+    def __init__(self, spec):
+        self.__url = urlparse(spec["url"])
+        self.__connection = None
+        self.__sslVerify = spec.get("sslVerify", True)
+
+    def __retry(self, request):
+        retry = True
+        while True:
+            try:
+                return (True, request())
+            except (http.client.HTTPException, OSError) as e:
+                self._resetConnection()
+                if not retry: return (False, e)
+                retry = False
+
+    def getPartialDownloader(self, path):
+        return self.PartialDownloader(self, path)
+
+    def _getConnection(self):
+        if self.__connection is not None:
+            return self.__connection
+
+        url = self.__url
+        if url.scheme == 'http':
+            connection = http.client.HTTPConnection(url.hostname, url.port)
+        elif url.scheme == 'https':
+            ctx = None if self.__sslVerify else sslNoVerifyContext()
+            connection = http.client.HTTPSConnection(url.hostname, url.port,
+                                                     context=ctx)
+        else:
+            raise BuildError("Unsupported URL scheme: '{}'".format(url.scheme))
+
+        # simple check if webdav is supported
+        connection.request("OPTIONS", '/'.join([self.__url.path, '']), headers=self._getHeaders())
+
+        response = connection.getresponse()
+        response.read()
+        dav_supported = response.getheader("DAV") is not None
+        if not dav_supported:
+            raise BuildError("WEBDAV not supported by server")
path: pym/bob/webdav.py
user: login: MarcKe
body: You are right, it might fuck up bob downloads.
created_at: 2025-03-04T07:25:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -737,182 +734,51 @@ def __exit__(self, exc_type, exc_value, traceback):
         return False
 
 
-class SimpleHttpArchive(BaseArchive):
+class HttpArchive(BaseArchive):
     def __init__(self, spec):
         super().__init__(spec)
         self.__url = urllib.parse.urlparse(spec["url"])
-        self.__connection = None
-        self.__sslVerify = spec.get("sslVerify", True)
+        self.webdav = WebDav(spec)
 
-    def __retry(self, request):
-        retry = True
-        while True:
-            try:
-                return (True, request())
-            except (http.client.HTTPException, OSError) as e:
-                self._resetConnection()
-                if not retry: return (False, e)
-                retry = False
+    def _resetConnection(self):
+        self.webdav._resetConnection()
 
-    def _makeUrl(self, buildId, suffix):
+    def _makePath(self, buildId, suffix):
         packageResultId = buildIdToName(buildId)
         return "/".join([self.__url.path, packageResultId[0:2], packageResultId[2:4],
             packageResultId[4:] + suffix])
 
-    def _remoteName(self, buildId, suffix):
-        url = self.__url
-        return urllib.parse.urlunparse((url.scheme, url.netloc, self._makeUrl(buildId, suffix), '', '', ''))
-
-    def _getConnection(self):
-        if self.__connection is not None:
-            return self.__connection
+    def _create_dirs(self, buildId):
+        packageResultId = buildIdToName(buildId)
+        path = "/".join([self.__url.path, packageResultId[0:2], ''])
+        self.webdav.mkdir(path)
+        path = "/".join([self.__url.path, packageResultId[0:2], packageResultId[2:4], ''])
+        self.webdav.mkdir(path)
 
+    def _remoteName(self, buildId, suffix):
         url = self.__url
-        if url.scheme == 'http':
-            connection = http.client.HTTPConnection(url.hostname, url.port)
-        elif url.scheme == 'https':
-            ctx = None if self.__sslVerify else sslNoVerifyContext()
-            connection = http.client.HTTPSConnection(url.hostname, url.port,
-                                                     context=ctx)
-        else:
-            raise BuildError("Unsupported URL scheme: '{}'".format(url.schema))
-
-        self.__connection = connection
-        return connection
-
-    def _resetConnection(self):
-        if self.__connection is not None:
-            self.__connection.close()
-            self.__connection = None
-
-    def _getHeaders(self):
-        headers = { 'User-Agent' : 'BobBuildTool/{}'.format(BOB_VERSION) }
-        if self.__url.username is not None:
-            username = urllib.parse.unquote(self.__url.username)
-            passwd = urllib.parse.unquote(self.__url.password)
-            userPass = username + ":" + passwd
-            headers['Authorization'] = 'Basic ' + base64.b64encode(
-                userPass.encode("utf-8")).decode("ascii")
-        return headers
+        return urllib.parse.urlunparse((url.scheme, url.netloc, self._makePath(buildId, suffix), '', '', ''))
 
     def _openDownloadFile(self, buildId, suffix):
-        (ok, result) = self.__retry(lambda: self.__openDownloadFile(buildId, suffix))
-        if ok:
-            return result
-        else:
-            raise ArtifactDownloadError(str(result))
-
-    def __openDownloadFile(self, buildId, suffix):
-        connection = self._getConnection()
-        url = self._makeUrl(buildId, suffix)
-        connection.request("GET", url, headers=self._getHeaders())
-        response = connection.getresponse()
-        if response.status == 200:
-            return SimpleHttpDownloader(self, response)
-        else:
-            response.read()
-            if response.status == 404:
-                raise ArtifactNotFoundError()
-            else:
-                raise ArtifactDownloadError("{} {}".format(response.status,
-                                                           response.reason))
+        url = self._makePath(buildId, suffix)
+        return HttpDownloader(self, self.webdav.download(url))
 
     def _openUploadFile(self, buildId, suffix, overwrite):
-        (ok, result) = self.__retry(lambda: self.__openUploadFile(buildId, suffix, overwrite))
-        if ok:
-            return result
-        else:
-            raise ArtifactUploadError(str(result))
-
-    def __openUploadFile(self, buildId, suffix, overwrite):
-        connection = self._getConnection()
-        url = self._makeUrl(buildId, suffix)
-
+        url = self._makePath(buildId, suffix)
         # check if already there
         if not overwrite:
-            connection.request("HEAD", url, headers=self._getHeaders())
-            response = connection.getresponse()
-            response.read()
-            if response.status == 200:
+            if self.webdav.check(url):
                 raise ArtifactExistsError()
-            elif response.status != 404:
-                raise ArtifactUploadError("HEAD {} {}".format(response.status, response.reason))
 
-        # create temporary file
-        return SimpleHttpUploader(self, url, overwrite)
+        # create dirs based on buildId to make 100% sure they are available
+        self._create_dirs(buildId)
+        return HttpUploader(self, url, overwrite)
 
     def _putUploadFile(self, url, tmp, overwrite):
-        (ok, result) = self.__retry(lambda: self.__putUploadFile(url, tmp, overwrite))
-        if ok:
-            return result
-        else:
-            raise ArtifactUploadError(str(result))
-
-    def __putUploadFile(self, url, tmp, overwrite):
-        # Quoting RFC 4918:
-        #
-        #   A PUT that would result in the creation of a resource without an
-        #   appropriately scoped parent collection MUST fail with a 409
-        #   (Conflict).
-        #
-        # We don't want to waste bandwith by uploading big artifacts into a
-        # missing directory. Thus make sure the directory always exists.
path: pym/bob/archive.py
user: login: MarcKe
body: Yes, will do.
created_at: 2025-03-04T07:26:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -556,6 +587,119 @@ async def downloadLocalFingerprint(self, step, key, executor=None):
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
                 raise BuildError("Download of fingerprint interrupted.")
 
+    def deleteFile(self, filepath):
+        # Set default signal handler so that KeyboardInterrupt is raised.
+        # Needed to gracefully handle ctrl+c.
+        signal.signal(signal.SIGINT, signal.default_int_handler)
path: pym/bob/archive.py
user: login: MarcKe
body: Yes, that is what I thought.
created_at: 2025-03-04T07:27:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -484,40 +484,52 @@ def doArchive(argv, bobRoot):
 
   bob archive {}
 """.format(subHelp))
+    parser.add_argument("-l", "--local", action='store_true',
+                        help="ignore the archive configuration and work on the current directory")
     parser.add_argument('subcommand', help="Subcommand")
     parser.add_argument('args', nargs=argparse.REMAINDER,
                         help="Arguments for subcommand")
 
     args = parser.parse_args(argv)
 
     # get archiver
-    recipes = RecipeSet()
-    recipes.parse()
-    archivespec = recipes.archiveSpec()
     archivers = []
-    if isinstance(archivespec, list):
-        if len(archivespec) == 0:
-            raise BobError("No archiver defined")
-        elif len(archivespec) == 1:
-            archiver = getSingleArchiver(recipes, archivespec[0])
-            if archiver.archiveCmdSupported():
-                archivers.append(archiver)
-            else:
-                raise BobError("Archiver does not support the archive command")
+    if args.local:
+        # provide local directory as backend
+        archivespec = [{"backend": "file", "path": "{}".format(os.getcwd())}]
+        archiver = getSingleArchiver(None, archivespec[0])
+        if archiver.archiveCmdSupported():
path: pym/bob/cmds/archive.py
user: login: MarcKe
body: You are right, it is just a copy from the other part.
created_at: 2025-03-04T07:34:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpUploadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpNotFoundError(http.client.HTTPException):
+    pass
+
+class HttpAlreadyExistsError(http.client.HTTPException):
+    pass
+
+class WebDav:
+
+    class PartialDownloader:
path: pym/bob/webdav.py
user: login: MarcKe
body: Yes, they are in _getAudit of HttpArchive
created_at: 2025-03-04T07:37:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -777,6 +950,41 @@ def _openUploadFile(self, buildId, suffix, overwrite):
     def _putUploadFile(self, url, tmp, overwrite):
         return self.webdav.upload(url, tmp, overwrite)
 
+    def _listDir(self, path):
+        path_info = self.webdav.list(path)
+        entries = []
+        for info in path_info:
+            if info["path"]:
+                entries.append(info["path"].removeprefix(path + "/"))
+        return entries
+
+    def _delete(self, filename):
+        self.webdav.delete(filename)
+
+    def _stat(self, filename):
+        stats = self.webdav.stat(filename)
+        if not stats['cdate'] or not stats['mdate'] or not stats['etag'] or not stats['len']:
+            print("error")
+        from datetime import datetime
+        from email.utils import parsedate_to_datetime
+        import binascii
+        return struct.pack('=ddLL', datetime.fromisoformat(stats['cdate']).timestamp(), parsedate_to_datetime(stats['mdate']).timestamp(), binascii.crc32(stats['etag'].encode('utf8')), stats['len'])
path: pym/bob/archive.py
user: login: MarcKe
body: I do agree that crc32 might be too weak.
I have looked at three different servers with webdav support: nginx, apache2 and our internal artifactory. I was not 100% sure what to use.
Nginx for example does not provide any etag. Apache2 has it. Artifactory provides a weak etag which is simply the last modified date. I considered the data from nginx to be quite useless.
What would you consider to be a good solution? Are lastmodified and length enough?
nginx: 
```
<D:prop>
<D:displayname>418b5ea011971ec9494d7c70705cdf9a81c4-1.tgz</D:displayname>
<D:getcontentlength>2669</D:getcontentlength>
<D:getlastmodified>Tue, 04 Mar 2025 08:19:39 GMT</D:getlastmodified>
<D:resourcetype></D:resourcetype>
<D:lockdiscovery/>
<D:supportedlock>
</D:supportedlock>
</D:prop>
```

apache2:
```
<D:prop>
<lp1:resourcetype/>
<lp1:creationdate>2025-03-04T08:01:13Z</lp1:creationdate>
<lp1:getcontentlength>2685</lp1:getcontentlength>
<lp1:getlastmodified>Tue, 04 Mar 2025 08:01:13 GMT</lp1:getlastmodified>
<lp1:getetag>"a7d-62f7fad3ac545"</lp1:getetag>
<lp2:executable>F</lp2:executable>
<D:supportedlock>
<D:lockentry>
<D:lockscope><D:exclusive/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
<D:lockentry>
<D:lockscope><D:shared/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
</D:supportedlock>
<D:lockdiscovery/>
<D:getcontenttype>application/x-gzip</D:getcontenttype>
</D:prop>
```

artifactory:
```
<D:prop><lp1:getcontentlength>6260</lp1:getcontentlength>
<lp1:getcontenttype>application/x-gzip</lp1:getcontenttype>
<lp1:resourcetype/><lp1:getlastmodified>Thu, 06 Feb 2025 12:47:05 GMT</lp1:getlastmodified>
<lp1:getetag>W/"6260-Thu, 06 Feb 2025 12:47:05 GMT"</lp1:getetag>
<lp1:creationdate>2024-07-11T14:07:47Z</lp1:creationdate>
<D:displayname><![CDATA[logviewer.tgz]]></D:displayname>
<D:source></D:source>
</D:prop>
```

created_at: 2025-03-04T08:28:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,15 +22,17 @@
 class ArchiveScanner:
     CUR_VERSION = 2
 
-    def __init__(self):
+    def __init__(self, archiver):
         self.__dirSchema = re.compile(r'[0-9a-zA-Z]{2}')
         self.__archiveSchema = re.compile(r'[0-9a-zA-Z]{36,}-1.tgz')
         self.__db = None
         self.__cleanup = False
+        self.__archiver = archiver
+        self.__dbName = ".bob-archive_{}.sqlite3".format(self.__archiver.getArchiveHash())
path: pym/bob/cmds/archive.py
user: login: MarcKe
body: I could change it to that.
Would you object to simply changing it to a sha256? 
created_at: 2025-03-04T08:38:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
path: pym/bob/webdav.py
user: login: jkloetzke
body: My original idea was to just let the `http.client.HTTPException` bubble up and return the other codes. But your approach is also fine. But in this case the exceptions do not need to derive from `http.client.HTTPException`.
created_at: 2025-03-06T19:33:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,276 @@
+from . import BOB_VERSION
+from .errors import BuildError
+from .utils import sslNoVerifyContext
+import base64
+import http.client
+import os
+from urllib.parse import unquote, urlsplit, urlparse
+from xml.etree.ElementTree import fromstring
+
+class HttpDownloadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpUploadError(http.client.HTTPException):
+    def __init__(self, reason):
+        self.reason = reason
+
+class HttpNotFoundError(http.client.HTTPException):
+    pass
+
+class HttpAlreadyExistsError(http.client.HTTPException):
+    pass
+
+class WebDav:
+
+    class PartialDownloader:
path: pym/bob/webdav.py
user: login: jkloetzke
body: Oh, I see.
created_at: 2025-03-06T19:35:25Z
author_association: MEMBER
diff_hunk: @@ -777,6 +950,41 @@ def _openUploadFile(self, buildId, suffix, overwrite):
     def _putUploadFile(self, url, tmp, overwrite):
         return self.webdav.upload(url, tmp, overwrite)
 
+    def _listDir(self, path):
+        path_info = self.webdav.list(path)
+        entries = []
+        for info in path_info:
+            if info["path"]:
+                entries.append(info["path"].removeprefix(path + "/"))
+        return entries
+
+    def _delete(self, filename):
+        self.webdav.delete(filename)
+
+    def _stat(self, filename):
+        stats = self.webdav.stat(filename)
+        if not stats['cdate'] or not stats['mdate'] or not stats['etag'] or not stats['len']:
+            print("error")
+        from datetime import datetime
+        from email.utils import parsedate_to_datetime
+        import binascii
+        return struct.pack('=ddLL', datetime.fromisoformat(stats['cdate']).timestamp(), parsedate_to_datetime(stats['mdate']).timestamp(), binascii.crc32(stats['etag'].encode('utf8')), stats['len'])
path: pym/bob/archive.py
user: login: jkloetzke
body: :face_with_spiral_eyes: As always, every server does it differently. :facepalm:  I would suggest to use the "getetag" if there is one that is not weak. Otherwise, the combination of length and last modification date looks like the best approach.
created_at: 2025-03-06T19:44:01Z
author_association: MEMBER
diff_hunk: @@ -23,15 +22,17 @@
 class ArchiveScanner:
     CUR_VERSION = 2
 
-    def __init__(self):
+    def __init__(self, archiver):
         self.__dirSchema = re.compile(r'[0-9a-zA-Z]{2}')
         self.__archiveSchema = re.compile(r'[0-9a-zA-Z]{36,}-1.tgz')
         self.__db = None
         self.__cleanup = False
+        self.__archiver = archiver
+        self.__dbName = ".bob-archive_{}.sqlite3".format(self.__archiver.getArchiveHash())
path: pym/bob/cmds/archive.py
user: login: jkloetzke
body: I would prefer to have only one database file. In the end, the database is just a cache so it's not an issue if we change the schema. This is already anticipated with the version check...
created_at: 2025-03-06T19:47:03Z
author_association: MEMBER
diff_hunk: @@ -559,6 +582,119 @@ async def downloadLocalFingerprint(self, step, key, executor=None):
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
                 raise BuildError("Download of fingerprint interrupted.")
 
+    def deleteFile(self, filepath):
+        # Set default signal handler so that KeyboardInterrupt is raised.
path: pym/bob/archive.py
user: login: jkloetzke
body: You mean the signal stuff? No, that is not necessary here.
created_at: 2025-03-06T19:49:36Z
author_association: MEMBER
diff_hunk: @@ -559,6 +582,119 @@ async def downloadLocalFingerprint(self, step, key, executor=None):
             except (concurrent.futures.CancelledError, concurrent.futures.process.BrokenProcessPool):
                 raise BuildError("Download of fingerprint interrupted.")
 
+    def deleteFile(self, filepath):
+        # Set default signal handler so that KeyboardInterrupt is raised.
path: pym/bob/archive.py
user: login: MarcKe
body: Sorry, that comment was in pending for a long time. I was not aware that I needed to press "submit".  I already saw your answer :P
created_at: 2025-03-07T08:53:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -777,6 +950,41 @@ def _openUploadFile(self, buildId, suffix, overwrite):
     def _putUploadFile(self, url, tmp, overwrite):
         return self.webdav.upload(url, tmp, overwrite)
 
+    def _listDir(self, path):
+        path_info = self.webdav.list(path)
+        entries = []
+        for info in path_info:
+            if info["path"]:
+                entries.append(info["path"].removeprefix(path + "/"))
+        return entries
+
+    def _delete(self, filename):
+        self.webdav.delete(filename)
+
+    def _stat(self, filename):
+        stats = self.webdav.stat(filename)
+        if not stats['cdate'] or not stats['mdate'] or not stats['etag'] or not stats['len']:
+            print("error")
+        from datetime import datetime
+        from email.utils import parsedate_to_datetime
+        import binascii
+        return struct.pack('=ddLL', datetime.fromisoformat(stats['cdate']).timestamp(), parsedate_to_datetime(stats['mdate']).timestamp(), binascii.crc32(stats['etag'].encode('utf8')), stats['len'])
path: pym/bob/archive.py
user: login: MarcKe
body: Good, I will do it like this.
created_at: 2025-03-07T08:54:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,15 +22,17 @@
 class ArchiveScanner:
     CUR_VERSION = 2
 
-    def __init__(self):
+    def __init__(self, archiver):
         self.__dirSchema = re.compile(r'[0-9a-zA-Z]{2}')
         self.__archiveSchema = re.compile(r'[0-9a-zA-Z]{36,}-1.tgz')
         self.__db = None
         self.__cleanup = False
+        self.__archiver = archiver
+        self.__dbName = ".bob-archive_{}.sqlite3".format(self.__archiver.getArchiveHash())
path: pym/bob/cmds/archive.py
user: login: MarcKe
body: Ok, I will do that.
created_at: 2025-03-07T08:54:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,15 +22,19 @@
 class ArchiveScanner:
     CUR_VERSION = 2
path: pym/bob/cmds/archive.py
user: login: jkloetzke
body: We must bump the version because the schema changed.
created_at: 2025-03-15T20:29:55Z
author_association: MEMBER
diff_hunk: @@ -500,6 +500,12 @@ def doArchive(argv, bobRoot):
 """.format(subHelp))
     parser.add_argument("-l", "--local", action='store_true',
                         help="ignore the archive configuration and work on the current directory")
+    parser.add_argument("-a", "--all", action='store_true',
path: pym/bob/cmds/archive.py
user: login: jkloetzke
body: I think you can use a `add_mutually_exclusive_group` for `-l`, `-a` and `-b`. That way, only one of them can be given and we do not have to define a precedence.
created_at: 2025-03-15T20:44:30Z
author_association: MEMBER

PR comments:
user: login: MarcKe
created_at: 2025-02-21T13:53:45Z
body: I want to get some first feedback from your side.
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-02-21T13:58:08Z
body: ## [Codecov](https://app.codecov.io/gh/BobBuildTool/bob/pull/617?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool) Report
Attention: Patch coverage is `82.82828%` with `85 lines` in your changes missing coverage. Please review.
> Project coverage is 88.79%. Comparing base [(`ca72c0f`)](https://app.codecov.io/gh/BobBuildTool/bob/commit/ca72c0fd370364ec972484f681e0d501373daf08?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool) to head [(`f1ba645`)](https://app.codecov.io/gh/BobBuildTool/bob/commit/f1ba6456dc844ef874c3e3461c480bc2558bbcdd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool).
> Report is 8 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/BobBuildTool/bob/pull/617?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool) | Patch % | Lines |
|---|---|---|
| [pym/bob/archive.py](https://app.codecov.io/gh/BobBuildTool/bob/pull/617?src=pr&el=tree&filepath=pym%2Fbob%2Farchive.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool#diff-cHltL2JvYi9hcmNoaXZlLnB5) | 63.05% | [75 Missing :warning: ](https://app.codecov.io/gh/BobBuildTool/bob/pull/617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool) |
| [pym/bob/cmds/archive.py](https://app.codecov.io/gh/BobBuildTool/bob/pull/617?src=pr&el=tree&filepath=pym%2Fbob%2Fcmds%2Farchive.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool#diff-cHltL2JvYi9jbWRzL2FyY2hpdmUucHk=) | 93.80% | [7 Missing :warning: ](https://app.codecov.io/gh/BobBuildTool/bob/pull/617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool) |
| [pym/bob/webdav.py](https://app.codecov.io/gh/BobBuildTool/bob/pull/617?src=pr&el=tree&filepath=pym%2Fbob%2Fwebdav.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool#diff-cHltL2JvYi93ZWJkYXYucHk=) | 98.32% | [3 Missing :warning: ](https://app.codecov.io/gh/BobBuildTool/bob/pull/617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master     #617      +/-   ##
==========================================
- Coverage   88.94%   88.79%   -0.16%     
==========================================
  Files          48       49       +1     
  Lines       15616    15890     +274     
==========================================
+ Hits        13890    14109     +219     
- Misses       1726     1781      +55     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/BobBuildTool/bob/pull/617?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BobBuildTool).

<details><summary>🚀 New features to boost your workflow: </summary>

- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: jkloetzke
created_at: 2025-02-22T11:27:24Z
body: Sure. First of all, this feature is very much appreciated. It's been on the TODO list for a long time. :+1: 

Before reviewing the code in detail, a couple of things that I noticed:

* The archive command should still support working locally. That is, it should be possible to execute the command for a local directory. This is useful to run `bob archive` as a cron-job on the storage server.
* I guess it should be possible to work on selected archive backends if there is more than one. I would even argue that it's better to require an explicit selection of the user on the desired servers and/or local directories. By doing the operation on all backends, it might lead to undesired data loss because the user was not aware it is affecting all at the same time.
* That probably requires something like a `name` attribute for a backend. This will be handy also in the DOWNLOAD/UPLOAD messages. It was always confusion if multiple backends are configured, which one exactly had a hit or miss when building. If no name is given, you simply cannot run bob-archive on it.
* Should we add a `managed` flag? You might not have the rights to enumerate or delete artifacts even though you can upload.

Code wise, I think the circular dependency between the webdav and archive modules needs to be properly resolved. IMHO, the webdav module should throw HTTP errors. That will require some adaptation in the archive module but this hack to resolve the circular dependency is, well, just a hack...
author_association: MEMBER
user: login: MarcKe
created_at: 2025-02-24T07:02:19Z
body: > Sure. First of all, this feature is very much appreciated. It's been on the TODO list for a long time. 👍
> 
> Before reviewing the code in detail, a couple of things that I noticed:
> 
>     * The archive command should still support working locally. That is, it should be possible to execute the command for a local directory. This is useful to run `bob archive` as a cron-job on the storage server.
> 

Yes, sounds reasonable, with an option like `-l/--local`? Would that be ok?

>     * I guess it should be possible to work on selected archive backends if there is more than one. I would even argue that it's better to require an explicit selection of the user on the desired servers and/or local directories. By doing the operation on all backends, it might lead to undesired data loss because the user was not aware it is affecting all at the same time.
> 

Sounds good to me. There could be an argument like `-a/--all` to just do it like I have implemented it now.

>     * That probably requires something like a `name` attribute for a backend. This will be handy also in the DOWNLOAD/UPLOAD messages. It was always confusion if multiple backends are configured, which one exactly had a hit or miss when building. If no name is given, you simply cannot run bob-archive on it.
> 

I will have a look at this together with the explicit selection.

>     * Should we add a `managed` flag? You might not have the rights to enumerate or delete artifacts even though you can upload.
> 

I think this is also a good idea.

> 
> Code wise, I think the circular dependency between the webdav and archive modules needs to be properly resolved. IMHO, the webdav module should throw HTTP errors. That will require some adaptation in the archive module but this hack to resolve the circular dependency is, well, just a hack...

Good idea. I will have a look at this, too.

author_association: CONTRIBUTOR
user: login: jkloetzke
created_at: 2025-02-24T20:13:17Z
body: > Yes, sounds reasonable, with an option like `-l/--local`? Would that be ok?

Yes, sounds good.
author_association: MEMBER
user: login: MarcKe
created_at: 2025-03-04T08:30:48Z
body: > This turned out to be quite a lot of comments. I certainly did not catch everything yet but I hope you bear with me...

No, thank you, this is great.
author_association: CONTRIBUTOR
user: login: MarcKe
created_at: 2025-03-11T15:46:47Z
body: I think I have managed to go through all your remarks. 
author_association: CONTRIBUTOR
user: login: jkloetzke
created_at: 2025-03-12T21:23:11Z
body: Thanks. I'll have a look on the weekend (hopefully).
author_association: MEMBER
user: login: MarcKe
created_at: 2025-03-19T15:04:34Z
body: > That looks quite good already. Would it be possible to add some testing for the Webdav parts? Maybe (re)use the HTTP mock server from the unit tests to have some minimally responsive Webdav server? Otherwise this is basically all untested and I fear it will break unnoticed whenever we touch the archive stuff...

Are those tests enough?
To compare the changes I used this (but there are master commits showing):
https://github.com/BobBuildTool/bob/compare/ea743015fb0edea4bfce231ba21578158339fa16...f1ba6456dc844ef874c3e3461c480bc2558bbcdd
Is there a better way to do that on github?

edit: ahh, windows...

author_association: CONTRIBUTOR
user: login: jkloetzke
created_at: 2025-03-20T19:31:38Z
body: Thanks a lot!
author_association: MEMBER