Repository: kopia/kopia
Pull request #2364 (2022-09-03 22:18:16)

feat(repository): added Set and Map backed by custom on-disk hashtable
This is just a new data structure, not used yet.

# Background:

We used to use `sync.Map` to keep track of various content IDs visited in various places. This is quite slow and wastes a lot of memory due to fragmentation.

For example on 2021 MacBook Pro M1 with 16 GB RAM, sync.Map can only keep ~55M keys, each of 32 bytes (SHA256 hashes) due to excessive memory fragmentation (without fragmentation it should be <2GB)

This custom implementation uses explicitly managed contiguous buffers, is faster than sync.Map and only uses 8 bytes per key in RAM and offloads all payload to memory-mapped files which are allocated in segments.

On a Core i5 machine with 32GB RAM, this supports ~100M Put() operations per minute (32-byte keys).


codecov[bot]: # [Codecov](https://codecov.io/gh/kopia/kopia/pull/2364?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) Report
> Merging [#2364](https://codecov.io/gh/kopia/kopia/pull/2364?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) (55b6d71) into [master](https://codecov.io/gh/kopia/kopia/commit/a3b1f69718408be607823368098f5581cbe76e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) (a3b1f69) will **decrease** coverage by `0.05%`.
> The diff coverage is `64.66%`.

```diff
@@            Coverage Diff             @@
##           master    #2364      +/-   ##
==========================================
- Coverage   74.38%   74.33%   -0.06%     
==========================================
  Files         436      439       +3     
  Lines       34313    34630     +317     
==========================================
+ Hits        25524    25742     +218     
- Misses       6932     7020      +88     
- Partials     1857     1868      +11     
```


| [Impacted Files](https://codecov.io/gh/kopia/kopia/pull/2364?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) | Coverage Δ | |
|---|---|---|
| [internal/bigmap/bigmapbench/main.go](https://codecov.io/gh/kopia/kopia/pull/2364/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvYmlnbWFwL2JpZ21hcGJlbmNoL21haW4uZ28=) | `0.00% <0.00%> (ø)` | |
| [internal/bigmap/bigmap\_set.go](https://codecov.io/gh/kopia/kopia/pull/2364/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvYmlnbWFwL2JpZ21hcF9zZXQuZ28=) | `64.28% <64.28%> (ø)` | |
| [internal/bigmap/bigmap.go](https://codecov.io/gh/kopia/kopia/pull/2364/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvYmlnbWFwL2JpZ21hcC5nbw==) | `80.99% <80.99%> (ø)` | |
| [snapshot/snapshotfs/upload.go](https://codecov.io/gh/kopia/kopia/pull/2364/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-c25hcHNob3Qvc25hcHNob3Rmcy91cGxvYWQuZ28=) | `83.18% <0.00%> (-0.23%)` | :arrow_down: |
| [internal/epoch/epoch\_manager.go](https://codecov.io/gh/kopia/kopia/pull/2364/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvZXBvY2gvZXBvY2hfbWFuYWdlci5nbw==) | `85.29% <0.00%> (+0.48%)` | :arrow_up: |
| [repo/open.go](https://codecov.io/gh/kopia/kopia/pull/2364/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9vcGVuLmdv) | `75.64% <0.00%> (+1.10%)` | :arrow_up: |
| [internal/server/source\_manager.go](https://codecov.io/gh/kopia/kopia/pull/2364/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvc2VydmVyL3NvdXJjZV9tYW5hZ2VyLmdv) | `82.82% <0.00%> (+1.53%)` | :arrow_up: |
| [internal/dirutil/mksubdirall.go](https://codecov.io/gh/kopia/kopia/pull/2364/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvZGlydXRpbC9ta3N1YmRpcmFsbC5nbw==) | `100.00% <0.00%> (+7.89%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia)


Patch:
diff --git a/.codecov.yml b/.codecov.yml
index ac94fc5e..fa4aeb64 100644
--- a/.codecov.yml
+++ b/.codecov.yml
@@ -25,4 +25,5 @@ ignore:
  - internal/repotesting/
  - internal/fshasher/
  - internal/blobtesting/
+ - internal/bigmap/bigmapbench/
  
\ No newline at end of file
diff --git a/Makefile b/Makefile
index 3ff6b293..8fc40ca1 100644
--- a/Makefile
+++ b/Makefile
@@ -344,6 +344,10 @@ endif
 htmlui-e2e-test:
 	HTMLUI_E2E_TEST=1 go test -timeout 600s github.com/kopia/kopia/tests/htmlui_e2e_test -v $(TEST_FLAGS)
 
+htmlui-e2e-test-local-htmlui-changes:
+	(cd ../htmlui && npm run build)
+	HTMLUI_E2E_TEST=1 HTMLUI_BUILD_DIR=$(CURDIR)/../htmlui/build go test -timeout 600s github.com/kopia/kopia/tests/htmlui_e2e_test -v $(TEST_FLAGS)
+
 godoc:
 	godoc -http=:33333
 
diff --git a/app/package-lock.json b/app/package-lock.json
index 53b3cd93..cf5ce660 100644
--- a/app/package-lock.json
+++ b/app/package-lock.json
@@ -20,7 +20,7 @@
       "devDependencies": {
         "asar": "^3.2.0",
         "concurrently": "^7.3.0",
-        "dotenv": "^16.0.1",
+        "dotenv": "^16.0.2",
         "electron": "^20.0.1",
         "electron-builder": "^23.3.3",
         "electron-notarize": "^1.2.1"
@@ -1413,9 +1413,9 @@
       }
     },
     "node_modules/dotenv": {
-      "version": "16.0.1",
-      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.0.1.tgz",
-      "integrity": "sha512-1K6hR6wtk2FviQ4kEiSjFiH5rpzEVi8WW0x96aztHVMhEspNpc4DVOUTEHtEva5VThQ8IaBX1Pe4gSzpVVUsKQ==",
+      "version": "16.0.2",
+      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.0.2.tgz",
+      "integrity": "sha512-JvpYKUmzQhYoIFgK2MOnF3bciIZoItIIoryihy0rIA+H4Jy0FmgyKYAHCTN98P5ybGSJcIFbh6QKeJdtZd1qhA==",
       "dev": true,
       "engines": {
         "node": ">=12"
@@ -4708,9 +4708,9 @@
       }
     },
     "dotenv": {
-      "version": "16.0.1",
-      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.0.1.tgz",
-      "integrity": "sha512-1K6hR6wtk2FviQ4kEiSjFiH5rpzEVi8WW0x96aztHVMhEspNpc4DVOUTEHtEva5VThQ8IaBX1Pe4gSzpVVUsKQ==",
+      "version": "16.0.2",
+      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.0.2.tgz",
+      "integrity": "sha512-JvpYKUmzQhYoIFgK2MOnF3bciIZoItIIoryihy0rIA+H4Jy0FmgyKYAHCTN98P5ybGSJcIFbh6QKeJdtZd1qhA==",
       "dev": true
     },
     "dotenv-expand": {
diff --git a/app/package.json b/app/package.json
index 4bfb8d73..d524ac9f 100644
--- a/app/package.json
+++ b/app/package.json
@@ -112,7 +112,7 @@
   "devDependencies": {
     "asar": "^3.2.0",
     "concurrently": "^7.3.0",
-    "dotenv": "^16.0.1",
+    "dotenv": "^16.0.2",
     "electron": "^20.0.1",
     "electron-builder": "^23.3.3",
     "electron-notarize": "^1.2.1"
diff --git a/cli/command_restore.go b/cli/command_restore.go
index 38a212b3..d402e7f5 100644
--- a/cli/command_restore.go
+++ b/cli/command_restore.go
@@ -8,15 +8,22 @@
 	"math"
 	"os"
 	"path/filepath"
+	"regexp"
+	"strconv"
 	"strings"
+	"time"
 
 	"github.com/pkg/errors"
 
 	"github.com/kopia/kopia/fs"
 	"github.com/kopia/kopia/fs/localfs"
+	"github.com/kopia/kopia/internal/clock"
 	"github.com/kopia/kopia/internal/timetrack"
 	"github.com/kopia/kopia/internal/units"
 	"github.com/kopia/kopia/repo"
+	"github.com/kopia/kopia/repo/content/index"
+	"github.com/kopia/kopia/repo/object"
+	"github.com/kopia/kopia/snapshot"
 	"github.com/kopia/kopia/snapshot/restore"
 	"github.com/kopia/kopia/snapshot/snapshotfs"
 )
@@ -114,6 +121,7 @@ type commandRestore struct {
 	restoreIgnoreErrors           bool
 	restoreShallowAtDepth         int32
 	minSizeForPlaceholder         int32
+	snapshotTime                  string
 
 	restores []restoreSourceTarget
 }
@@ -139,6 +147,7 @@ func (c *commandRestore) setup(svc appServices, parent commandParent) {
 	cmd.Flag("skip-existing", "Skip files and symlinks that exist in the output").BoolVar(&c.restoreIncremental)
 	cmd.Flag("shallow", "Shallow restore the directory hierarchy starting at this level (default is to deep restore the entire hierarchy.)").Int32Var(&c.restoreShallowAtDepth)
 	cmd.Flag("shallow-minsize", "When doing a shallow restore, write actual files instead of placeholders smaller than this size.").Int32Var(&c.minSizeForPlaceholder)
+	cmd.Flag("snapshot-time", "When using a path as the source, use the latest snapshot available before this date. Default is latest").StringVar(&c.snapshotTime)
 	cmd.Action(svc.repositoryReaderAction(c.run))
 }
 
@@ -154,7 +163,7 @@ func (c *commandRestore) setup(svc appServices, parent commandParent) {
 // constructTargetPairs builds the sourceIdPathPairs array for this
 // command for the two forms of command: expansion of one or more
 // placeholders or restoring of a single source to a single destination.
-func (c *commandRestore) constructTargetPairs() error {
+func (c *commandRestore) constructTargetPairs(rep repo.Repository) error {
 	targetPairs := make([]restoreSourceTarget, 0, len(c.restoreTargetPaths))
 
 	for _, p := range c.restoreTargetPaths {
@@ -174,6 +183,33 @@ func (c *commandRestore) constructTargetPairs() error {
 	}
 
 	switch tplen, restpslen := len(targetPairs), len(c.restoreTargetPaths); {
+	case tplen == 0 && restpslen == 1:
+		// This means that none of the restoreTargetPaths are placeholders and we
+		// have 1 arg: a source path that should also be used as a destination.
+		source := c.restoreTargetPaths[0]
+
+		si, err := snapshot.ParseSourceInfo(source, rep.ClientOptions().Hostname, rep.ClientOptions().Username)
+		if err != nil {
+			return errors.Errorf("invalid path to be used as source: '%s': %s", source, err)
+		}
+
+		if si.Path == "" {
+			return errors.New("the source must contain a path element")
+		}
+
+		if si.Host != rep.ClientOptions().Hostname || si.UserName != rep.ClientOptions().Username {
+			return errors.New("the source must be a path in with the same username/hostname to be used as a target too")
+		}
+
+		c.restores = []restoreSourceTarget{
+			{
+				source:        source,
+				target:        si.Path,
+				isplaceholder: false,
+			},
+		}
+
+		return nil
 	case tplen == 0 && restpslen == 2:
 		// This means that none of the restoreTargetPaths are placeholders and we
 		// we have two args: a sourceID and a destination directory.
@@ -201,8 +237,8 @@ func (c *commandRestore) constructTargetPairs() error {
 	return errors.Errorf("restore requires a source and targetpath or placeholders")
 }
 
-func (c *commandRestore) restoreOutput(ctx context.Context) (restore.Output, error) {
-	err := c.constructTargetPairs()
+func (c *commandRestore) restoreOutput(ctx context.Context, rep repo.Repository) (restore.Output, error) {
+	err := c.constructTargetPairs(rep)
 	if err != nil {
 		return nil, err
 	}
@@ -330,7 +366,7 @@ func (c *commandRestore) setupPlaceholderExpansion(ctx context.Context, rep repo
 }
 
 func (c *commandRestore) run(ctx context.Context, rep repo.Repository) error {
-	output, oerr := c.restoreOutput(ctx)
+	output, oerr := c.restoreOutput(ctx, rep)
 	if oerr != nil {
 		return errors.Wrap(oerr, "unable to initialize output")
 	}
@@ -346,7 +382,12 @@ func (c *commandRestore) run(ctx context.Context, rep repo.Repository) error {
 
 			rootEntry = re
 		} else {
-			re, err := snapshotfs.FilesystemEntryFromIDWithPath(ctx, rep, rstp.source, c.restoreConsistentAttributes)
+			source, err := c.tryToConvertPathToID(ctx, rep, rstp.source)
+			if err != nil {
+				return err
+			}
+
+			re, err := snapshotfs.FilesystemEntryFromIDWithPath(ctx, rep, source, c.restoreConsistentAttributes)
 			if err != nil {
 				return errors.Wrap(err, "unable to get filesystem entry")
 			}
@@ -404,3 +445,234 @@ func (c *commandRestore) run(ctx context.Context, rep repo.Repository) error {
 
 	return nil
 }
+
+// tryToConvertPathToID checks if the source is a path and in this case returns the ID of the snapshot
+// containing the latest version available.
+func (c *commandRestore) tryToConvertPathToID(ctx context.Context, rep repo.Repository, source string) (string, error) {
+	pathElements := strings.Split(filepath.ToSlash(source), "/")
+
+	if pathElements[0] != "" {
+		_, err := index.ParseID(pathElements[0])
+		if err == nil {
+			// source is an ID
+			return source, nil
+		}
+	}
+
+	// Consider source as a path
+
+	if c.snapshotTime == "" {
+		return "", errors.New("a snapshot time is needed to use a path as source")
+	}
+
+	filter, err := createSnapshotTimeFilter(c.snapshotTime)
+	if err != nil {
+		return "", err
+	}
+
+	si, err := snapshot.ParseSourceInfo(source, rep.ClientOptions().Hostname, rep.ClientOptions().Username)
+	if err != nil {
+		return "", errors.Errorf("invalid directory: '%s': %s", source, err)
+	}
+
+	if si.Path == "" {
+		return "", errors.Errorf("the source must contain a path element")
+	}
+
+	manifestIDs, err := findSnapshotsForSource(ctx, rep, si, map[string]string{})
+	if err != nil {
+		return "", err
+	}
+
+	if len(manifestIDs) == 0 {
+		return "", errors.Errorf("no snapshots contain data for %v", source)
+	}
+
+	ms, err := snapshot.LoadSnapshots(ctx, rep, manifestIDs)
+	if err != nil {
+		return "", errors.Wrap(err, "unable to load snapshots")
+	}
+
+	m, relPath, ohid := findLastManifestWithPath(ctx, rep, ms, si.Path, filter)
+	if m == nil {
+		return "", errors.Errorf("no snapshots contain data for %v", source)
+	}
+
+	log(ctx).Infof("Restoring from\n"+
+		"   Snapshot source: %v\n"+
+		"   Snapshot time: %v\n"+
+		"   Relative path: %v\n"+
+		"   Object ID: %v", m.Source, formatTimestamp(m.StartTime.ToTime()), relPath, ohid)
+
+	return ohid.String(), nil
+}
+
+func createSnapshotTimeFilter(timespec string) (func(*snapshot.Manifest, int, int) bool, error) {
+	if timespec == "" || timespec == "latest" {
+		return func(m *snapshot.Manifest, i, total int) bool {
+			return i == 0
+		}, nil
+	}
+
+	if timespec == "oldest" {
+		return func(m *snapshot.Manifest, i, total int) bool {
+			return i == total-1
+		}, nil
+	}
+
+	t, err := computeMaxTime(timespec)
+	if err != nil {
+		return nil, err
+	}
+
+	return func(m *snapshot.Manifest, i, total int) bool {
+		return m.StartTime.ToTime().Before(t)
+	}, nil
+}
+
+var timeAgoRE = regexp.MustCompile(`^(?:(\d{1,3})(?:y|year|years)-)?(?:(\d{1,3})(?:m|mo|month|months)-)?(?:(\d{1,3})(?:d|day|days)-)?ago$`)
+
+// computeMaxTime returns the first time after the max allowed.
+func computeMaxTime(timespec string) (time.Time, error) {
+	now := clock.Now()
+
+	if timespec == "yesterday" {
+		return time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.Local), nil
+	}
+
+	if timespec == "last-month" {
+		return time.Date(now.Year(), now.Month(), 1, 0, 0, 0, 0, time.Local), nil
+	}
+
+	if timespec == "last-year" {
+		return time.Date(now.Year(), 1, 1, 0, 0, 0, 0, time.Local), nil
+	}
+
+	if strings.HasSuffix(timespec, "-ago") {
+		ymd := timeAgoRE.FindStringSubmatch(timespec)
+		if ymd != nil {
+			t := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.Local)
+
+			years, _ := strconv.Atoi(ymd[1])
+			months, _ := strconv.Atoi(ymd[2])
+			days, _ := strconv.Atoi(ymd[3])
+
+			// +1 to compute end time of current day
+			return t.AddDate(-years, -months, -days+1), nil
+		}
+	}
+
+	// Just used as markers, the value does not really matter
+	day := 24 * time.Hour //nolint:gomnd
+	month := 30 * day     //nolint:gomnd
+	year := 12 * month    //nolint:gomnd
+
+	formats := []struct {
+		format    string
+		precision time.Duration
+	}{
+		// Used by kopia output
+		{"2006-01-02 15:04:05 MST", time.Second},
+		{"2006-01-02 15:04:05.000 MST", time.Millisecond},
+
+		// Others
+		{"2006-1-2T15:04:05Z07:00", time.Second},
+		{"2006-1-2T15:04:05Z0700", time.Second},
+		{"2006-1-2T15:04:05Z07", time.Second},
+		{"2006-1-2T15:04:05", time.Second},
+		{"2006-1-2T15:04Z07", time.Minute},
+		{"2006-1-2T15:04", time.Minute},
+		{"2006-1-2T15", time.Hour},
+		{"2006-1-2 15:04:05Z0700", time.Second},
+		{"2006-1-2 15:04:05Z07", time.Second},
+		{"2006-1-2 15:04:05", time.Second},
+		{"2006-1-2 15:04Z07", time.Minute},
+		{"2006-1-2 15:04", time.Minute},
+		{"2006-1-2 15", time.Hour},
+		{"2006-1-2", day},
+		{"2006-1", month},
+		{"2006", year},
+	}
+	for _, f := range formats {
+		t, err := time.Parse(f.format, timespec)
+		if err != nil {
+			continue
+		}
+
+		// If no timezone is given, assume local time
+		if !strings.Contains(f.format, "Z") && !strings.Contains(f.format, "MST") {
+			t = time.Date(t.Year(), t.Month(), t.Day(), t.Hour(), t.Minute(), t.Second(), t.Nanosecond(), time.Local)
+		}
+
+		switch f.precision {
+		case year:
+			t = t.AddDate(1, 0, 0)
+		case month:
+			t = t.AddDate(0, 1, 0)
+		case day:
+			t = t.AddDate(0, 0, 1)
+		default:
+			t = t.Add(f.precision)
+		}
+
+		return t, nil
+	}
+
+	return now, errors.Errorf("Invalid time spec: %v", timespec)
+}
+
+func findLastManifestWithPath(ctx context.Context, rep repo.Repository, ms []*snapshot.Manifest, path string, filter func(*snapshot.Manifest, int, int) bool) (*snapshot.Manifest, string, object.ID) {
+	ms = snapshot.SortByTime(ms, true)
+
+	type candidateInfo struct {
+		m    *snapshot.Manifest
+		pe   []string
+		ohid object.HasObjectID
+	}
+
+	var candidates []candidateInfo
+
+	for _, m := range ms {
+		if m.IncompleteReason != "" {
+			// Ignore this snapshot
+			continue
+		}
+
+		root, err := snapshotfs.SnapshotRoot(rep, m)
+		if err != nil {
+			// Ignore this snapshot
+			continue
+		}
+
+		pathElements, err := findRelativePathParts(m, path)
+		if err != nil {
+			// Ignore this snapshot
+			continue
+		}
+
+		ent, err := snapshotfs.GetNestedEntry(ctx, root, pathElements)
+		if err != nil {
+			// Ignore this snapshot
+			continue
+		}
+
+		ohid, ok := ent.(object.HasObjectID)
+		if !ok {
+			// Ignore this snapshot
+			continue
+		}
+
+		candidates = append(candidates, candidateInfo{m, pathElements, ohid})
+	}
+
+	for i, c := range candidates {
+		if !filter(c.m, i, len(candidates)) {
+			// Ignore this snapshot
+			continue
+		}
+
+		return c.m, filepath.Join(c.pe...), c.ohid.ObjectID()
+	}
+
+	return nil, "", object.ID{}
+}
diff --git a/cli/command_snapshot_list.go b/cli/command_snapshot_list.go
index 62da00a6..efce0fcd 100644
--- a/cli/command_snapshot_list.go
+++ b/cli/command_snapshot_list.go
@@ -62,25 +62,19 @@ func (c *commandSnapshotList) setup(svc appServices, parent commandParent) {
 	cmd.Action(svc.repositoryReaderAction(c.run))
 }
 
-func findSnapshotsForSource(ctx context.Context, rep repo.Repository, sourceInfo snapshot.SourceInfo, tags map[string]string) (manifestIDs []manifest.ID, relPath string, err error) {
+func findSnapshotsForSource(ctx context.Context, rep repo.Repository, sourceInfo snapshot.SourceInfo, tags map[string]string) (manifestIDs []manifest.ID, err error) {
+	var result []manifest.ID
+
 	for len(sourceInfo.Path) > 0 {
 		list, err := snapshot.ListSnapshotManifests(ctx, rep, &sourceInfo, tags)
 		if err != nil {
-			return nil, "", errors.Wrapf(err, "error listing manifests for %v", sourceInfo)
+			return nil, errors.Wrapf(err, "error listing manifests for %v", sourceInfo)
 		}
 
 		if len(list) > 0 {
-			return list, relPath, nil
+			result = append(result, list...)
 		}
 
-		if len(relPath) > 0 {
-			relPath = filepath.Base(sourceInfo.Path) + "/" + relPath
-		} else {
-			relPath = filepath.Base(sourceInfo.Path)
-		}
-
-		log(ctx).Debugf("No snapshots of %v@%v:%v", sourceInfo.UserName, sourceInfo.Host, sourceInfo.Path)
-
 		parentPath := filepath.Dir(sourceInfo.Path)
 		if parentPath == sourceInfo.Path {
 			break
@@ -89,7 +83,24 @@ func findSnapshotsForSource(ctx context.Context, rep repo.Repository, sourceInfo
 		sourceInfo.Path = parentPath
 	}
 
-	return nil, "", nil
+	return result, nil
+}
+
+func findRelativePathParts(m *snapshot.Manifest, path string) ([]string, error) {
+	if path == "" {
+		return nil, nil
+	}
+
+	relPath, err := filepath.Rel(m.Source.Path, path)
+	if err != nil {
+		return nil, err //nolint:wrapcheck
+	}
+
+	if relPath == "." {
+		return nil, nil
+	}
+
+	return strings.Split(filepath.ToSlash(relPath), "/"), nil
 }
 
 func findManifestIDs(ctx context.Context, rep repo.Repository, source string, tags map[string]string) ([]manifest.ID, string, error) {
@@ -103,12 +114,9 @@ func findManifestIDs(ctx context.Context, rep repo.Repository, source string, ta
 		return nil, "", errors.Errorf("invalid directory: '%s': %s", source, err)
 	}
 
-	manifestIDs, relPath, err := findSnapshotsForSource(ctx, rep, si, tags)
-	if relPath != "" {
-		relPath = "/" + relPath
-	}
+	manifestIDs, err := findSnapshotsForSource(ctx, rep, si, tags)
 
-	return manifestIDs, relPath, err
+	return manifestIDs, si.Path, err
 }
 
 func (c *commandSnapshotList) run(ctx context.Context, rep repo.Repository) error {
@@ -117,7 +125,7 @@ func (c *commandSnapshotList) run(ctx context.Context, rep repo.Repository) erro
 		return err
 	}
 
-	manifestIDs, relPath, err := findManifestIDs(ctx, rep, c.snapshotListPath, tags)
+	manifestIDs, fullPath, err := findManifestIDs(ctx, rep, c.snapshotListPath, tags)
 	if err != nil {
 		return err
 	}
@@ -131,7 +139,7 @@ func (c *commandSnapshotList) run(ctx context.Context, rep repo.Repository) erro
 		return c.outputJSON(ctx, rep, manifests)
 	}
 
-	return c.outputManifestGroups(ctx, rep, manifests, strings.Split(relPath, "/"))
+	return c.outputManifestGroups(ctx, rep, manifests, fullPath)
 }
 
 // SnapshotManifest defines the JSON output for the CLI snapshot commands.
@@ -190,7 +198,7 @@ func (c *commandSnapshotList) shouldOutputSnapshotSource(rep repo.Repository, sr
 	return src.UserName == co.Username
 }
 
-func (c *commandSnapshotList) outputManifestGroups(ctx context.Context, rep repo.Repository, manifests []*snapshot.Manifest, relPathParts []string) error {
+func (c *commandSnapshotList) outputManifestGroups(ctx context.Context, rep repo.Repository, manifests []*snapshot.Manifest, path string) error {
 	separator := ""
 
 	var anyOutput bool
@@ -214,6 +222,11 @@ func (c *commandSnapshotList) outputManifestGroups(ctx context.Context, rep repo
 			pol.RetentionPolicy.ComputeRetentionReasons(snapshotGroup)
 		}
 
+		relPathParts, err := findRelativePathParts(snapshotGroup[0], path)
+		if err != nil {
+			return err
+		}
+
 		if err := c.outputManifestFromSingleSource(ctx, rep, snapshotGroup, relPathParts); err != nil {
 			return err
 		}
diff --git a/go.mod b/go.mod
index f93c8326..fc44390f 100644
--- a/go.mod
+++ b/go.mod
@@ -9,7 +9,7 @@ require (
 	github.com/GehirnInc/crypt v0.0.0-20200316065508-bb7000b8a962 // indirect
 	github.com/alecthomas/kingpin v1.3.8-0.20220615105907-eae6867f4166 // this is pulling master, which is newer than v2
 	github.com/alecthomas/units v0.0.0-20210927113745-59d0afb8317a
-	github.com/aws/aws-sdk-go v1.44.84
+	github.com/aws/aws-sdk-go v1.44.94
 	github.com/chmduquesne/rollinghash v4.0.0+incompatible
 	github.com/dustinkirkland/golang-petname v0.0.0-20191129215211-8e5a1ed0cff0
 	github.com/fatih/color v1.13.0
@@ -19,7 +19,7 @@ require (
 	github.com/golang-jwt/jwt/v4 v4.4.2
 	github.com/golang/protobuf v1.5.2
 	github.com/google/fswalker v0.2.1-0.20200214223026-f0e929ba4126
-	github.com/google/go-cmp v0.5.8
+	github.com/google/go-cmp v0.5.9
 	github.com/google/readahead v0.0.0-20161222183148-eaceba169032 // indirect
 	github.com/google/uuid v1.3.0
 	github.com/gorilla/mux v1.8.0
@@ -30,7 +30,7 @@ require (
 	github.com/kylelemons/godebug v1.1.0
 	github.com/mattn/go-colorable v0.1.13
 	github.com/minio/md5-simd v1.1.2 // indirect
-	github.com/minio/minio-go/v7 v7.0.34
+	github.com/minio/minio-go/v7 v7.0.36
 	github.com/minio/sha256-simd v1.0.0 // indirect
 	github.com/natefinch/atomic v1.0.1
 	github.com/pierrec/lz4 v2.6.1+incompatible
@@ -52,12 +52,12 @@ require (
 	golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa
 	golang.org/x/mod v0.5.1
 	golang.org/x/net v0.0.0-20220722155237-a158d28d115b
-	golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2
+	golang.org/x/oauth2 v0.0.0-20220822191816-0ebed06d0094
 	golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f
-	golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
+	golang.org/x/sys v0.0.0-20220825204002-c680a09ffe64
 	golang.org/x/term v0.0.0-20210927222741-03fcf44c2211
 	golang.org/x/text v0.3.7
-	google.golang.org/api v0.93.0
+	google.golang.org/api v0.95.0
 	google.golang.org/grpc v1.49.0
 	google.golang.org/protobuf v1.28.1
 	gopkg.in/ini.v1 v1.66.6 // indirect
@@ -83,7 +83,7 @@ require (
 	github.com/googleapis/gax-go/v2 v2.4.0 // indirect
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
-	github.com/kopia/htmluibuild v0.0.0-20220817045620-d7803267445c
+	github.com/kopia/htmluibuild v0.0.0-20220909025317-0e48dcdc4db4
 	github.com/kr/fs v0.1.0 // indirect
 	github.com/mattn/go-ieproxy v0.0.1 // indirect
 	github.com/mattn/go-isatty v0.0.16 // indirect
@@ -103,13 +103,13 @@ require (
 )
 
 require (
-	github.com/chromedp/cdproto v0.0.0-20220812200530-d0d83820bffc
-	github.com/chromedp/chromedp v0.8.4
+	github.com/chromedp/cdproto v0.0.0-20220827030233-358ed4af73cf
+	github.com/chromedp/chromedp v0.8.5
 	github.com/edsrzf/mmap-go v1.1.0
 	github.com/hashicorp/golang-lru v0.5.4
 	github.com/klauspost/reedsolomon v1.10.0
 	go.opentelemetry.io/otel v1.9.0
-	go.opentelemetry.io/otel/exporters/jaeger v1.8.0
+	go.opentelemetry.io/otel/exporters/jaeger v1.9.0
 	go.opentelemetry.io/otel/sdk v1.9.0
 	go.opentelemetry.io/otel/trace v1.9.0
 )
diff --git a/go.sum b/go.sum
index df45688f..3bf44969 100644
--- a/go.sum
+++ b/go.sum
@@ -101,8 +101,8 @@ github.com/alecthomas/units v0.0.0-20210927113745-59d0afb8317a/go.mod h1:OMCwj8V
 github.com/alessio/shellescape v1.4.1 h1:V7yhSDDn8LP4lc4jS8pFkt0zCnzVJlG5JXy9BVKJUX0=
 github.com/alessio/shellescape v1.4.1/go.mod h1:PZAiSCk0LJaZkiCSkPv8qIobYglO3FPpyFjDCtHLS30=
 github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
-github.com/aws/aws-sdk-go v1.44.84 h1:orGogGRrizQSqn3lBnaP/FQIcjPMLf9azDO0h+oTJr0=
-github.com/aws/aws-sdk-go v1.44.84/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
+github.com/aws/aws-sdk-go v1.44.94 h1:hDqJSv03ZVvqT448gUE63JEIHKx++vKLoDkiZxbNmIk=
+github.com/aws/aws-sdk-go v1.44.94/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
 github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8=
 github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@@ -117,10 +117,10 @@ github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cb
 github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/chmduquesne/rollinghash v4.0.0+incompatible h1:hnREQO+DXjqIw3rUTzWN7/+Dpw+N5Um8zpKV0JOEgbo=
 github.com/chmduquesne/rollinghash v4.0.0+incompatible/go.mod h1:Uc2I36RRfTAf7Dge82bi3RU0OQUmXT9iweIcPqvr8A0=
-github.com/chromedp/cdproto v0.0.0-20220812200530-d0d83820bffc h1:xGhCpiX5oNMoZGnzYvv1ne4muVRl2SDHH5fL7oUbZAY=
-github.com/chromedp/cdproto v0.0.0-20220812200530-d0d83820bffc/go.mod h1:5Y4sD/eXpwrChIuxhSr/G20n9CdbCmoerOHnuAf0Zr0=
-github.com/chromedp/chromedp v0.8.4 h1:50NSLCXC38kGsCmLCi8tXZ5V5FtQ3BOV/90u1O06Gq4=
-github.com/chromedp/chromedp v0.8.4/go.mod h1:ikuiJrEMoOnTPFUBu6jV0XZFOGW6W/Nhgk/OZyvh00o=
+github.com/chromedp/cdproto v0.0.0-20220827030233-358ed4af73cf h1:e0oJZmGJidTRZ0FvWXNhdj9OaOMN30Yf+T3K2Tf3L+s=
+github.com/chromedp/cdproto v0.0.0-20220827030233-358ed4af73cf/go.mod h1:5Y4sD/eXpwrChIuxhSr/G20n9CdbCmoerOHnuAf0Zr0=
+github.com/chromedp/chromedp v0.8.5 h1:HAVg54yQFcn7sg5reVjXtoI1eQaFxhjAjflHACicUFw=
+github.com/chromedp/chromedp v0.8.5/go.mod h1:xal2XY5Di7m/bzlGwtoYpmgIOfDqCakOIVg5OfdkPZ4=
 github.com/chromedp/sysutil v1.0.0 h1:+ZxhTpfpZlmchB58ih/LBHX52ky7w2VhQVKQMucy3Ic=
 github.com/chromedp/sysutil v1.0.0/go.mod h1:kgWmDdq8fTzXYcKIBqIYvRRTnYb9aNS9moAV0xufSww=
 github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
@@ -255,8 +255,9 @@ github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/
 github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.7/go.mod h1:n+brtR0CgQNWTVd5ZUFpTBC8YFBDLK/h/bpaJ8/DtOE=
-github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
 github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
+github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
@@ -340,8 +341,8 @@ github.com/klauspost/reedsolomon v1.10.0 h1:MonMtg979rxSHjwtsla5dZLhreS0Lu42AyQ2
 github.com/klauspost/reedsolomon v1.10.0/go.mod h1:qHMIzMkuZUWqIh8mS/GruPdo3u0qwX2jk/LH440ON7Y=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
-github.com/kopia/htmluibuild v0.0.0-20220817045620-d7803267445c h1:UCbhxRmJ2AoBaVtpv0xBjteJ7UtRJCNlbU9uu0/Aoyc=
-github.com/kopia/htmluibuild v0.0.0-20220817045620-d7803267445c/go.mod h1:eWer4rx9P8lJo2eKc+Q7AZ1dE1x1hJNdkbDFPzMu1Hw=
+github.com/kopia/htmluibuild v0.0.0-20220909025317-0e48dcdc4db4 h1:rfylAI7XtcqMXgcVANjHfSBZYdesDBptqP4z6Nup2wA=
+github.com/kopia/htmluibuild v0.0.0-20220909025317-0e48dcdc4db4/go.mod h1:eWer4rx9P8lJo2eKc+Q7AZ1dE1x1hJNdkbDFPzMu1Hw=
 github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8=
 github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
@@ -373,8 +374,8 @@ github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0j
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
 github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
 github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
-github.com/minio/minio-go/v7 v7.0.34 h1:JMfS5fudx1mN6V2MMNyCJ7UMrjEzZzIvMgfkWc1Vnjk=
-github.com/minio/minio-go/v7 v7.0.34/go.mod h1:nCrRzjoSUQh8hgKKtu3Y708OLvRLtuASMg2/nvmbarw=
+github.com/minio/minio-go/v7 v7.0.36 h1:KPzAl8C6jcRFEUsGUHR6deRivvKATPNZThzi7D9y/sc=
+github.com/minio/minio-go/v7 v7.0.36/go.mod h1:nCrRzjoSUQh8hgKKtu3Y708OLvRLtuASMg2/nvmbarw=
 github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g=
 github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -488,15 +489,12 @@ go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
 go.opencensus.io v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M=
 go.opencensus.io v0.23.0/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E=
-go.opentelemetry.io/otel v1.8.0/go.mod h1:2pkj+iMj0o03Y+cW6/m8Y4WkRdYN3AvCXCnzRMp9yvM=
 go.opentelemetry.io/otel v1.9.0 h1:8WZNQFIB2a71LnANS9JeyidJKKGOOremcUtb/OtHISw=
 go.opentelemetry.io/otel v1.9.0/go.mod h1:np4EoPGzoPs3O67xUVNoPPcmSvsfOxNlNA4F4AC+0Eo=
-go.opentelemetry.io/otel/exporters/jaeger v1.8.0 h1:TLLqD6kDhLPziEC7pgPrMvP9lAqdk3n1gf8DiFSnfW8=
-go.opentelemetry.io/otel/exporters/jaeger v1.8.0/go.mod h1:GbWg+ng88rDtx+id26C34QLqw2erqJeAjsCx9AFeHfE=
-go.opentelemetry.io/otel/sdk v1.8.0/go.mod h1:uPSfc+yfDH2StDM/Rm35WE8gXSNdvCg023J6HeGNO0c=
+go.opentelemetry.io/otel/exporters/jaeger v1.9.0 h1:gAEgEVGDWwFjcis9jJTOJqZNxDzoZfR12WNIxr7g9Ww=
+go.opentelemetry.io/otel/exporters/jaeger v1.9.0/go.mod h1:hquezOLVAybNW6vanIxkdLXTXvzlj2Vn3wevSP15RYs=
 go.opentelemetry.io/otel/sdk v1.9.0 h1:LNXp1vrr83fNXTHgU8eO89mhzxb/bbWAsHG6fNf3qWo=
 go.opentelemetry.io/otel/sdk v1.9.0/go.mod h1:AEZc8nt5bd2F7BC24J5R0mrjYnpEgYHyTcM/vrSple4=
-go.opentelemetry.io/otel/trace v1.8.0/go.mod h1:0Bt3PXY8w+3pheS3hQUt+wow8b1ojPaTBoTCh2zIFI4=
 go.opentelemetry.io/otel/trace v1.9.0 h1:oZaCNJUjWcg60VXWee8lJKlqhPbXAPB51URuR47pQYc=
 go.opentelemetry.io/otel/trace v1.9.0/go.mod h1:2737Q0MuG8q1uILYm2YYVkAyLtOofiTNGg6VODnOiPo=
 go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
@@ -632,8 +630,9 @@ golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j
 golang.org/x/oauth2 v0.0.0-20220309155454-6242fa91716a/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
 golang.org/x/oauth2 v0.0.0-20220411215720-9780585627b5/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
 golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb/go.mod h1:jaDAt6Dkxork7LmZnYtzbRWj0W47D86a3TGe0YHBvmE=
-golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2 h1:+jnHzr9VPj32ykQVai5DNahi9+NSp7yYuCsl5eAQtL0=
 golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2/go.mod h1:jaDAt6Dkxork7LmZnYtzbRWj0W47D86a3TGe0YHBvmE=
+golang.org/x/oauth2 v0.0.0-20220822191816-0ebed06d0094 h1:2o1E+E8TpNLklK9nHiPiK1uzIYrIHt+cQx3ynCwq9V8=
+golang.org/x/oauth2 v0.0.0-20220822191816-0ebed06d0094/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -723,8 +722,9 @@ golang.org/x/sys v0.0.0-20220624220833-87e55d714810/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab h1:2QkjZIsXupsJbJIdSjjUOgWK3aEtzyuh2mPt3l/CkeU=
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220825204002-c680a09ffe64 h1:UiNENfZ8gDvpiWw7IpOMQ27spWmThO1RwwdQVbJahJM=
+golang.org/x/sys v0.0.0-20220825204002-c680a09ffe64/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 h1:JGgROgKl9N8DuW20oFS5gxc+lE67/N3FcwmBPMe7ArY=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
@@ -840,8 +840,8 @@ google.golang.org/api v0.78.0/go.mod h1:1Sg78yoMLOhlQTeF+ARBoytAcH1NNyyl390YMy6r
 google.golang.org/api v0.80.0/go.mod h1:xY3nI94gbvBrE0J6NHXhxOmW97HG7Khjkku6AFB3Hyg=
 google.golang.org/api v0.84.0/go.mod h1:NTsGnUFJMYROtiquksZHBWtHfeMC7iYthki7Eq3pa8o=
 google.golang.org/api v0.88.0/go.mod h1:+Sem1dnrKlrXMR/X0bPnMWyluQe4RsNoYfmNLhOIkzw=
-google.golang.org/api v0.93.0 h1:T2xt9gi0gHdxdnRkVQhT8mIvPaXKNsDNWz+L696M66M=
-google.golang.org/api v0.93.0/go.mod h1:+Sem1dnrKlrXMR/X0bPnMWyluQe4RsNoYfmNLhOIkzw=
+google.golang.org/api v0.95.0 h1:d1c24AAS01DYqXreBeuVV7ewY/U8Mnhh47pwtsgVtYg=
+google.golang.org/api v0.95.0/go.mod h1:eADj+UBuxkh5zlrSntJghuNeg8HwQ1w5lTKkuqaETEI=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
diff --git a/internal/bigmap/bigmap_internal.go b/internal/bigmap/bigmap_internal.go
new file mode 100644
index 00000000..691acaf8
--- /dev/null
+++ b/internal/bigmap/bigmap_internal.go
@@ -0,0 +1,476 @@
+// Package bigmap implements a custom hashmap data structure where keys and values are binary
+// and keys are meant to be well-distributed hashes, such as content IDs, object IDs, etc.
+//
+// Unlike regular maps this map is limited to adding and getting elements but does not support
+// any iteration or deletion, but is much more efficient in terms of memory usage.
+//
+// Data for the hash table is stored in large contiguous memory blocks (for small sets)
+// and automatically spills over to memory-mapped files for larger sets using only 8 bytes
+// per key in RAM.
+package bigmap
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"os"
+	"path/filepath"
+	"sync"
+
+	"github.com/edsrzf/mmap-go"
+	"github.com/google/uuid"
+	"github.com/pkg/errors"
+
+	"github.com/kopia/kopia/repo/logging"
+)
+
+const (
+	defaultNumMemorySegments    = 8          // number of segments to keep in RAM
+	defaultMemorySegmentSize    = 18 * 1e6   // 18MB enough to store >1M 16-17-byte keys
+	defaultFileSegmentSize      = 1024 << 20 // 1 GiB
+	defaultInitialSizeLogarithm = 20
+	mmapFileMode                = 0o600
+
+	// grow hash table above this percentage utilization, higher values (close to 100) will be very slow,
+	// smaller values will waste memory.
+	defaultLoadFactorPercentage = 75
+
+	minKeyLength = 4 // first 4 bytes of each key are used as uint32 hash value
+	maxKeyLength = 255
+
+	minSizeLogarithm = 8
+)
+
+var log = logging.Module("bigmap")
+
+// Options provides options for the internalMap.
+type Options struct {
+	LoadFactorPercentage int   // grow the size of the hash table when this percentage full
+	NumMemorySegments    int   // number of segments to keep in RAM
+	MemorySegmentSize    int64 // size of a single memory segment
+	FileSegmentSize      int   // size of a single file segment, defaults to 1 GiB
+
+	InitialSizeLogarithm int // logarithm of the initial size of the hash table, default - 20
+}
+
+// internalMap is a custom hashtable implementation using https://en.wikipedia.org/wiki/Double_hashing
+// that stores all (key,value) pairs densely packed in segments of fixed size to minimize data
+// fragmentation.
+type internalMap struct {
+	hasValues bool    // +checklocksignore
+	opts      Options // +checklocksignore
+
+	mu sync.RWMutex
+	// +checklocks:mu
+	tableSizeIndex int // index into tableSizesPrimes == len(slots)
+	// +checklocks:mu
+	segments []mmap.MMap // ([keyLength][key][valueLength][value])*
+	// +checklocks:mu
+	slots []entry // current hash table slots
+
+	// +checklocks:mu
+	count int // number of elements in the hash tables
+	// +checklocks:mu
+	h2Prime uint64 // prime < len(slots)
+
+	// +checklocks:mu
+	cleanups []func()
+
+	// +checklocks:mu
+	tempDir string
+}
+
+// The list of prime numbers close to 2^N from https://primes.utm.edu/lists/2small/0bit.html
+
+//nolint:gochecknoglobals
+var tableSizesPrimes = []uint64{
+	1<<8 - 59, // minSizeLogarithm
+	1<<9 - 55,
+	1<<10 - 57,
+	1<<11 - 61,
+	1<<12 - 77,
+	1<<13 - 91,
+	1<<14 - 111,
+	1<<15 - 135,
+	1<<16 - 123,
+	1<<17 - 99,
+	1<<18 - 93,
+	1<<19 - 87,
+	1<<20 - 185, // initial H2 prime
+	1<<21 - 129, // initial number of hashtable slots
+	1<<22 - 123,
+	1<<23 - 159,
+	1<<24 - 167,
+	1<<25 - 183,
+	1<<26 - 135,
+	1<<27 - 235,
+	1<<28 - 273,
+	1<<29 - 133,
+	1<<30 - 173,
+	1<<31 - 171,
+	1<<32 - 267,
+	1<<33 - 355,
+	1<<34 - 281,
+	1<<35 - 325,
+	1<<36 - 233,
+	1<<37 - 375,
+	1<<38 - 257,
+	1<<39 - 301,
+	1<<40 - 437,
+	1<<41 - 139,
+	1<<42 - 227,
+	1<<43 - 369,
+	1<<44 - 377,
+	1<<45 - 229,
+	1<<46 - 311,
+	1<<47 - 649,
+	1<<48 - 257,
+	1<<49 - 339,
+	1<<50 - 233,
+	1<<51 - 439,
+	1<<52 - 395,
+	1<<53 - 421,
+	1<<54 - 327,
+	1<<55 - 267,
+	1<<56 - 195,
+	1<<57 - 423,
+	1<<58 - 251,
+	1<<59 - 871,
+	1<<60 - 453,
+	1<<61 - 465,
+	1<<62 - 273,
+	1<<63 - 471,
+}
+
+type entry struct {
+	segment uint32 // 0-empty, otherwise index into internalMap.segments+1
+	offset  uint32 // offset within a segment
+}
+
+// Contains returns true if the provided key is in the map.
+func (m *internalMap) Contains(key []byte) bool {
+	m.mu.RLock()
+	defer m.mu.RUnlock()
+
+	slot := m.findSlot(key)
+
+	return m.slots[slot].segment != 0
+}
+
+// Get gets the value associated with a given key. It is appended to the provided buffer.
+func (m *internalMap) Get(buf, key []byte) ([]byte, bool) {
+	m.mu.RLock()
+	defer m.mu.RUnlock()
+
+	slot := m.findSlot(key)
+	e := m.slots[slot]
+
+	if e.segment == 0 {
+		return nil, false
+	}
+
+	if !m.hasValues {
+		return nil, true
+	}
+
+	data := m.segments[e.segment-1] // 1-indexed
+	koff := e.offset
+
+	off := koff + uint32(data[koff]) + 1
+	vlen, vlenLen := binary.Uvarint(data[off:])
+	start := off + uint32(vlenLen)
+
+	return append(buf, data[start:start+uint32(vlen)]...), true
+}
+
+func (m *internalMap) hashValue(key []byte) uint64 {
+	if len(key) < 8 { //nolint:gomnd
+		return uint64(binary.BigEndian.Uint32(key))
+	}
+
+	return binary.BigEndian.Uint64(key)
+}
+
+// h2 returns the secondary hash value used for double hashing.
+func (m *internalMap) h2(key []byte) uint64 {
+	if len(key) < 16 { //nolint:gomnd
+		// use linear scan.
+		return 1
+	}
+
+	return binary.BigEndian.Uint64(key[8:])
+}
+
+// +checklocksread:m.mu
+func (m *internalMap) keyEquals(e entry, key []byte) bool {
+	data := m.segments[e.segment-1] // 1-indexed
+	koff := e.offset
+	keyLen := uint32(data[koff])
+
+	return bytes.Equal(key, data[koff+1:koff+1+keyLen])
+}
+
+// +checklocksread:m.mu
+func (m *internalMap) findSlotInSlice(key []byte, slots []entry, h2Prime uint64) uint64 {
+	slot := m.hashValue(key) % uint64(len(slots))
+
+	delta := m.h2(key) % h2Prime
+	if delta == 0 {
+		delta = 1
+	}
+
+	for slots[slot].segment != 0 && !m.keyEquals(slots[slot], key) {
+		slot = (slot + delta) % uint64(len(slots))
+	}
+
+	return slot
+}
+
+// +checklocksread:m.mu
+func (m *internalMap) findSlot(key []byte) uint64 {
+	return m.findSlotInSlice(key, m.slots, m.h2Prime)
+}
+
+// +checklocks:m.mu
+func (m *internalMap) growLocked(newSize uint64) {
+	newSlots := make([]entry, newSize)
+	newH2Prime := uint64(len(m.slots))
+
+	for segNum, seg := range m.segments {
+		p := 0
+
+		for p < len(seg) && seg[p] != 0 {
+			koff := p
+			key := seg[p+1 : p+1+int(seg[p])]
+			p += len(key) + 1
+
+			if m.hasValues {
+				vlen, vlenLen := binary.Uvarint(seg[p:])
+				p += vlenLen + int(vlen)
+			}
+
+			slot := m.findSlotInSlice(key, newSlots, newH2Prime)
+			newSlots[slot] = entry{segment: uint32(segNum) + 1, offset: uint32(koff)}
+		}
+	}
+
+	m.h2Prime = newH2Prime
+	m.slots = newSlots
+}
+
+// PutIfAbsent conditionally adds the provided (key, value) to the map if the provided key is absent.
+func (m *internalMap) PutIfAbsent(ctx context.Context, key, value []byte) bool {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	if len(key) < minKeyLength {
+		panic("key too short")
+	}
+
+	if len(key) > maxKeyLength {
+		panic("key too long")
+	}
+
+	if !m.hasValues && len(value) > 0 {
+		panic("values are not enabled")
+	}
+
+	slot := m.findSlot(key)
+	if m.slots[slot].segment != 0 {
+		return false
+	}
+
+	if m.count >= len(m.slots)*m.opts.LoadFactorPercentage/100 {
+		m.tableSizeIndex++
+		m.growLocked(tableSizesPrimes[m.tableSizeIndex])
+		slot = m.findSlot(key)
+	}
+
+	payloadLength := 1 + len(key) + binary.MaxVarintLen32 + len(value)
+
+	current := m.segments[len(m.segments)-1]
+
+	// ensure that the key we're adding fits in the segment, start a new segment if not enough room.
+	if len(current)+payloadLength > cap(current) {
+		current = m.newSegment(ctx)
+		m.segments = append(m.segments, current)
+	}
+
+	koff := uint32(len(current))
+
+	current = append(current, byte(len(key)))
+	current = append(current, key...)
+
+	// append the value
+	if m.hasValues {
+		var vlen [binary.MaxVarintLen32]byte
+
+		n := binary.PutUvarint(vlen[:], uint64(len(value)))
+
+		current = append(current, vlen[0:n]...)
+		current = append(current, value...)
+	}
+
+	m.segments[len(m.segments)-1] = current
+	m.count++
+
+	m.slots[slot] = entry{
+		segment: uint32(len(m.segments)), // this is 1-based, 0==empty slot
+		offset:  koff,
+	}
+
+	return true
+}
+
+// +checklocks:m.mu
+func (m *internalMap) newMemoryMappedSegment(ctx context.Context) (mmap.MMap, error) {
+	flags := 0
+
+	f, err := m.maybeCreateMappedFile(ctx)
+	if err != nil {
+		return nil, err
+	}
+
+	s, err := mmap.MapRegion(f, m.opts.FileSegmentSize, mmap.RDWR, flags, 0)
+	if err != nil {
+		return nil, errors.Wrap(err, "unable to map region")
+	}
+
+	m.cleanups = append(m.cleanups, func() {
+		if err := s.Unmap(); err != nil {
+			log(ctx).Warnf("unable to unmap memory region: %v", err)
+		}
+	})
+
+	return s[:0], nil
+}
+
+// +checklocks:m.mu
+func (m *internalMap) maybeCreateMappedFile(ctx context.Context) (*os.File, error) {
+	if m.tempDir == "" {
+		tempDir, err := os.MkdirTemp("", "kopia-map")
+		if err != nil {
+			return nil, errors.Wrap(err, "unable to create temp directory")
+		}
+
+		m.tempDir = tempDir
+	}
+
+	fname := filepath.Join(m.tempDir, uuid.NewString())
+
+	f, err := os.OpenFile(fname, os.O_CREATE|os.O_EXCL|os.O_RDWR, mmapFileMode) //nolint:gosec
+	if err != nil {
+		return nil, errors.Wrap(err, "unable to create memory-mapped file")
+	}
+
+	if err := f.Truncate(int64(m.opts.FileSegmentSize)); err != nil {
+		closeAndRemoveFile(ctx, f, fname)
+
+		return nil, errors.Wrap(err, "unable to truncate memory-mapped file")
+	}
+
+	m.cleanups = append(m.cleanups, func() {
+		closeAndRemoveFile(ctx, f, fname)
+	})
+
+	return f, nil
+}
+
+func closeAndRemoveFile(ctx context.Context, f *os.File, fname string) {
+	if err := f.Close(); err != nil {
+		log(ctx).Warnf("unable to close segment file: %v", err)
+	}
+
+	if err := os.Remove(fname); err != nil {
+		log(ctx).Warnf("unable to remove segment file: %v", err)
+	}
+}
+
+// +checklocks:m.mu
+func (m *internalMap) newSegment(ctx context.Context) mmap.MMap {
+	var s mmap.MMap
+
+	if len(m.segments) >= m.opts.NumMemorySegments {
+		var err error
+
+		s, err = m.newMemoryMappedSegment(ctx)
+		if err != nil {
+			log(ctx).Warnf("unable to create memory-mapped segment: %v", err)
+		}
+	}
+
+	if s == nil {
+		s = make([]byte, 0, m.opts.MemorySegmentSize)
+	}
+
+	return s
+}
+
+// Close releases all resources associated with a map.
+func (m *internalMap) Close(ctx context.Context) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	for i := len(m.cleanups) - 1; i >= 0; i-- {
+		m.cleanups[i]()
+	}
+
+	m.cleanups = nil
+	m.segments = nil
+
+	if m.tempDir != "" {
+		os.RemoveAll(m.tempDir) //nolint:errcheck
+	}
+}
+
+// newInternalMap creates new internalMap.
+func newInternalMap(ctx context.Context) (*internalMap, error) {
+	return newInternalMapWithOptions(ctx, true, nil)
+}
+
+// newInternalMapWithOptions creates a new instance of internalMap.
+func newInternalMapWithOptions(ctx context.Context, hasValues bool, opts *Options) (*internalMap, error) {
+	if opts == nil {
+		opts = &Options{}
+	}
+
+	if opts.LoadFactorPercentage == 0 {
+		opts.LoadFactorPercentage = defaultLoadFactorPercentage
+	}
+
+	if opts.MemorySegmentSize == 0 {
+		opts.MemorySegmentSize = defaultMemorySegmentSize
+	}
+
+	if opts.NumMemorySegments == 0 {
+		opts.NumMemorySegments = defaultNumMemorySegments
+	}
+
+	if opts.FileSegmentSize == 0 {
+		opts.FileSegmentSize = defaultFileSegmentSize
+	}
+
+	if opts.InitialSizeLogarithm == 0 {
+		opts.InitialSizeLogarithm = defaultInitialSizeLogarithm
+	}
+
+	tablewSizeIndex := opts.InitialSizeLogarithm - minSizeLogarithm
+
+	if tablewSizeIndex < 1 {
+		return nil, errors.Errorf("invalid initial size")
+	}
+
+	m := &internalMap{
+		hasValues:      hasValues,
+		opts:           *opts,
+		tableSizeIndex: tablewSizeIndex,
+		h2Prime:        tableSizesPrimes[tablewSizeIndex-1], // h2 prime < number of slots
+		slots:          make([]entry, tableSizesPrimes[tablewSizeIndex]),
+	}
+
+	m.mu.Lock()
+	m.segments = append(m.segments, m.newSegment(ctx))
+	m.mu.Unlock()
+
+	return m, nil
+}
diff --git a/internal/bigmap/bigmap_map.go b/internal/bigmap/bigmap_map.go
new file mode 100644
index 00000000..e210c677
--- /dev/null
+++ b/internal/bigmap/bigmap_map.go
@@ -0,0 +1,126 @@
+package bigmap
+
+import (
+	"context"
+	"crypto/aes"
+	"crypto/cipher"
+	"crypto/rand"
+	"encoding/binary"
+	"sync/atomic"
+
+	"github.com/pkg/errors"
+
+	"github.com/kopia/kopia/internal/gather"
+)
+
+const (
+	aesNonceSize = 12
+	aesKeySize   = 32 // AES-256
+)
+
+// Map is a wrapper around internalMap that adds encryption.
+type Map struct {
+	aead      cipher.AEAD
+	inner     *internalMap
+	nextNonce *uint64
+}
+
+// PutIfAbsent adds the element to the map, returns true if the element was added (as opposed to having
+// existed before).
+func (s *Map) PutIfAbsent(ctx context.Context, key, value []byte) bool {
+	if len(value) == 0 {
+		return s.inner.PutIfAbsent(ctx, key, nil)
+	}
+
+	var tmp gather.WriteBuffer
+	defer tmp.Close()
+
+	buf := tmp.MakeContiguous(aesNonceSize + len(value) + s.aead.Overhead())
+	nonce, input := buf[0:aesNonceSize], buf[aesNonceSize:aesNonceSize+len(value)]
+	copy(input, value)
+
+	nonceVal := atomic.AddUint64(s.nextNonce, 1)
+	if nonceVal == 0 {
+		panic("nonce counter wrapped beyond 64 bits, that should never happen")
+	}
+
+	binary.BigEndian.PutUint64(nonce, nonceVal)
+	s.aead.Seal(input[:0], nonce, input, key)
+
+	return s.inner.PutIfAbsent(ctx, key, buf)
+}
+
+// Get gets the element from the map and appends the value to the provided buffer.
+func (s *Map) Get(ctx context.Context, output, key []byte) (result []byte, ok bool, err error) {
+	if v, ok := s.inner.Get(output, key); ok {
+		result, err := s.decrypt(key, v)
+
+		return result, true, err
+	}
+
+	return nil, false, nil
+}
+
+func (s *Map) decrypt(key, buf []byte) ([]byte, error) {
+	if len(buf) == 0 {
+		return nil, nil
+	}
+
+	nonce, input := buf[0:aesNonceSize], buf[aesNonceSize:]
+
+	result, err := s.aead.Open(input[:0], nonce, input, key)
+	if err != nil {
+		return nil, errors.Errorf("unable to decrypt content: %v", err)
+	}
+
+	return result, nil
+}
+
+// Contains returns true if a given key exists in the set.
+func (s *Map) Contains(key []byte) bool {
+	return s.inner.Contains(key)
+}
+
+// Close releases resources associated with the set.
+func (s *Map) Close(ctx context.Context) {
+	s.inner.Close(ctx)
+}
+
+// NewMap creates new Map.
+func NewMap(ctx context.Context) (*Map, error) {
+	return NewMapWithOptions(ctx, nil)
+}
+
+// NewMapWithOptions creates new Map with options.
+func NewMapWithOptions(ctx context.Context, opt *Options) (*Map, error) {
+	inner, err := newInternalMapWithOptions(ctx, true, opt)
+	if err != nil {
+		return nil, err
+	}
+
+	key := make([]byte, aesKeySize)
+
+	if _, err = rand.Read(key); err != nil {
+		return nil, errors.Wrap(err, "error initializing map key")
+	}
+
+	enc, err := aes.NewCipher(key)
+	if err != nil {
+		return nil, errors.Wrap(err, "error initializing map cipher")
+	}
+
+	aead, err := cipher.NewGCM(enc)
+	if err != nil {
+		return nil, errors.Wrap(err, "error initializing map AEAD")
+	}
+
+	if aead.NonceSize() != aesNonceSize {
+		return nil, errors.Errorf("unexpected nonce size: %v, expected %v", aead.NonceSize(), aesNonceSize)
+	}
+
+	return &Map{
+		aead:      aead,
+		inner:     inner,
+		nextNonce: new(uint64),
+	}, nil
+}
diff --git a/internal/bigmap/bigmap_set.go b/internal/bigmap/bigmap_set.go
new file mode 100644
index 00000000..a170245a
--- /dev/null
+++ b/internal/bigmap/bigmap_set.go
@@ -0,0 +1,39 @@
+package bigmap
+
+import "context"
+
+// Set is a wrapper around Map that only supports Put() and Contains().
+type Set struct {
+	inner *internalMap
+}
+
+// Put adds the element to a set, returns true if the element was added (as opposed to having
+// existed before).
+func (s *Set) Put(ctx context.Context, key []byte) bool {
+	return s.inner.PutIfAbsent(ctx, key, nil)
+}
+
+// Contains returns true if a given key exists in the set.
+func (s *Set) Contains(key []byte) bool {
+	return s.inner.Contains(key)
+}
+
+// Close releases resources associated with the set.
+func (s *Set) Close(ctx context.Context) {
+	s.inner.Close(ctx)
+}
+
+// NewSet creates new Set.
+func NewSet(ctx context.Context) (*Set, error) {
+	return NewSetWithOptions(ctx, nil)
+}
+
+// NewSetWithOptions creates new Set with options.
+func NewSetWithOptions(ctx context.Context, opt *Options) (*Set, error) {
+	inner, err := newInternalMapWithOptions(ctx, false, opt)
+	if err != nil {
+		return nil, err
+	}
+
+	return &Set{inner}, nil
+}
diff --git a/internal/bigmap/bigmapbench/main.go b/internal/bigmap/bigmapbench/main.go
new file mode 100644
index 00000000..c276ee4e
--- /dev/null
+++ b/internal/bigmap/bigmapbench/main.go
@@ -0,0 +1,113 @@
+package main
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/binary"
+	"flag"
+	"fmt"
+	"os"
+	"runtime"
+	"sync"
+	"time"
+
+	"github.com/pkg/profile"
+
+	"github.com/kopia/kopia/internal/bigmap"
+	"github.com/kopia/kopia/internal/clock"
+	"github.com/kopia/kopia/repo/logging"
+)
+
+const (
+	implSyncMap           = 0
+	implMapWithEmptyValue = 1
+	implMapWithValues     = 2
+)
+
+//nolint:gochecknoglobals
+var (
+	impl          = flag.Int("impl", implMapWithEmptyValue, "Select implementation")
+	profileDir    = flag.String("profile-dir", "", "Profile directory")
+	profileCPU    = flag.Bool("profile-cpu", false, "Profile CPU")
+	profileMemory = flag.Bool("profile-memory", false, "Profile RAM")
+)
+
+func main() {
+	flag.Parse()
+
+	ctx := logging.WithLogger(context.Background(), logging.ToWriter(os.Stderr))
+
+	var (
+		bm     *bigmap.Map
+		sm     *sync.Map
+		num    [8]byte
+		keyBuf [sha256.Size]byte
+		ms0    runtime.MemStats
+	)
+
+	if *profileDir != "" {
+		pp := profile.ProfilePath(*profileDir)
+
+		if *profileCPU {
+			defer profile.Start(pp, profile.CPUProfile).Stop()
+		}
+
+		if *profileMemory {
+			defer profile.Start(pp, profile.MemProfile).Stop()
+		}
+	}
+
+	switch *impl {
+	case implSyncMap:
+		sm = &sync.Map{}
+
+		fmt.Println("using sync.Map")
+	case implMapWithEmptyValue:
+		fmt.Println("using bigmap.Map without values")
+
+		bm, _ = bigmap.NewMapWithOptions(ctx, &bigmap.Options{})
+	case implMapWithValues:
+		fmt.Println("using bigmap.Map with values")
+
+		bm, _ = bigmap.NewMapWithOptions(ctx, &bigmap.Options{})
+	}
+
+	h := sha256.New()
+
+	runtime.ReadMemStats(&ms0)
+
+	t0 := clock.Now()
+
+	for i := 0; i < 300_000_000; i++ {
+		if i%1_000_000 == 0 && i > 0 {
+			var ms runtime.MemStats
+
+			runtime.ReadMemStats(&ms)
+
+			alloc := ms.HeapAlloc - ms0.HeapAlloc
+			dur := clock.Now().Sub(t0).Truncate(time.Second)
+
+			fmt.Printf("elapsed %v count: %v M bytes: %v MB bytes/item: %v Mitems/sec: %.1f\n",
+				dur,
+				float64(i)/1e6,
+				alloc/1e6,
+				alloc/uint64(i),
+				float64(i)/dur.Seconds()/1e6)
+		}
+
+		// generate key=sha256(i) without allocations.
+		h.Reset()
+		binary.LittleEndian.PutUint64(num[:], uint64(i))
+		h.Write(num[:])
+		h.Sum(keyBuf[:0])
+
+		switch *impl {
+		case implSyncMap:
+			sm.LoadOrStore(keyBuf, nil)
+		case implMapWithEmptyValue:
+			bm.PutIfAbsent(ctx, keyBuf[:], nil)
+		case implMapWithValues:
+			bm.PutIfAbsent(ctx, keyBuf[:], keyBuf[:])
+		}
+	}
+}
diff --git a/repo/format/format_manager.go b/repo/format/format_manager.go
index 1b78ad6d..dcb86f63 100644
--- a/repo/format/format_manager.go
+++ b/repo/format/format_manager.go
@@ -9,6 +9,7 @@
 
 	"github.com/pkg/errors"
 
+	"github.com/kopia/kopia/internal/ctxutil"
 	"github.com/kopia/kopia/internal/feature"
 	"github.com/kopia/kopia/internal/gather"
 	"github.com/kopia/kopia/repo/blob"
@@ -401,7 +402,7 @@ func NewManagerWithCache(
 	}
 
 	m := &Manager{
-		ctx:                       ctx,
+		ctx:                       ctxutil.Detach(ctx),
 		blobs:                     st,
 		validDuration:             validDuration,
 		password:                  password,
diff --git a/snapshot/snapshotfs/objref.go b/snapshot/snapshotfs/objref.go
index 0203f4c3..6284074e 100644
--- a/snapshot/snapshotfs/objref.go
+++ b/snapshot/snapshotfs/objref.go
@@ -3,6 +3,7 @@
 import (
 	"context"
 	"encoding/json"
+	"path/filepath"
 	"strings"
 
 	"github.com/pkg/errors"
@@ -143,7 +144,7 @@ func latestManifest(mans []*snapshot.Manifest) *snapshot.Manifest {
 // If multiple snapshots match and they don't agree on root object attributes and consistentAttributes==true
 // the function fails, otherwise it returns the latest of the snapshots.
 func FilesystemEntryFromIDWithPath(ctx context.Context, rep repo.Repository, rootID string, consistentAttributes bool) (fs.Entry, error) {
-	pathElements := strings.Split(rootID, "/")
+	pathElements := strings.Split(filepath.ToSlash(rootID), "/")
 
 	if len(pathElements) > 1 {
 		// if a path is provided, consistentAttributes is meaningless since descending into nested path is
diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
index edca8a70..8fdc6170 100644
--- a/tools/docker/Dockerfile
+++ b/tools/docker/Dockerfile
@@ -23,7 +23,7 @@ RUN 		apt-get update && \
 	
 WORKDIR /app
 
-COPY 		bin-${TARGETARCH}/kopia .
+COPY 		bin-${TARGETARCH}/kopia /bin/kopia
 COPY 		bin-${TARGETARCH}/rclone /bin/rclone
   
-ENTRYPOINT 	["/app/kopia"]
+ENTRYPOINT 	["/bin/kopia"]
diff --git a/tools/tools.mk b/tools/tools.mk
index ecdb00a9..958df918 100644
--- a/tools/tools.mk
+++ b/tools/tools.mk
@@ -157,6 +157,7 @@ checklocks=$(checklocks_dir)$(slash)bin$(slash)checklocks$(exe_suffix)
 $(checklocks): export GOPATH=$(checklocks_dir)
 $(checklocks):
 	go install gvisor.dev/gvisor/tools/checklocks/cmd/checklocks@$(CHECKLOCKS_VERSION)
+	go clean -modcache
 
 # hugo
 hugo_dir=$(TOOLS_DIR)$(slash)hugo-$(HUGO_VERSION)


Test patch:
diff --git a/cli/command_restore_test.go b/cli/command_restore_test.go
new file mode 100644
index 00000000..4df5bc5c
--- /dev/null
+++ b/cli/command_restore_test.go
@@ -0,0 +1,62 @@
+package cli
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/require"
+
+	"github.com/kopia/kopia/internal/clock"
+)
+
+func TestRestoreSnapshotMaxTime(t *testing.T) {
+	t.Parallel()
+
+	now := clock.Now()
+	ago := func(y, m, d int) time.Time {
+		r := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, now.Location())
+		return r.AddDate(y, m, d)
+	}
+	at := func(y, mo, d, h, m, s int) time.Time {
+		return time.Date(y, time.Month(mo), d, h, m, s, 0, now.Location())
+	}
+
+	requireTime := func(expected time.Time, timespect string) {
+		mt, err := computeMaxTime(timespect)
+		require.NoError(t, err)
+		require.Equal(t, expected, mt)
+	}
+
+	requireTime(ago(0, 0, 0), "yesterday")
+	requireTime(ago(0, 0, 0), "1d-ago")
+	requireTime(ago(0, 0, 0), "1day-ago")
+
+	requireTime(at(now.Year(), int(now.Month()), 1, 0, 0, 0), "last-month")
+	requireTime(at(now.Year(), 1, 1, 0, 0, 0), "last-year")
+
+	requireTime(ago(0, -1, 1), "1month-ago")
+	requireTime(ago(0, -1, 1), "1m-ago")
+	requireTime(ago(-1, 0, 1), "1year-ago")
+	requireTime(ago(-1, 0, 1), "1y-ago")
+	requireTime(ago(-2, -2, -1), "2years-2months-2days-ago")
+	requireTime(ago(-2, -2, -1), "2y-2m-2d-ago")
+
+	requireTime(at(2020, 1, 1, 0, 0, 0), "2019")
+	requireTime(at(2019, 2, 1, 0, 0, 0), "2019-1")
+	requireTime(at(2019, 1, 2, 0, 0, 0), "2019-01-1")
+	requireTime(at(2019, 1, 1, 14, 0, 0), "2019-01-1 13")
+	requireTime(at(2019, 1, 1, 13, 2, 0), "2019-01-1 13:01")
+	requireTime(at(2019, 1, 1, 13, 1, 16), "2019-01-1 13:01:15")
+}
+
+func TestRestoreSnapshotFilter(t *testing.T) {
+	f, err := createSnapshotTimeFilter("latest")
+	require.NoError(t, err)
+	require.Equal(t, true, f(nil, 0, 2))
+	require.Equal(t, false, f(nil, 1, 2))
+
+	f, err = createSnapshotTimeFilter("oldest")
+	require.NoError(t, err)
+	require.Equal(t, false, f(nil, 0, 2))
+	require.Equal(t, true, f(nil, 1, 2))
+}
diff --git a/cli/command_snapshot_list_test.go b/cli/command_snapshot_list_test.go
index b482323c..8d906064 100644
--- a/cli/command_snapshot_list_test.go
+++ b/cli/command_snapshot_list_test.go
@@ -78,3 +78,47 @@ func TestSnapshotList(t *testing.T) {
 	require.Contains(t, lines[4], " 8 B ")
 	require.Contains(t, lines[4], " files:3 dirs:1 ")
 }
+
+func TestSnapshotListWithSameFileInMultipleSnapshots(t *testing.T) {
+	t.Parallel()
+
+	runner := testenv.NewInProcRunner(t)
+	e := testenv.NewCLITest(t, testenv.RepoFormatNotImportant, runner)
+
+	defer e.RunAndExpectSuccess(t, "repo", "disconnect")
+	e.RunAndExpectSuccess(t, "repo", "create", "filesystem", "--path", e.RepoDir)
+
+	srcdir := testutil.TempDirectory(t)
+
+	require.NoError(t, os.MkdirAll(filepath.Join(srcdir, "a", "b", "c", "d"), 0o755))
+	require.NoError(t, os.WriteFile(filepath.Join(srcdir, "a", "b", "c", "d", "e.txt"), []byte{1, 2, 3}, 0o755))
+
+	e.RunAndExpectSuccess(t, "snapshot", "create", srcdir)
+	e.RunAndExpectSuccess(t, "snapshot", "create", filepath.Join(srcdir, "a"))
+	e.RunAndExpectSuccess(t, "snapshot", "create", filepath.Join(srcdir, "a", "b"))
+	e.RunAndExpectSuccess(t, "snapshot", "create", filepath.Join(srcdir, "a", "b", "c"))
+	e.RunAndExpectSuccess(t, "snapshot", "create", filepath.Join(srcdir, "a", "b", "c", "d"))
+	e.RunAndExpectSuccess(t, "snapshot", "create", filepath.Join(srcdir, "a", "b", "c", "d", "e.txt"))
+
+	var snapshots []*cli.SnapshotManifest
+
+	testutil.MustParseJSONLines(t, e.RunAndExpectSuccess(t, "snapshot", "list",
+		filepath.Join(srcdir, "a", "b", "c", "d", "e.txt"), "--json"), &snapshots)
+
+	require.Len(t, snapshots, 6)
+
+	var sps []string
+
+	for _, s := range snapshots {
+		sps = append(sps, s.Source.Path)
+	}
+
+	require.Equal(t, []string{
+		srcdir,
+		filepath.Join(srcdir, "a"),
+		filepath.Join(srcdir, "a", "b"),
+		filepath.Join(srcdir, "a", "b", "c"),
+		filepath.Join(srcdir, "a", "b", "c", "d"),
+		filepath.Join(srcdir, "a", "b", "c", "d", "e.txt"),
+	}, sps)
+}
diff --git a/internal/bigmap/bigmap_internal_test.go b/internal/bigmap/bigmap_internal_test.go
new file mode 100644
index 00000000..6c3b868a
--- /dev/null
+++ b/internal/bigmap/bigmap_internal_test.go
@@ -0,0 +1,290 @@
+package bigmap
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"encoding/binary"
+	"hash"
+	"sync"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+
+	"github.com/kopia/kopia/internal/testlogging"
+)
+
+func TestInternalMap(t *testing.T) {
+	ctx := testlogging.Context(t)
+
+	impl, err := newInternalMap(ctx)
+	require.NoError(t, err)
+
+	defer impl.Close(ctx)
+
+	key1 := []byte("key1")
+	key2 := []byte("longerkey2")
+	val1 := []byte("val1")
+	val2 := []byte("val2")
+
+	v, ok := impl.Get(nil, key1)
+	require.Nil(t, v)
+	require.False(t, ok)
+
+	impl.PutIfAbsent(ctx, key1, val1)
+
+	v, ok = impl.Get(nil, key1)
+	require.True(t, ok)
+	require.Equal(t, val1, v)
+
+	v, ok = impl.Get(nil, key2)
+	require.Nil(t, v)
+	require.False(t, ok)
+
+	impl.PutIfAbsent(ctx, key2, val2)
+
+	v, ok = impl.Get(nil, key2)
+	require.True(t, ok)
+	require.Equal(t, val2, v)
+
+	v, ok = impl.Get(nil, key1)
+	require.True(t, ok)
+	require.Equal(t, val1, v)
+}
+
+func TestGrowingMap(t *testing.T) {
+	ctx := testlogging.Context(t)
+
+	impl, err := newInternalMapWithOptions(ctx, true, &Options{
+		InitialSizeLogarithm: 9,
+		NumMemorySegments:    3,
+		MemorySegmentSize:    1000,
+		FileSegmentSize:      4 << 20,
+	})
+	require.NoError(t, err)
+
+	defer impl.Close(ctx)
+
+	h := sha256.New()
+
+	// insert 20K hashes
+	for i := 0; i < 20000; i++ {
+		var keybuf, valbuf, valbuf2 [sha256.Size]byte
+
+		k := sha256Key(h, keybuf[:0], i)
+		v := sha256Key(h, valbuf[:0], i+3)
+
+		require.True(t, impl.PutIfAbsent(ctx, k, v))
+
+		// ensure that previously written key is still there.
+		pkindex := i / 2
+		pk := sha256Key(h, keybuf[:0], pkindex)
+		require.True(t, impl.Contains(pk))
+
+		pv, ok := impl.Get(valbuf2[:0], pk)
+		require.True(t, ok)
+		require.Equal(t, pv, sha256Key(h, valbuf[:0], pkindex+3))
+
+		// ensure that key not written yet is not there.
+		nk := sha256Key(h, keybuf[:0], i+1)
+
+		require.False(t, impl.Contains(nk))
+
+		_, ok2 := impl.Get(valbuf2[:0], nk)
+		require.False(t, ok2)
+	}
+}
+
+func TestGrowingSet(t *testing.T) {
+	ctx := testlogging.Context(t)
+
+	impl, err := NewSetWithOptions(ctx, &Options{
+		InitialSizeLogarithm: 9,
+		NumMemorySegments:    3,
+		MemorySegmentSize:    1000,
+		FileSegmentSize:      4 << 20,
+	})
+	require.NoError(t, err)
+
+	defer impl.Close(ctx)
+
+	h := sha256.New()
+
+	// insert 20K hashes
+	for i := 0; i < 20000; i++ {
+		var keybuf [sha256.Size]byte
+
+		k := sha256Key(h, keybuf[:0], i)
+
+		require.True(t, impl.Put(ctx, k))
+		require.False(t, impl.Put(ctx, k))
+
+		// ensure that previously written key is still there.
+		pkindex := i / 2
+		pk := sha256Key(h, keybuf[:0], pkindex)
+		require.True(t, impl.Contains(pk))
+
+		// ensure that key not written yet is not there.
+		nk := sha256Key(h, keybuf[:0], i+1)
+
+		require.False(t, impl.Contains(nk))
+	}
+}
+
+func sha256Key(h hash.Hash, out []byte, i int) []byte {
+	var num [8]byte
+
+	// generate key=sha256(i) without allocations.
+	h.Reset()
+	binary.LittleEndian.PutUint64(num[:], uint64(i))
+	h.Write(num[:])
+
+	s := h.Sum(out)
+
+	return s
+}
+
+func BenchmarkInternalMap_NoValue(b *testing.B) {
+	ctx := testlogging.Context(b)
+	m, err := newInternalMapWithOptions(ctx, false, nil)
+	require.NoError(b, err)
+
+	defer m.Close(ctx)
+
+	benchmarkInternalMap(b, m, []byte{})
+}
+
+func BenchmarkInternalMap_WithValue(b *testing.B) {
+	ctx := testlogging.Context(b)
+	m, err := newInternalMap(ctx)
+	require.NoError(b, err)
+
+	defer m.Close(ctx)
+
+	benchmarkInternalMap(b, m, []byte{1, 2, 3})
+}
+
+//nolint:thelper
+func benchmarkInternalMap(b *testing.B, m *internalMap, someVal []byte) {
+	ctx := testlogging.Context(b)
+
+	b.ResetTimer()
+
+	var (
+		h      = sha256.New()
+		num    [8]byte
+		keyBuf [sha256.Size]byte
+	)
+
+	for i := 0; i < b.N; i++ {
+		// generate key=sha256(i) without allocations.
+		h.Reset()
+		binary.LittleEndian.PutUint64(num[:], uint64(i))
+		h.Write(num[:])
+		key := h.Sum(keyBuf[:0])
+
+		m.PutIfAbsent(ctx, key, someVal)
+	}
+
+	valBuf := make([]byte, 10)
+
+	for j := 0; j < 4; j++ {
+		for i := 0; i < b.N; i++ {
+			// generate key=sha256(i) without allocations.
+			h.Reset()
+			binary.LittleEndian.PutUint64(num[:], uint64(i))
+			h.Write(num[:])
+			key := h.Sum(keyBuf[:0])
+
+			_, ok := m.Get(valBuf[:0], key)
+			require.True(b, ok)
+		}
+	}
+}
+
+func BenchmarkSyncMap_NoValue(b *testing.B) {
+	benchmarkSyncMap(b, []byte{})
+}
+
+func BenchmarkSyncMap_WithValue(b *testing.B) {
+	someVal := []byte{1, 2, 3}
+
+	benchmarkSyncMap(b, someVal)
+}
+
+//nolint:thelper
+func benchmarkSyncMap(b *testing.B, someVal []byte) {
+	var m sync.Map
+
+	var (
+		h      = sha256.New()
+		num    [8]byte
+		keyBuf [sha256.Size]byte
+	)
+
+	b.ResetTimer()
+
+	for i := 0; i < b.N; i++ {
+		// generate key=sha256(i) without allocations.
+		h.Reset()
+		binary.LittleEndian.PutUint64(num[:], uint64(i))
+		h.Write(num[:])
+		key := h.Sum(keyBuf[:0])
+
+		m.Store(string(key), append([]byte{}, someVal...))
+	}
+
+	for j := 0; j < 4; j++ {
+		for i := 0; i < b.N; i++ {
+			// generate key=sha256(i) without allocations.
+			h.Reset()
+			binary.LittleEndian.PutUint64(num[:], uint64(i))
+			h.Write(num[:])
+			key := h.Sum(keyBuf[:0])
+
+			val, ok := m.Load(string(key))
+			require.True(b, ok)
+			require.Equal(b, someVal, val)
+		}
+	}
+}
+
+func TestErrors(t *testing.T) {
+	ctx := testlogging.Context(t)
+
+	_, err := newInternalMapWithOptions(ctx, true, &Options{
+		InitialSizeLogarithm: 8,
+	})
+
+	require.ErrorContains(t, err, "invalid initial size")
+}
+
+func TestPanics(t *testing.T) {
+	ctx := testlogging.Context(t)
+
+	m, err := NewSet(ctx)
+	require.NoError(t, err)
+
+	// too short keys
+	require.Panics(t, func() { m.Put(ctx, nil) })
+	require.Panics(t, func() { m.Put(ctx, []byte{1}) })
+
+	// too long key
+	require.Panics(t, func() { m.Put(ctx, bytes.Repeat([]byte{1}, 256)) })
+}
+
+func TestMapWithoutValue(t *testing.T) {
+	ctx := testlogging.Context(t)
+
+	// this is a corner case, it's possible to create a map that can only support zero-length values.
+	m, err := newInternalMapWithOptions(ctx, false, nil)
+	require.NoError(t, err)
+
+	key := []byte{1, 2, 3, 4}
+	m.PutIfAbsent(ctx, key, nil)
+
+	require.Panics(t, func() { m.PutIfAbsent(ctx, key, []byte{3, 4, 5}) })
+
+	v, ok := m.Get(nil, key)
+	require.True(t, ok)
+	require.Nil(t, v)
+}
diff --git a/internal/bigmap/bigmap_map_test.go b/internal/bigmap/bigmap_map_test.go
new file mode 100644
index 00000000..b72d7bd5
--- /dev/null
+++ b/internal/bigmap/bigmap_map_test.go
@@ -0,0 +1,130 @@
+package bigmap_test
+
+import (
+	"crypto/sha256"
+	"encoding/binary"
+	"hash"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+
+	"github.com/kopia/kopia/internal/bigmap"
+	"github.com/kopia/kopia/internal/testlogging"
+)
+
+func TestGrowingMap(t *testing.T) {
+	ctx := testlogging.Context(t)
+
+	impl, err := bigmap.NewMapWithOptions(ctx, &bigmap.Options{
+		InitialSizeLogarithm: 9,
+		NumMemorySegments:    3,
+		MemorySegmentSize:    1000,
+		FileSegmentSize:      4 << 20,
+	})
+	require.NoError(t, err)
+
+	defer impl.Close(ctx)
+
+	h := sha256.New()
+
+	// insert 20K hashes
+	for i := 0; i < 20000; i++ {
+		var keybuf, valbuf, valbuf2 [sha256.Size]byte
+
+		k := sha256Key(h, keybuf[:0], i)
+		v := sha256Key(h, valbuf[:0], i+3)
+
+		require.True(t, impl.PutIfAbsent(ctx, k, v))
+
+		// ensure that previously written key is still there.
+		pkindex := i / 2
+		pk := sha256Key(h, keybuf[:0], pkindex)
+		require.True(t, impl.Contains(pk))
+
+		pv, ok, err := impl.Get(ctx, valbuf2[:0], pk)
+		require.NoError(t, err)
+		require.True(t, ok)
+		require.Equal(t, pv, sha256Key(h, valbuf[:0], pkindex+3))
+
+		// ensure that key not written yet is not there.
+		nk := sha256Key(h, keybuf[:0], i+1)
+
+		require.False(t, impl.Contains(nk))
+
+		_, ok2, err := impl.Get(ctx, valbuf2[:0], nk)
+		require.NoError(t, err)
+		require.False(t, ok2)
+	}
+}
+
+func sha256Key(h hash.Hash, out []byte, i int) []byte {
+	var num [8]byte
+
+	// generate key=sha256(i) without allocations.
+	h.Reset()
+	binary.LittleEndian.PutUint64(num[:], uint64(i))
+	h.Write(num[:])
+
+	s := h.Sum(out)
+
+	return s
+}
+
+func BenchmarkMap_NoValue(b *testing.B) {
+	ctx := testlogging.Context(b)
+	m, err := bigmap.NewMapWithOptions(ctx, nil)
+	require.NoError(b, err)
+
+	defer m.Close(ctx)
+
+	benchmarkMap(b, m, []byte{})
+}
+
+func BenchmarkMap_WithValue(b *testing.B) {
+	ctx := testlogging.Context(b)
+	m, err := bigmap.NewMap(ctx)
+	require.NoError(b, err)
+
+	defer m.Close(ctx)
+
+	benchmarkMap(b, m, []byte{1, 2, 3})
+}
+
+//nolint:thelper
+func benchmarkMap(b *testing.B, m *bigmap.Map, someVal []byte) {
+	ctx := testlogging.Context(b)
+
+	b.ResetTimer()
+
+	var (
+		h      = sha256.New()
+		num    [8]byte
+		keyBuf [sha256.Size]byte
+	)
+
+	for i := 0; i < b.N; i++ {
+		// generate key=sha256(i) without allocations.
+		h.Reset()
+		binary.LittleEndian.PutUint64(num[:], uint64(i))
+		h.Write(num[:])
+		key := h.Sum(keyBuf[:0])
+
+		m.PutIfAbsent(ctx, key, someVal)
+	}
+
+	valBuf := make([]byte, 10)
+
+	for j := 0; j < 4; j++ {
+		for i := 0; i < b.N; i++ {
+			// generate key=sha256(i) without allocations.
+			h.Reset()
+			binary.LittleEndian.PutUint64(num[:], uint64(i))
+			h.Write(num[:])
+			key := h.Sum(keyBuf[:0])
+
+			_, ok, err := m.Get(ctx, valBuf[:0], key)
+			require.NoError(b, err)
+			require.True(b, ok)
+		}
+	}
+}
diff --git a/internal/bigmap/bigmap_set_test.go b/internal/bigmap/bigmap_set_test.go
new file mode 100644
index 00000000..12bada23
--- /dev/null
+++ b/internal/bigmap/bigmap_set_test.go
@@ -0,0 +1,101 @@
+package bigmap_test
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"encoding/binary"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+
+	"github.com/kopia/kopia/internal/bigmap"
+	"github.com/kopia/kopia/internal/testlogging"
+)
+
+func TestGrowingSet(t *testing.T) {
+	ctx := testlogging.Context(t)
+
+	impl, err := bigmap.NewSetWithOptions(ctx, &bigmap.Options{
+		InitialSizeLogarithm: 9,
+		NumMemorySegments:    3,
+		MemorySegmentSize:    1000,
+		FileSegmentSize:      4 << 20,
+	})
+	require.NoError(t, err)
+
+	defer impl.Close(ctx)
+
+	h := sha256.New()
+
+	// insert 20K hashes
+	for i := 0; i < 20000; i++ {
+		var keybuf [sha256.Size]byte
+
+		k := sha256Key(h, keybuf[:0], i)
+
+		require.True(t, impl.Put(ctx, k))
+		require.False(t, impl.Put(ctx, k))
+
+		// ensure that previously written key is still there.
+		pkindex := i / 2
+		pk := sha256Key(h, keybuf[:0], pkindex)
+		require.True(t, impl.Contains(pk))
+
+		// ensure that key not written yet is not there.
+		nk := sha256Key(h, keybuf[:0], i+1)
+
+		require.False(t, impl.Contains(nk))
+	}
+}
+
+func BenchmarkSet(b *testing.B) {
+	ctx := testlogging.Context(b)
+	m, err := bigmap.NewSet(ctx)
+	require.NoError(b, err)
+
+	defer m.Close(ctx)
+
+	b.ResetTimer()
+
+	var (
+		h      = sha256.New()
+		num    [8]byte
+		keyBuf [sha256.Size]byte
+	)
+
+	for i := 0; i < b.N; i++ {
+		// generate key=sha256(i) without allocations.
+		h.Reset()
+		binary.LittleEndian.PutUint64(num[:], uint64(i))
+		h.Write(num[:])
+		key := h.Sum(keyBuf[:0])
+
+		m.Put(ctx, key)
+	}
+
+	for j := 0; j < 4; j++ {
+		for i := 0; i < b.N; i++ {
+			// generate key=sha256(i) without allocations.
+			h.Reset()
+			binary.LittleEndian.PutUint64(num[:], uint64(i))
+			h.Write(num[:])
+			key := h.Sum(keyBuf[:0])
+
+			require.True(b, m.Contains(key))
+		}
+	}
+}
+
+func TestSetPanics(t *testing.T) {
+	ctx := testlogging.Context(t)
+
+	m, err := bigmap.NewSet(ctx)
+	require.NoError(t, err)
+
+	// too short keys
+	require.Panics(t, func() { m.Put(ctx, nil) })
+	require.Panics(t, func() { m.Put(ctx, []byte{1}) })
+
+	// too long key
+	require.Panics(t, func() { m.Put(ctx, bytes.Repeat([]byte{1}, 256)) })
+}
diff --git a/snapshot/snapshotfs/upload_test.go b/snapshot/snapshotfs/upload_test.go
index 28bc4eb3..5fb82244 100644
--- a/snapshot/snapshotfs/upload_test.go
+++ b/snapshot/snapshotfs/upload_test.go
@@ -914,6 +914,9 @@ func TestUpload_VirtualDirectoryWithStreamingFileWithModTime(t *testing.T) {
 				tc.getFile(),
 			})
 
+			// wait a little bit to ensure clock moves forward which is not always the case on Windows.
+			time.Sleep(100 * time.Millisecond)
+
 			// Second upload may find some cached files depending on timestamps.
 			man2, err := u.Upload(ctx, staticRoot, policyTree, snapshot.SourceInfo{}, man1)
 			require.NoError(t, err)
diff --git a/tests/end_to_end_test/restore_test.go b/tests/end_to_end_test/restore_test.go
index aad17e85..c4c00eed 100644
--- a/tests/end_to_end_test/restore_test.go
+++ b/tests/end_to_end_test/restore_test.go
@@ -827,3 +827,65 @@ func verifyValidTarGzipFile(t *testing.T, fname string) {
 
 	verifyValidTarReader(t, tar.NewReader(gz))
 }
+
+func TestSnapshotRestoreByPath(t *testing.T) {
+	t.Parallel()
+
+	runner := testenv.NewInProcRunner(t)
+	e := testenv.NewCLITest(t, testenv.RepoFormatNotImportant, runner)
+
+	defer e.RunAndExpectSuccess(t, "repo", "disconnect")
+
+	e.RunAndExpectSuccess(t, "repo", "create", "filesystem", "--path", e.RepoDir)
+
+	source := testutil.TempDirectory(t)
+
+	// create a file with well-known name.
+	f, err := os.Create(filepath.Join(source, "single-file"))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	fmt.Fprintf(f, "some-data")
+	f.Close()
+
+	// Create snapshot
+	e.RunAndExpectSuccess(t, "snapshot", "create", source)
+
+	// Restore based on source path
+	restoreDir := testutil.TempDirectory(t)
+	e.RunAndExpectSuccess(t, "snapshot", "restore", source, restoreDir, "--snapshot-time=latest")
+
+	// Restored contents should match source
+	compareDirs(t, source, restoreDir)
+}
+
+func TestRestoreByPathWithoutTarget(t *testing.T) {
+	t.Parallel()
+
+	runner := testenv.NewInProcRunner(t)
+	e := testenv.NewCLITest(t, testenv.RepoFormatNotImportant, runner)
+
+	defer e.RunAndExpectSuccess(t, "repo", "disconnect")
+	e.RunAndExpectSuccess(t, "repo", "create", "filesystem", "--path", e.RepoDir)
+
+	srcdir := testutil.TempDirectory(t)
+	file := filepath.Join(srcdir, "a.txt")
+	originalData := []byte{1, 2, 3}
+
+	require.NoError(t, os.WriteFile(file, originalData, 0o755))
+
+	e.RunAndExpectSuccess(t, "snapshot", "create", srcdir)
+
+	require.NoError(t, os.WriteFile(file, []byte{4, 5, 6, 7}, 0o755))
+
+	e.RunAndExpectSuccess(t, "restore", srcdir, "--snapshot-time=latest")
+
+	data, err := os.ReadFile(file)
+
+	require.NoError(t, err)
+	require.Equal(t, originalData, data)
+
+	// Must pass snapshot time
+	e.RunAndExpectFailure(t, "restore", srcdir)
+}
diff --git a/tests/htmlui_e2e_test/htmlui_e2e_test.go b/tests/htmlui_e2e_test/htmlui_e2e_test.go
index f2846318..5b6e17d3 100644
--- a/tests/htmlui_e2e_test/htmlui_e2e_test.go
+++ b/tests/htmlui_e2e_test/htmlui_e2e_test.go
@@ -42,7 +42,7 @@ func runInBrowser(t *testing.T, run func(ctx context.Context, sp *testutil.Serve
 
 	var sp testutil.ServerParameters
 
-	_, kill := e.RunAndProcessStderr(t, sp.ProcessOutput,
+	args := []string{
 		"server", "start",
 		"--ui",
 		"--address=localhost:0",
@@ -54,8 +54,14 @@ func runInBrowser(t *testing.T, run func(ctx context.Context, sp *testutil.Serve
 		"--without-password",
 		"--override-hostname=the-hostname",
 		"--override-username=the-username",
-		// "--html="+os.Getenv("HTMLUI_BUILD_DIR"),
-	)
+	}
+
+	if h := os.Getenv("HTMLUI_BUILD_DIR"); h != "" {
+		args = append(args, "--html="+os.Getenv("HTMLUI_BUILD_DIR"))
+	}
+
+	_, kill := e.RunAndProcessStderr(t, sp.ProcessOutput, args...)
+
 	defer kill()
 
 	t.Logf("detected server parameters %#v", sp)
diff --git a/tests/robustness/snapmeta/kopia_persister_light_test.go b/tests/robustness/snapmeta/kopia_persister_light_test.go
index b11eeb76..155f0619 100644
--- a/tests/robustness/snapmeta/kopia_persister_light_test.go
+++ b/tests/robustness/snapmeta/kopia_persister_light_test.go
@@ -131,8 +131,6 @@ func TestPersistence(t *testing.T) {
 }
 
 func TestS3Connect(t *testing.T) {
-	t.Skip("TODO - fix me")
-
 	repoPath, err := os.MkdirTemp("", "kopia-test-repo-")
 	assertNoError(t, err)