Repository: kopia/kopia
Pull request #2561 (2022-11-08 23:55:32)

feat(repository): new `repository upgrade` sub-command to validate indices.
This PR adds functionality to validate indices and replaces #2528.

The full feature will add a validation phase to the current V0 to V1 index upgrade command.  This phase will be run before the `commitUpgrade` phase.  If validation fails, the tool will fail with an error indicating the first failed blob(s), leaving the repository locked.  The user may then either rollback and re-run the upgrade, or continue the upgrade (by re-executing `repo upgrade begin` with the `--commit-mode=always` option).  The user may also choose to change the index blob(s) using a Kopia blob-editor tool and then re-validate the index (to check the work) using `repo upgrade validate`.

Re-running `repo upgrade begin` command with the `--commit-mode=always` flag will commit the existing v1 index, delete the v0 index backup, and unlock the repository.

upgrade validate examples:
```
$ KOPIA_UPGRADE_LOCK_ENABLED=1 kopia repo upgrade --upgrade-owner-id jdoe@example.com validate

upgraded index matches old index
```

commit mode "never" example:
```
$ KOPIA_UPGRADE_LOCK_ENABLED=1 kopia repo upgrade --upgrade-owner-id jdoe@example.com begin --commit-mode=never

deleting /Users/aalpar/Library/Caches/kopia/9ba004d5e92006b9/kopia.repository
Repository upgrade lock intent has been placed.
.
.
.
migrating current indices to epoch format
deleting /Users/aalpar/Library/Caches/kopia/9ba004d5e92006b9/kopia.repository
deleting /Users/aalpar/Library/Caches/kopia/9ba004d5e92006b9/kopia.blobcfg
Repository indices have been upgraded.
upgraded index matches old index
WARN Commit mode is set to 'never'.  Skipping commit.

$ KOPIA_UPGRADE_LOCK_ENABLED=1 kopia repo status --upgrade-owner-id jdoe@example.com

Config file:         /Users/jdoe/Library/Application Support/kopia/repository.config
.
.
.
Max pack length:     21 MB
Index Format:        v2

Epoch Manager:       enabled
Current Epoch: 0
.
.
.
Ongoing upgrade:     Upgrading from format version 1 -> 3
Upgrade Time:        2022-11-09 09:56:27.861252 -0800 PST
Upgrade lock:        Locked
Lock status:         Fully Established
```

failed validation and commit mode "always" example:
```
$ KOPIA_UPGRADE_LOCK_ENABLED=1 kopia repo upgrade --upgrade-owner-id jdoe@example.com begin


deleting /Users/aalpar/Library/Caches/kopia/9ba004d5e92006b9/kopia.repository
Repository upgrade lock intent has been placed.
.
.
.
ERROR failed to load index entries for v1 index entry: failed to load index blob with BlobID xn0_081213a775af0bae06d4d81f4f5f0666-sb28e5d344128a4ed-c1: unable to decrypt index blob: error decrypting BLOB xn0_081213a775af0bae06d4d81f4f5f0666-sb28e5d344128a4ed-c1: unable to decrypt content: cipher: message authentication failed

$ KOPIA_UPGRADE_LOCK_ENABLED=1 kopia repo upgrade --upgrade-owner-id jdoe@example.com validate

ERROR failed to load index entries for v1 index entry: failed to load index blob with BlobID xn0_081213a775af0bae06d4d81f4f5f0666-sb28e5d344128a4ed-c1: unable to decrypt index blob: error decrypting BLOB xn0_081213a775af0bae06d4d81f4f5f0666-sb28e5d344128a4ed-c1: unable to decrypt content: cipher: message authentication failed

```

codecov[bot]: # [Codecov](https://codecov.io/gh/kopia/kopia/pull/2561?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) Report
Base: **74.89**% // Head: **74.84**% // Decreases project coverage by **`-0.05%`** :warning:
> Coverage data is based on head [(`6c10fda`)](https://codecov.io/gh/kopia/kopia/pull/2561?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) compared to base [(`a7d7418`)](https://codecov.io/gh/kopia/kopia/commit/a7d7418111885aaf4ece163497b1e3ee682521f0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia).
> Patch coverage: 60.81% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #2561      +/-   ##
==========================================
- Coverage   74.89%   74.84%   -0.06%     
==========================================
  Files         452      452              
  Lines       35477    35608     +131     
==========================================
+ Hits        26572    26650      +78     
- Misses       7022     7064      +42     
- Partials     1883     1894      +11     
```


| [Impacted Files](https://codecov.io/gh/kopia/kopia/pull/2561?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) | Coverage Δ | |
|---|---|---|
| [cli/command\_repository\_set\_parameters.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-Y2xpL2NvbW1hbmRfcmVwb3NpdG9yeV9zZXRfcGFyYW1ldGVycy5nbw==) | `80.98% <44.44%> (-1.30%)` | :arrow_down: |
| [cli/command\_repository\_upgrade.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-Y2xpL2NvbW1hbmRfcmVwb3NpdG9yeV91cGdyYWRlLmdv) | `65.10% <58.41%> (-2.23%)` | :arrow_down: |
| [repo/content/index\_blob\_manager\_v0.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9jb250ZW50L2luZGV4X2Jsb2JfbWFuYWdlcl92MC5nbw==) | `70.58% <64.28%> (-0.74%)` | :arrow_down: |
| [repo/content/committed\_read\_manager.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9jb250ZW50L2NvbW1pdHRlZF9yZWFkX21hbmFnZXIuZ28=) | `71.46% <70.00%> (-0.04%)` | :arrow_down: |
| [repo/format/upgrade\_lock.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9mb3JtYXQvdXBncmFkZV9sb2NrLmdv) | `75.89% <70.00%> (-0.58%)` | :arrow_down: |
| [cli/command\_index\_inspect.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-Y2xpL2NvbW1hbmRfaW5kZXhfaW5zcGVjdC5nbw==) | `80.55% <100.00%> (ø)` | |
| [repo/content/content\_manager\_indexes.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9jb250ZW50L2NvbnRlbnRfbWFuYWdlcl9pbmRleGVzLmdv) | `78.87% <100.00%> (+4.22%)` | :arrow_up: |
| [repo/content/index\_blob\_manager\_v1.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9jb250ZW50L2luZGV4X2Jsb2JfbWFuYWdlcl92MS5nbw==) | `64.21% <100.00%> (+0.76%)` | :arrow_up: |
| [internal/dirutil/mksubdirall.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvZGlydXRpbC9ta3N1YmRpcmFsbC5nbw==) | `92.10% <0.00%> (-7.90%)` | :arrow_down: |
| [internal/cache/persistent\_lru\_cache.go](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvY2FjaGUvcGVyc2lzdGVudF9scnVfY2FjaGUuZ28=) | `96.06% <0.00%> (-1.75%)` | :arrow_down: |
| ... and [7 more](https://codecov.io/gh/kopia/kopia/pull/2561/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/kopia/kopia/pull/2561?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia).

Shrekster: Thanks for taking in the feedbacks. I'll take a final pass on this today as soon as I get out of meetings.

Patch:
diff --git a/cli/command_index_inspect.go b/cli/command_index_inspect.go
index d878b827..cc14884f 100644
--- a/cli/command_index_inspect.go
+++ b/cli/command_index_inspect.go
@@ -162,7 +162,7 @@ func (c *commandIndexInspect) inspectSingleIndexBlob(ctx context.Context, rep re
 		return errors.Wrapf(err, "unable to get data for %v", blobID)
 	}
 
-	entries, err := content.ParseIndexBlob(ctx, blobID, data.Bytes(), rep.ContentReader().ContentFormat())
+	entries, err := content.ParseIndexBlob(blobID, data.Bytes(), rep.ContentReader().ContentFormat())
 	if err != nil {
 		return errors.Wrapf(err, "unable to recover index from %v", blobID)
 	}
diff --git a/cli/command_repository_set_parameters.go b/cli/command_repository_set_parameters.go
index 30cc0df0..bfa161c6 100644
--- a/cli/command_repository_set_parameters.go
+++ b/cli/command_repository_set_parameters.go
@@ -11,7 +11,6 @@
 	"github.com/kopia/kopia/internal/units"
 	"github.com/kopia/kopia/repo"
 	"github.com/kopia/kopia/repo/blob"
-	"github.com/kopia/kopia/repo/content"
 	"github.com/kopia/kopia/repo/format"
 )
 
@@ -143,7 +142,7 @@ func updateRepositoryParameters(
 	}
 
 	if upgradeToEpochManager {
-		if err := content.WriteLegacyIndexPoisonBlob(ctx, rep.BlobStorage()); err != nil {
+		if err := format.WriteLegacyIndexPoisonBlob(ctx, rep.BlobStorage()); err != nil {
 			log(ctx).Errorf("unable to write legacy index poison blob: %v", err)
 		}
 	}
diff --git a/cli/command_repository_upgrade.go b/cli/command_repository_upgrade.go
index 1e9235f8..ce67fb0d 100644
--- a/cli/command_repository_upgrade.go
+++ b/cli/command_repository_upgrade.go
@@ -10,16 +10,19 @@
 	"github.com/pkg/errors"
 
 	"github.com/kopia/kopia/internal/epoch"
+	"github.com/kopia/kopia/internal/gather"
 	"github.com/kopia/kopia/repo"
 	"github.com/kopia/kopia/repo/content"
+	"github.com/kopia/kopia/repo/content/index"
 	"github.com/kopia/kopia/repo/format"
 )
 
 type commandRepositoryUpgrade struct {
-	forceRollback bool
-	skip          bool
-	force         bool
-	lockOnly      bool
+	forceRollback             bool
+	skip                      bool
+	allowUnsafeUpgradeTimings bool
+	commitMode                string
+	lockOnly                  bool
 
 	// lock settings
 	ioDrainTimeout         time.Duration
@@ -38,6 +41,11 @@ type commandRepositoryUpgrade struct {
 	maxPermittedClockDriftDefault = 5 * time.Minute
 )
 
+const (
+	commitModeAlwaysCommit = "always"
+	commitModeNeverCommit  = "never"
+)
+
 func (c *commandRepositoryUpgrade) setup(svc advancedAppServices, parent commandParent) {
 	// override the parent, the upgrade sub-command becomes the new parent here-onwards
 	parent = parent.Command("upgrade", fmt.Sprintf("Upgrade repository format.\n\n%s", warningColor.Sprint(experimentalWarning))).Hidden().
@@ -50,10 +58,11 @@ func (c *commandRepositoryUpgrade) setup(svc advancedAppServices, parent command
 
 	beginCmd := parent.Command("begin", "Begin upgrade.").Default()
 	beginCmd.Flag("io-drain-timeout", "Max time it should take all other Kopia clients to drop repository connections").Default(format.DefaultRepositoryBlobCacheDuration.String()).DurationVar(&c.ioDrainTimeout)
-	beginCmd.Flag("allow-unsafe-upgrade", "Force using an unsafe io-drain-timeout for the upgrade lock").Default("false").Hidden().BoolVar(&c.force)
+	beginCmd.Flag("allow-unsafe-upgrade", "Force using an unsafe io-drain-timeout for the upgrade lock").Default("false").Hidden().BoolVar(&c.allowUnsafeUpgradeTimings)
 	beginCmd.Flag("status-poll-interval", "An advisory polling interval to check for the status of upgrade").Default("60s").DurationVar(&c.statusPollInterval)
 	beginCmd.Flag("max-permitted-clock-drift", "The maximum drift between repository and client clocks").Default(maxPermittedClockDriftDefault.String()).DurationVar(&c.maxPermittedClockDrift)
 	beginCmd.Flag("lock-only", "Advertise the upgrade lock and exit without actually performing the drain or upgrade").Default("false").Hidden().BoolVar(&c.lockOnly) // this is used by tests
+	beginCmd.Flag("commit-mode", "Change behavior of commit. When not set, commit on validation success. 'always': always commit. 'never': always exit before commit.").Hidden().EnumVar(&c.commitMode, commitModeAlwaysCommit, commitModeNeverCommit)
 
 	// upgrade phases
 
@@ -63,6 +72,8 @@ func (c *commandRepositoryUpgrade) setup(svc advancedAppServices, parent command
 	beginCmd.Action(svc.directRepositoryWriteAction(c.runPhase(c.drainOrCommit)))
 	// If the lock is fully established then perform the upgrade.
 	beginCmd.Action(svc.directRepositoryWriteAction(c.runPhase(c.upgrade)))
+	// Validate index upgrade success
+	beginCmd.Action(svc.directRepositoryWriteAction(c.runPhase(c.ignoreErrorOnAlwaysCommit(c.validateAction))))
 	// Commit the upgrade and revoke the lock, this will also cleanup any
 	// backups used for rollback.
 	beginCmd.Action(svc.directRepositoryWriteAction(c.runPhase(c.commitUpgrade)))
@@ -72,9 +83,151 @@ func (c *commandRepositoryUpgrade) setup(svc advancedAppServices, parent command
 
 	rollbackCmd.Action(svc.directRepositoryWriteAction(c.forceRollbackAction))
 
+	validateCmd := parent.Command("validate", "Validate the upgraded indexes.")
+
+	validateCmd.Action(svc.directRepositoryWriteAction(c.validateAction))
+
 	c.svc = svc
 }
 
+// assign store the info struct in a map that can be used to compare indexes.
+func assign(iif content.Info, i int, m map[content.ID][2]index.Info) {
+	v := m[iif.GetContentID()]
+	v[i] = iif
+	m[iif.GetContentID()] = v
+}
+
+// loadIndexBlobs load index blobs into indexEntries map.  indexEntries map will allow comparison betweel two indexes (index at which == 0 and index at which == 1).
+func loadIndexBlobs(ctx context.Context, indexEntries map[content.ID][2]index.Info, sm *content.SharedManager, which int, indexBlobInfos []content.IndexBlobInfo) error {
+	d := gather.WriteBuffer{}
+
+	for _, indexBlobInfo := range indexBlobInfos {
+		blobID := indexBlobInfo.BlobID
+
+		indexInfos, err := sm.LoadIndexBlob(ctx, blobID, &d)
+		if err != nil {
+			return errors.Wrapf(err, "failed to load index blob with BlobID %s", blobID)
+		}
+
+		for _, indexInfo := range indexInfos {
+			assign(indexInfo, which, indexEntries)
+		}
+	}
+
+	return nil
+}
+
+// validateAction returns an error if the new V1 index blob content does not match the source V0 index blob content.
+// This is used to check that the upgraded index (V1 index) reflects the content of the old V0 index.
+func (c *commandRepositoryUpgrade) validateAction(ctx context.Context, rep repo.DirectRepositoryWriter) error {
+	indexEntries := map[content.ID][2]index.Info{}
+
+	sm := rep.ContentManager().SharedManager
+
+	indexBlobInfos0, _, err := sm.IndexReaderV0().ListIndexBlobInfos(ctx)
+	if err != nil {
+		return errors.Wrapf(err, "failed to list index blobs for old index")
+	}
+
+	indexBlobInfos1, _, err := sm.IndexReaderV1().ListIndexBlobInfos(ctx)
+	if err != nil {
+		log(ctx).Errorf("failed to list index blobs for new index. upgrade may have failed.: %v", err)
+		return nil
+	}
+
+	if len(indexBlobInfos0) == 0 && len(indexBlobInfos1) > 0 {
+		log(ctx).Infof("old index is empty (possibly due to upgrade), nothing to compare against")
+		return nil
+	}
+
+	// load index blobs into their appropriate positions inside the indexEntries map
+
+	err = loadIndexBlobs(ctx, indexEntries, sm, 0, indexBlobInfos0)
+	if err != nil {
+		return errors.Wrapf(err, "failed to load index entries for v0 index entry")
+	}
+
+	err = loadIndexBlobs(ctx, indexEntries, sm, 1, indexBlobInfos1)
+	if err != nil {
+		return errors.Wrapf(err, "failed to load index entries for new index")
+	}
+
+	var msgs []string // a place to keep messages from the index comparison process
+
+	// both indexes will have matching contentiDs with matching indexInfo sructures.
+	for contentID, indexEntryPairs := range indexEntries {
+		iep0 := indexEntryPairs[0] // first entry of index entry pair
+		iep1 := indexEntryPairs[1] // second entry of index entry pair
+
+		// check that both the new and old indexes have entries for the same content
+		if iep0 != nil && iep1 != nil {
+			// this is the happy-path, check the entries.  any problems found will be added to msgs
+			msgs = append(msgs, CheckIndexInfo(iep0, iep1)...)
+			continue
+		}
+
+		// one of iep0 or iep1 are nil .. find out which one and add an appropriate message.
+		if iep0 != nil {
+			msgs = append(msgs, fmt.Sprintf("lop-sided index entries for contentID %q at blob %q", contentID, iep0.GetPackBlobID()))
+			continue
+		}
+
+		msgs = append(msgs, fmt.Sprintf("lop-sided index entries for contentID %q at blob %q", contentID, iep1.GetPackBlobID()))
+	}
+
+	// no msgs means the check passed without finding anything wrong
+	if len(msgs) == 0 {
+		log(ctx).Infof("index validation succeeded")
+		return nil
+	}
+
+	// otherwise there's a problem somewhere ... log the problems
+	log(ctx).Error("inconsistencies found in migrated index:")
+
+	for _, m := range msgs {
+		log(ctx).Error(m)
+	}
+
+	// and return an error that states something's wrong.
+	return errors.Wrap(err, "repository will remain locked until index differences are resolved")
+}
+
+// CheckIndexInfo compare two index infos.  If a mismatch exists, return an error with diagnostic information.
+func CheckIndexInfo(i0, i1 index.Info) []string {
+	var q []string
+
+	switch {
+	case i0.GetFormatVersion() != i1.GetFormatVersion():
+		q = append(q, fmt.Sprintf("mismatched FormatVersions: %v %v", i0.GetFormatVersion(), i1.GetFormatVersion()))
+	case i0.GetOriginalLength() != i1.GetOriginalLength():
+		q = append(q, fmt.Sprintf("mismatched OriginalLengths: %v %v", i0.GetOriginalLength(), i1.GetOriginalLength()))
+	case i0.GetPackBlobID() != i1.GetPackBlobID():
+		q = append(q, fmt.Sprintf("mismatched PackBlobIDs: %v %v", i0.GetPackBlobID(), i1.GetPackBlobID()))
+	case i0.GetPackedLength() != i1.GetPackedLength():
+		q = append(q, fmt.Sprintf("mismatched PackedLengths: %v %v", i0.GetPackedLength(), i1.GetPackedLength()))
+	case i0.GetPackOffset() != i1.GetPackOffset():
+		q = append(q, fmt.Sprintf("mismatched PackOffsets: %v %v", i0.GetPackOffset(), i1.GetPackOffset()))
+	case i0.GetEncryptionKeyID() != i1.GetEncryptionKeyID():
+		q = append(q, fmt.Sprintf("mismatched EncryptionKeyIDs: %v %v", i0.GetEncryptionKeyID(), i1.GetEncryptionKeyID()))
+	case i0.GetCompressionHeaderID() != i1.GetCompressionHeaderID():
+		q = append(q, fmt.Sprintf("mismatched GetCompressionHeaderID: %v %v", i0.GetCompressionHeaderID(), i1.GetCompressionHeaderID()))
+	case i0.GetDeleted() != i1.GetDeleted():
+		q = append(q, fmt.Sprintf("mismatched Deleted flags: %v %v", i0.GetDeleted(), i1.GetDeleted()))
+	case i0.GetTimestampSeconds() != i1.GetTimestampSeconds():
+		q = append(q, fmt.Sprintf("mismatched TimestampSeconds: %v %v", i0.GetTimestampSeconds(), i1.GetTimestampSeconds()))
+	}
+
+	if len(q) == 0 {
+		return nil
+	}
+
+	for i := range q {
+		q[i] = fmt.Sprintf("index blobs do not match: %q, %q: %s", i0.GetPackBlobID(), i1.GetPackBlobID(), q[i])
+	}
+
+	return q
+}
+
 func (c *commandRepositoryUpgrade) forceRollbackAction(ctx context.Context, rep repo.DirectRepositoryWriter) error {
 	if !c.forceRollback {
 		return errors.New("repository upgrade lock can only be revoked unsafely; please use the --force flag")
@@ -109,10 +262,33 @@ func (c *commandRepositoryUpgrade) runPhase(act func(context.Context, repo.Direc
 	}
 }
 
+func (c *commandRepositoryUpgrade) ignoreErrorOnAlwaysCommit(act func(context.Context, repo.DirectRepositoryWriter) error) func(context.Context, repo.DirectRepositoryWriter) error {
+	return func(ctx context.Context, rep repo.DirectRepositoryWriter) error {
+		err := act(ctx, rep)
+		if err == nil {
+			return nil
+		}
+
+		if c.commitMode == commitModeAlwaysCommit {
+			log(ctx).Errorf("%v", err)
+			return nil
+		}
+
+		// Explicitly skip all stages on error because tests do not
+		// skip/exit on error. Tests override os.Exit() that prevents
+		// running rest of the phases until we set the skip flag here.
+		// This flag is designed for testability and also to support
+		// rollback.
+		c.skip = true
+
+		return err
+	}
+}
+
 // setLockIntent is an upgrade phase which sets the upgrade lock intent with
 // desired parameters.
 func (c *commandRepositoryUpgrade) setLockIntent(ctx context.Context, rep repo.DirectRepositoryWriter) error {
-	if c.ioDrainTimeout < format.DefaultRepositoryBlobCacheDuration && !c.force {
+	if c.ioDrainTimeout < format.DefaultRepositoryBlobCacheDuration && !c.allowUnsafeUpgradeTimings {
 		return errors.Errorf("minimum required io-drain-timeout is %s", format.DefaultRepositoryBlobCacheDuration)
 	}
 
@@ -283,11 +459,6 @@ func (c *commandRepositoryUpgrade) upgrade(ctx context.Context, rep repo.DirectR
 		return errors.Wrap(err, "error setting parameters")
 	}
 
-	// poison V0 index so that old readers won't be able to open it.
-	if err := content.WriteLegacyIndexPoisonBlob(ctx, rep.BlobStorage()); err != nil {
-		log(ctx).Errorf("unable to write legacy index poison blob: %v", err)
-	}
-
 	// we need to reopen the repository after this point
 
 	log(ctx).Infof("Repository indices have been upgraded.")
@@ -301,6 +472,11 @@ func (c *commandRepositoryUpgrade) upgrade(ctx context.Context, rep repo.DirectR
 // cleanup and backups used for the rollback mechanism, so we cannot rollback
 // after this phase.
 func (c *commandRepositoryUpgrade) commitUpgrade(ctx context.Context, rep repo.DirectRepositoryWriter) error {
+	if c.commitMode == commitModeNeverCommit {
+		log(ctx).Infof("Commit mode is set to 'never'.  Skipping commit.")
+		return nil
+	}
+
 	if err := rep.FormatManager().CommitUpgrade(ctx); err != nil {
 		return errors.Wrap(err, "error finalizing upgrade")
 	}
diff --git a/repo/content/committed_read_manager.go b/repo/content/committed_read_manager.go
index 1aa99fa7..d07e575c 100644
--- a/repo/content/committed_read_manager.go
+++ b/repo/content/committed_read_manager.go
@@ -63,6 +63,11 @@
 	epoch.RangeCheckpointIndexBlobPrefix,
 }
 
+// IndexBlobReader provides an API for reading index blobs.
+type IndexBlobReader interface {
+	ListIndexBlobInfos(context.Context) ([]IndexBlobInfo, time.Time, error)
+}
+
 // indexBlobManager is the API of index blob manager as used by content manager.
 type indexBlobManager interface {
 	writeIndexBlobs(ctx context.Context, data []gather.Bytes, sessionID SessionID) ([]blob.Metadata, error)
@@ -114,6 +119,26 @@ type SharedManager struct {
 	metricsStruct
 }
 
+// LoadIndexBlob return index information loaded from the specified blob.
+func (sm *SharedManager) LoadIndexBlob(ctx context.Context, ibid blob.ID, d *gather.WriteBuffer) ([]Info, error) {
+	err := sm.st.GetBlob(ctx, ibid, 0, -1, d)
+	if err != nil {
+		return nil, errors.Wrapf(err, "could not find index blob %q", ibid)
+	}
+
+	return ParseIndexBlob(ibid, d.Bytes(), sm.format)
+}
+
+// IndexReaderV0 return an index reader for reading V0 indexes.
+func (sm *SharedManager) IndexReaderV0() IndexBlobReader {
+	return sm.indexBlobManagerV0
+}
+
+// IndexReaderV1 return an index reader for reading V1 indexes.
+func (sm *SharedManager) IndexReaderV1() IndexBlobReader {
+	return sm.indexBlobManagerV1
+}
+
 func (sm *SharedManager) readPackFileLocalIndex(ctx context.Context, packFile blob.ID, packFileLength int64, output *gather.WriteBuffer) error {
 	var err error
 
diff --git a/repo/content/content_manager_indexes.go b/repo/content/content_manager_indexes.go
index 76eded95..b3946a90 100644
--- a/repo/content/content_manager_indexes.go
+++ b/repo/content/content_manager_indexes.go
@@ -80,7 +80,7 @@ func (sm *SharedManager) CompactIndexes(ctx context.Context, opt CompactOptions)
 }
 
 // ParseIndexBlob loads entries in a given index blob and returns them.
-func ParseIndexBlob(ctx context.Context, blobID blob.ID, encrypted gather.Bytes, crypter crypter) ([]Info, error) {
+func ParseIndexBlob(blobID blob.ID, encrypted gather.Bytes, crypter crypter) ([]Info, error) {
 	var data gather.WriteBuffer
 	defer data.Close()
 
diff --git a/repo/content/index_blob_manager_v0.go b/repo/content/index_blob_manager_v0.go
index 8e674f0e..46fe5e20 100644
--- a/repo/content/index_blob_manager_v0.go
+++ b/repo/content/index_blob_manager_v0.go
@@ -19,8 +19,6 @@
 const LegacyIndexBlobPrefix = "n"
 
 const (
-	legacyIndexPoisonBlobID = "n00000000000000000000000000000000-repository_unreadable_by_this_kopia_version_upgrade_required"
-
 	defaultIndexShardSize = 16e6 // slightly less than 2^24, which lets index use 24-bit/3-byte indexes
 
 	defaultEventualConsistencySettleTime = 1 * time.Hour
@@ -67,6 +65,27 @@ type indexBlobManagerV0 struct {
 	formattingOptions IndexFormattingOptions
 }
 
+// ListIndexBlobInfos list active blob info structs.  Also returns time of latest content deletion commit.
+func (m *indexBlobManagerV0) ListIndexBlobInfos(ctx context.Context) ([]IndexBlobInfo, time.Time, error) {
+	activeIndexBlobs, t0, err := m.listActiveIndexBlobs(ctx)
+	if err != nil {
+		return nil, time.Time{}, err
+	}
+
+	q := make([]IndexBlobInfo, 0, len(activeIndexBlobs))
+
+	for _, activeIndexBlob := range activeIndexBlobs {
+		// skip the V0 blob poison that is used to prevent client reads.
+		if activeIndexBlob.BlobID == format.LegacyIndexPoisonBlobID {
+			continue
+		}
+
+		q = append(q, activeIndexBlob)
+	}
+
+	return q, t0, nil
+}
+
 func (m *indexBlobManagerV0) listActiveIndexBlobs(ctx context.Context) ([]IndexBlobInfo, time.Time, error) {
 	var compactionLogMetadata, storageIndexBlobs []blob.Metadata
 
@@ -533,18 +552,6 @@ func (m *indexBlobManagerV0) dropContentsFromBuilder(bld index.Builder, opt Comp
 	}
 }
 
-// WriteLegacyIndexPoisonBlob writes a "poison blob" that will prevent old kopia clients
-// that have not been upgraded from being able to open the repository after its format
-// has been upgraded.
-func WriteLegacyIndexPoisonBlob(ctx context.Context, st blob.Storage) error {
-	//nolint:wrapcheck
-	return st.PutBlob(
-		ctx,
-		legacyIndexPoisonBlobID,
-		gather.FromSlice([]byte("The format of this repository has been upgraded and cannot be read by old clients")),
-		blob.PutOptions{})
-}
-
 func addIndexBlobsToBuilder(ctx context.Context, enc *encryptedBlobMgr, bld index.Builder, indexBlobID blob.ID) error {
 	var data gather.WriteBuffer
 	defer data.Close()
diff --git a/repo/content/index_blob_manager_v1.go b/repo/content/index_blob_manager_v1.go
index a143aa36..e02bd23c 100644
--- a/repo/content/index_blob_manager_v1.go
+++ b/repo/content/index_blob_manager_v1.go
@@ -24,6 +24,11 @@ type indexBlobManagerV1 struct {
 	formattingOptions IndexFormattingOptions
 }
 
+// ListIndexBlobInfos list active blob info structs.  Also returns time of latest content deletion commit.
+func (m *indexBlobManagerV1) ListIndexBlobInfos(ctx context.Context) ([]IndexBlobInfo, time.Time, error) {
+	return m.listActiveIndexBlobs(ctx)
+}
+
 func (m *indexBlobManagerV1) listActiveIndexBlobs(ctx context.Context) ([]IndexBlobInfo, time.Time, error) {
 	active, deletionWatermark, err := m.epochMgr.GetCompleteIndexSet(ctx, epoch.LatestEpoch)
 	if err != nil {
diff --git a/repo/format/upgrade_lock.go b/repo/format/upgrade_lock.go
index faee00bb..85712f95 100644
--- a/repo/format/upgrade_lock.go
+++ b/repo/format/upgrade_lock.go
@@ -9,10 +9,15 @@
 	"github.com/kopia/kopia/repo/blob"
 )
 
-// BackupBlobIDPrefix is the prefix for all identifiers of the BLOBs that
-// keep a backup copy of the FormatBlobID BLOB for the purposes of rollback
-// during upgrade.
-const BackupBlobIDPrefix = "kopia.repository.backup."
+const (
+	// BackupBlobIDPrefix is the prefix for all identifiers of the BLOBs that
+	// keep a backup copy of the FormatBlobID BLOB for the purposes of rollback
+	// during upgrade.
+	BackupBlobIDPrefix = "kopia.repository.backup."
+
+	// LegacyIndexPoisonBlobID used to pollute V0 indexes after upgrade to prevent legacy clients from corrupting V1 indexes.
+	LegacyIndexPoisonBlobID = "n00000000000000000000000000000000-repository_unreadable_by_this_kopia_version_upgrade_required"
+)
 
 // BackupBlobID gets the upgrade backu pblob-id fro mthe lock.
 func BackupBlobID(l UpgradeLockIntent) blob.ID {
@@ -24,7 +29,7 @@ func BackupBlobID(l UpgradeLockIntent) blob.ID {
 // it updates the existing lock using the output of the UpgradeLock.Update().
 //
 // This method also backs up the original format version on the upgrade lock
-// intent and sets the latest format-version o nthe repository blob. This
+// intent and sets the latest format-version to the repository blob. This
 // should cause the unsupporting clients (non-upgrade capable) to fail
 // connecting to the repository.
 func (m *Manager) SetUpgradeLockIntent(ctx context.Context, l UpgradeLockIntent) (*UpgradeLockIntent, error) {
@@ -53,7 +58,7 @@ func (m *Manager) SetUpgradeLockIntent(ctx context.Context, l UpgradeLockIntent)
 			return nil, errors.Wrap(err, "failed to backup the repo format blob")
 		}
 
-		// set a new lock or revoke an existing lock
+		// set a new lock or revoke an existing lock.
 		m.repoConfig.UpgradeLock = &l
 		// mark the upgrade to the new format version, this will ensure that older
 		// clients won't be able to parse the new version
@@ -71,6 +76,18 @@ func (m *Manager) SetUpgradeLockIntent(ctx context.Context, l UpgradeLockIntent)
 	return m.repoConfig.UpgradeLock.Clone(), nil
 }
 
+// WriteLegacyIndexPoisonBlob writes a "poison blob" that will prevent old kopia clients
+// that have not been upgraded from being able to open the repository after its format
+// has been upgraded.
+func WriteLegacyIndexPoisonBlob(ctx context.Context, st blob.Storage) error {
+	//nolint:wrapcheck
+	return st.PutBlob(
+		ctx,
+		LegacyIndexPoisonBlobID,
+		gather.FromSlice([]byte("The format of this repository has been upgraded and cannot be read by old clients")),
+		blob.PutOptions{})
+}
+
 // CommitUpgrade removes the upgrade lock from the from the repository format
 // blob. This in-effect commits the new repository format t othe repository and
 // resumes all access to the repository.
@@ -86,6 +103,11 @@ func (m *Manager) CommitUpgrade(ctx context.Context) error {
 		return errors.New("no upgrade in progress")
 	}
 
+	// poison V0 index so that old readers won't be able to open it.
+	if err := WriteLegacyIndexPoisonBlob(ctx, m.blobs); err != nil {
+		log(ctx).Errorf("unable to write legacy index poison blob: %v", err)
+	}
+
 	// restore the old format version
 	m.repoConfig.UpgradeLock = nil
 


Test patch:
diff --git a/cli/command_repository_upgrade_test.go b/cli/command_repository_upgrade_test.go
index 20ccda71..cd19bb6f 100644
--- a/cli/command_repository_upgrade_test.go
+++ b/cli/command_repository_upgrade_test.go
@@ -1,11 +1,14 @@
 package cli_test
 
 import (
+	"fmt"
 	"testing"
 	"time"
 
 	"github.com/stretchr/testify/require"
 
+	"github.com/kopia/kopia/cli"
+	"github.com/kopia/kopia/repo/content/index"
 	"github.com/kopia/kopia/repo/format"
 	"github.com/kopia/kopia/tests/testenv"
 )
@@ -52,6 +55,144 @@ func (s *formatSpecificTestSuite) TestRepositoryUpgrade(t *testing.T) {
 	require.Contains(t, out, "Format version:      3")
 }
 
+func (s *formatSpecificTestSuite) TestRepositoryCorruptedUpgrade(t *testing.T) {
+	env := testenv.NewCLITest(t, s.formatFlags, testenv.NewInProcRunner(t))
+
+	env.RunAndExpectSuccess(t, "repo", "create", "filesystem", "--path", env.RepoDir)
+	out := env.RunAndExpectSuccess(t, "repository", "status", "--upgrade-no-block")
+
+	env.Environment["KOPIA_UPGRADE_LOCK_ENABLED"] = "1"
+
+	switch s.formatVersion {
+	case format.FormatVersion1:
+		require.Contains(t, out, "Format version:      1")
+		// run upgrade first with commit-mode set to never.  this leaves the lock and new index intact so that
+		// the file can be corrupted with "TweakFile".
+		_, stderr := env.RunAndExpectSuccessWithErrOut(t, "repository", "upgrade",
+			"--upgrade-owner-id", "owner",
+			"--commit-mode", "never",
+			"--io-drain-timeout", "1s", "--allow-unsafe-upgrade",
+			"--status-poll-interval", "1s",
+			"--max-permitted-clock-drift", "1s")
+		require.Contains(t, stderr, "Repository indices have been upgraded.")
+		require.Contains(t, stderr, "Commit mode is set to 'never'.  Skipping commit.")
+		require.Contains(t, stderr, "index validation succeeded")
+		env.TweakFile(t, env.RepoDir, "x*/*/*.f")
+		// then re-run the upgrade with the corrupted index.  This should fail on index validation.
+		_, stderr = env.RunAndExpectFailure(t, "repository", "upgrade",
+			"--upgrade-owner-id", "owner")
+		require.Regexp(t, "failed to load index entries for new index: failed to load index blob with BlobID", stderr)
+	case format.FormatVersion2:
+		require.Contains(t, out, "Format version:      2")
+		_, stderr := env.RunAndExpectSuccessWithErrOut(t, "repository", "upgrade",
+			"--upgrade-owner-id", "owner",
+			"--io-drain-timeout", "1s", "--allow-unsafe-upgrade",
+			"--status-poll-interval", "1s",
+			"--max-permitted-clock-drift", "1s")
+		require.Contains(t, stderr, "Repository indices have already been migrated to the epoch format, no need to drain other clients")
+	default:
+		require.Contains(t, out, "Format version:      3")
+		env.RunAndExpectFailure(t, "repository", "upgrade",
+			"--upgrade-owner-id", "owner",
+			"--io-drain-timeout", "1s", "--allow-unsafe-upgrade",
+			"--status-poll-interval", "1s",
+			"--max-permitted-clock-drift", "1s")
+	}
+}
+
+func (s *formatSpecificTestSuite) TestRepositoryUpgradeCommitNever(t *testing.T) {
+	env := testenv.NewCLITest(t, s.formatFlags, testenv.NewInProcRunner(t))
+
+	env.RunAndExpectSuccess(t, "repo", "create", "filesystem", "--path", env.RepoDir)
+	stdout := env.RunAndExpectSuccess(t, "repository", "status", "--upgrade-no-block")
+
+	env.Environment["KOPIA_UPGRADE_LOCK_ENABLED"] = "1"
+
+	switch s.formatVersion {
+	case format.FormatVersion1:
+		require.Contains(t, stdout, "Format version:      1")
+		_, stderr := env.RunAndExpectSuccessWithErrOut(t, "repository", "upgrade",
+			"--commit-mode", "never",
+			"--upgrade-owner-id", "owner",
+			"--io-drain-timeout", "1s", "--allow-unsafe-upgrade",
+			"--status-poll-interval", "1s",
+			"--max-permitted-clock-drift", "1s")
+		require.Contains(t, stderr, "Repository indices have been upgraded.")
+		require.Contains(t, stderr, "Commit mode is set to 'never'.  Skipping commit.")
+
+		_, stderr = env.RunAndExpectFailure(t, "repository", "status", "--upgrade-no-block")
+		require.Contains(t, stderr, "failed to open repository: repository upgrade in progress")
+	case format.FormatVersion2:
+		require.Contains(t, stdout, "Format version:      2")
+		_, stderr := env.RunAndExpectSuccessWithErrOut(t, "repository", "upgrade",
+			"--commit-mode", "never",
+			"--upgrade-owner-id", "owner",
+			"--io-drain-timeout", "1s", "--allow-unsafe-upgrade",
+			"--status-poll-interval", "1s",
+			"--max-permitted-clock-drift", "1s")
+		require.Contains(t, stderr, "Repository indices have already been migrated to the epoch format, no need to drain other clients")
+		require.Contains(t, stderr, "Commit mode is set to 'never'.  Skipping commit.")
+
+		_, stderr = env.RunAndExpectFailure(t, "repository", "status", "--upgrade-no-block")
+		require.Contains(t, stderr, "failed to open repository: repository upgrade in progress")
+	default:
+		require.Contains(t, stdout, "Format version:      3")
+		env.RunAndExpectFailure(t, "repository", "upgrade",
+			"--commit-mode", "never",
+			"--upgrade-owner-id", "owner",
+			"--io-drain-timeout", "1s", "--allow-unsafe-upgrade",
+			"--status-poll-interval", "1s",
+			"--max-permitted-clock-drift", "1s")
+
+		env.RunAndExpectSuccess(t, "repository", "status", "--upgrade-no-block")
+	}
+}
+
+func (s *formatSpecificTestSuite) TestRepositoryUpgradeCommitAlways(t *testing.T) {
+	env := testenv.NewCLITest(t, s.formatFlags, testenv.NewInProcRunner(t))
+
+	env.RunAndExpectSuccess(t, "repo", "create", "filesystem", "--path", env.RepoDir)
+	out := env.RunAndExpectSuccess(t, "repository", "status", "--upgrade-no-block")
+
+	env.Environment["KOPIA_UPGRADE_LOCK_ENABLED"] = "1"
+
+	switch s.formatVersion {
+	case format.FormatVersion1:
+		require.Contains(t, out, "Format version:      1")
+		_, stderr := env.RunAndExpectSuccessWithErrOut(t, "repository", "upgrade",
+			"--commit-mode", "always",
+			"--upgrade-owner-id", "owner",
+			"--io-drain-timeout", "1s", "--allow-unsafe-upgrade",
+			"--status-poll-interval", "1s",
+			"--max-permitted-clock-drift", "1s")
+		require.Contains(t, stderr, "Repository indices have been upgraded.")
+		require.Contains(t, stderr, "Repository has been successfully upgraded.")
+	case format.FormatVersion2:
+		require.Contains(t, out, "Format version:      2")
+		_, stderr := env.RunAndExpectSuccessWithErrOut(t, "repository", "upgrade",
+			"--commit-mode", "always",
+			"--upgrade-owner-id", "owner",
+			"--io-drain-timeout", "1s", "--allow-unsafe-upgrade",
+			"--status-poll-interval", "1s",
+			"--max-permitted-clock-drift", "1s")
+		require.Contains(t, stderr, "Repository indices have already been migrated to the epoch format, no need to drain other clients")
+		require.Contains(t, stderr, "Repository has been successfully upgraded.")
+	default:
+		require.Contains(t, out, "Format version:      3")
+		env.RunAndExpectFailure(t, "repository", "upgrade",
+			"--commit-mode", "always",
+			"--upgrade-owner-id", "owner",
+			"--io-drain-timeout", "1s", "--allow-unsafe-upgrade",
+			"--status-poll-interval", "1s",
+			"--max-permitted-clock-drift", "1s")
+	}
+
+	out = env.RunAndExpectSuccess(t, "repository", "status", "--upgrade-no-block")
+	require.Contains(t, out, "Epoch Manager:       enabled")
+	require.Contains(t, out, "Index Format:        v2")
+	require.Contains(t, out, "Format version:      3")
+}
+
 func lockRepositoryForUpgrade(t *testing.T, env *testenv.CLITest) {
 	t.Helper()
 
@@ -145,3 +286,135 @@ func (s *formatSpecificTestSuite) TestRepositoryUpgradeStatusWhileLocked(t *test
 	require.Contains(t, out, "Index Format:        v2")
 	require.Contains(t, out, "Format version:      3")
 }
+
+func TestRepositoryUpgrade_checkIndexInfo(t *testing.T) {
+	tcs := []struct {
+		indexInfo0   index.Info
+		indexInfo1   index.Info
+		expectRegexs []string
+	}{
+		{
+			indexInfo0:   &index.InfoStruct{PackBlobID: "a"},
+			indexInfo1:   &index.InfoStruct{PackBlobID: "a"},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0: &index.InfoStruct{PackBlobID: "a"},
+			indexInfo1: &index.InfoStruct{PackBlobID: "b"},
+			expectRegexs: []string{
+				`do not match: "a", "b".*PackBlobID`,
+			},
+		},
+		{
+			indexInfo0:   &index.InfoStruct{TimestampSeconds: 1},
+			indexInfo1:   &index.InfoStruct{TimestampSeconds: 1},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0: &index.InfoStruct{TimestampSeconds: 1},
+			indexInfo1: &index.InfoStruct{TimestampSeconds: 2},
+			expectRegexs: []string{
+				"do not match.*TimestampSeconds",
+			},
+		},
+		{
+			indexInfo0:   &index.InfoStruct{OriginalLength: 1},
+			indexInfo1:   &index.InfoStruct{OriginalLength: 1},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0: &index.InfoStruct{OriginalLength: 1},
+			indexInfo1: &index.InfoStruct{OriginalLength: 2},
+			expectRegexs: []string{
+				"do not match.*OriginalLength",
+			},
+		},
+		{
+			indexInfo0:   &index.InfoStruct{PackedLength: 1},
+			indexInfo1:   &index.InfoStruct{PackedLength: 1},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0: &index.InfoStruct{PackedLength: 1},
+			indexInfo1: &index.InfoStruct{PackedLength: 2},
+			expectRegexs: []string{
+				"do not match.*PackedLength",
+			},
+		},
+		{
+			indexInfo0:   &index.InfoStruct{PackOffset: 1},
+			indexInfo1:   &index.InfoStruct{PackOffset: 1},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0: &index.InfoStruct{PackOffset: 1},
+			indexInfo1: &index.InfoStruct{PackOffset: 2},
+			expectRegexs: []string{
+				"do not match.*PackOffset",
+			},
+		},
+		{
+			indexInfo0:   &index.InfoStruct{Deleted: true},
+			indexInfo1:   &index.InfoStruct{Deleted: true},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0: &index.InfoStruct{Deleted: false},
+			indexInfo1: &index.InfoStruct{Deleted: true},
+			expectRegexs: []string{
+				"do not match.*Deleted",
+			},
+		},
+		// simple logic error can make result of this false... so check
+		{
+			indexInfo0:   &index.InfoStruct{Deleted: false},
+			indexInfo1:   &index.InfoStruct{Deleted: false},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0:   &index.InfoStruct{FormatVersion: 1},
+			indexInfo1:   &index.InfoStruct{FormatVersion: 1},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0: &index.InfoStruct{FormatVersion: 1},
+			indexInfo1: &index.InfoStruct{FormatVersion: 2},
+			expectRegexs: []string{
+				"do not match.*FormatVersion",
+			},
+		},
+		{
+			indexInfo0:   &index.InfoStruct{CompressionHeaderID: 1},
+			indexInfo1:   &index.InfoStruct{CompressionHeaderID: 1},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0: &index.InfoStruct{CompressionHeaderID: 1},
+			indexInfo1: &index.InfoStruct{CompressionHeaderID: 2},
+			expectRegexs: []string{
+				"do not match.*CompressionHeaderID",
+			},
+		},
+		{
+			indexInfo0:   &index.InfoStruct{EncryptionKeyID: 1},
+			indexInfo1:   &index.InfoStruct{EncryptionKeyID: 1},
+			expectRegexs: []string{},
+		},
+		{
+			indexInfo0: &index.InfoStruct{EncryptionKeyID: 1},
+			indexInfo1: &index.InfoStruct{EncryptionKeyID: 2},
+			expectRegexs: []string{
+				"do not match.*EncryptionKeyID",
+			},
+		},
+	}
+	for i, tc := range tcs {
+		t.Run(fmt.Sprintf("%d", i), func(t *testing.T) {
+			report := cli.CheckIndexInfo(tc.indexInfo0, tc.indexInfo1)
+			require.Equal(t, len(report), len(tc.expectRegexs), "unexpected report length")
+			for i := range tc.expectRegexs {
+				require.Regexp(t, tc.expectRegexs[i], report[i], "report does not match regular expression")
+			}
+		})
+	}
+}
diff --git a/tests/testenv/cli_test_env.go b/tests/testenv/cli_test_env.go
index ef42a818..07d1cc21 100644
--- a/tests/testenv/cli_test_env.go
+++ b/tests/testenv/cli_test_env.go
@@ -4,7 +4,10 @@
 import (
 	"bufio"
 	"io"
+	"io/fs"
+	"math/rand"
 	"os"
+	"path"
 	"path/filepath"
 	"runtime"
 	"strings"
@@ -106,6 +109,37 @@ func (e *CLITest) RunAndExpectSuccess(t *testing.T, args ...string) []string {
 	return stdout
 }
 
+// TweakFile writes a xor-ed byte at a random point in a file.  Used to simulate file corruption.
+func (e *CLITest) TweakFile(t *testing.T, dirn, fglob string) {
+	t.Helper()
+
+	const RwUserGroupOther = 0o666
+
+	// find a file within the repository to corrupt
+	mch, err := fs.Glob(os.DirFS(dirn), fglob)
+	require.NoError(t, err)
+	require.Greater(t, len(mch), 0)
+
+	// grab a random file in the directory dirn
+	fn := mch[rand.Intn(len(mch))]
+	f, err := os.OpenFile(path.Join(dirn, fn), os.O_RDWR, os.FileMode(RwUserGroupOther))
+	require.NoError(t, err)
+
+	// find the length of the file, then seek to a random location
+	l, err := f.Seek(0, io.SeekEnd)
+	require.NoError(t, err)
+
+	i := rand.Int63n(l)
+	bs := [1]byte{}
+
+	_, err = f.ReadAt(bs[:], i)
+	require.NoError(t, err)
+
+	// write the byte
+	_, err = f.WriteAt([]byte{^bs[0]}, i)
+	require.NoError(t, err)
+}
+
 // RunAndProcessStderr runs the given command, and streams its output line-by-line to a given function until it returns false.
 func (e *CLITest) RunAndProcessStderr(t *testing.T, callback func(line string) bool, args ...string) (wait func() error, kill func()) {
 	t.Helper()
@@ -143,15 +177,17 @@ func (e *CLITest) RunAndExpectSuccessWithErrOut(t *testing.T, args ...string) (s
 }
 
 // RunAndExpectFailure runs the given command, expects it to fail and returns its output lines.
-func (e *CLITest) RunAndExpectFailure(t *testing.T, args ...string) []string {
+func (e *CLITest) RunAndExpectFailure(t *testing.T, args ...string) (stdout, stderr []string) {
 	t.Helper()
 
-	stdout, _, err := e.Run(t, true, args...)
+	var err error
+
+	stdout, stderr, err = e.Run(t, true, args...)
 	if err == nil {
 		t.Fatalf("'kopia %v' succeeded, but expected failure", strings.Join(args, " "))
 	}
 
-	return stdout
+	return stdout, stderr
 }
 
 // RunAndVerifyOutputLineCount runs the given command and asserts it returns the given number of output lines, then returns them.