Repository: kopia/kopia
Pull request #1576 (2021-12-12 05:12:44)

test coverage improvements for filesystem provider
#1559 

codecov[bot]: # [Codecov](https://codecov.io/gh/kopia/kopia/pull/1576?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) Report
> Merging [#1576](https://codecov.io/gh/kopia/kopia/pull/1576?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) (072593a) into [master](https://codecov.io/gh/kopia/kopia/commit/e870fcc4aacef30d3165b58960a444099957beba?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) (e870fcc) will **increase** coverage by `0.14%`.
> The diff coverage is `92.03%`.

[![Impacted file tree graph](https://codecov.io/gh/kopia/kopia/pull/1576/graphs/tree.svg?width=650&height=150&src=pr&token=CRK4RMRFSH&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia)](https://codecov.io/gh/kopia/kopia/pull/1576?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia)

```diff
@@            Coverage Diff             @@
##           master    #1576      +/-   ##
==========================================
+ Coverage   69.32%   69.46%   +0.14%     
==========================================
  Files         377      378       +1     
  Lines       30048    30105      +57     
==========================================
+ Hits        20830    20913      +83     
+ Misses       7525     7506      -19     
+ Partials     1693     1686       -7     
```


| [Impacted Files](https://codecov.io/gh/kopia/kopia/pull/1576?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) | Coverage Δ | |
|---|---|---|
| [repo/blob/filesystem/filesystem\_options.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9ibG9iL2ZpbGVzeXN0ZW0vZmlsZXN5c3RlbV9vcHRpb25zLmdv) | `100.00% <ø> (ø)` | |
| [repo/blob/filesystem/osinterface\_realos.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9ibG9iL2ZpbGVzeXN0ZW0vb3NpbnRlcmZhY2VfcmVhbG9zLmdv) | `88.37% <88.37%> (ø)` | |
| [repo/blob/filesystem/filesystem\_storage.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9ibG9iL2ZpbGVzeXN0ZW0vZmlsZXN5c3RlbV9zdG9yYWdlLmdv) | `91.16% <94.20%> (+25.98%)` | :arrow_up: |
| [internal/scrubber/scrub\_sensitive.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvc2NydWJiZXIvc2NydWJfc2Vuc2l0aXZlLmdv) | `72.72% <100.00%> (ø)` | |
| [repo/content/encrypted\_blob\_mgr.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9jb250ZW50L2VuY3J5cHRlZF9ibG9iX21nci5nbw==) | `65.51% <0.00%> (-10.35%)` | :arrow_down: |
| [repo/maintenance/maintenance\_schedule.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9tYWludGVuYW5jZS9tYWludGVuYW5jZV9zY2hlZHVsZS5nbw==) | `66.37% <0.00%> (-5.18%)` | :arrow_down: |
| [repo/object/object\_writer.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9vYmplY3Qvb2JqZWN0X3dyaXRlci5nbw==) | `82.58% <0.00%> (-3.99%)` | :arrow_down: |
| [internal/server/server.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvc2VydmVyL3NlcnZlci5nbw==) | `66.09% <0.00%> (-1.72%)` | :arrow_down: |
| [internal/cache/persistent\_lru\_cache.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvY2FjaGUvcGVyc2lzdGVudF9scnVfY2FjaGUuZ28=) | `85.10% <0.00%> (-1.60%)` | :arrow_down: |
| [repo/maintenance/maintenance\_run.go](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-cmVwby9tYWludGVuYW5jZS9tYWludGVuYW5jZV9ydW4uZ28=) | `75.07% <0.00%> (-0.91%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/kopia/kopia/pull/1576/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/kopia/kopia/pull/1576?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kopia/kopia/pull/1576?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia). Last update [e870fcc...072593a](https://codecov.io/gh/kopia/kopia/pull/1576?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia).


Patch:
diff --git a/internal/scrubber/scrub_sensitive.go b/internal/scrubber/scrub_sensitive.go
index 4eece5de..562bd5f0 100644
--- a/internal/scrubber/scrub_sensitive.go
+++ b/internal/scrubber/scrub_sensitive.go
@@ -25,7 +25,7 @@ func ScrubSensitiveData(v reflect.Value) reflect.Value {
 				if sf.Type.Kind() == reflect.String {
 					res.Field(i).SetString(strings.Repeat("*", fv.Len()))
 				}
-			} else {
+			} else if sf.IsExported() {
 				res.Field(i).Set(fv)
 			}
 		}
diff --git a/repo/blob/filesystem/filesystem_options.go b/repo/blob/filesystem/filesystem_options.go
index 4271e7af..d87561fc 100644
--- a/repo/blob/filesystem/filesystem_options.go
+++ b/repo/blob/filesystem/filesystem_options.go
@@ -19,6 +19,8 @@ type Options struct {
 
 	sharded.Options
 	throttling.Limits
+
+	osInterfaceOverride osInterface
 }
 
 func (fso *Options) fileMode() os.FileMode {
diff --git a/repo/blob/filesystem/filesystem_storage.go b/repo/blob/filesystem/filesystem_storage.go
index ee07af4f..9d7b5fd8 100644
--- a/repo/blob/filesystem/filesystem_storage.go
+++ b/repo/blob/filesystem/filesystem_storage.go
@@ -37,34 +37,32 @@ type fsStorage struct {
 
 type fsImpl struct {
 	Options
+
+	osi osInterface
 }
 
 var errRetriableInvalidLength = errors.Errorf("invalid length (retriable)")
 
-func isRetriable(err error) bool {
+func (fs *fsImpl) isRetriable(err error) bool {
 	if err == nil {
 		return false
 	}
 
 	err = errors.Cause(err)
 
-	if os.IsNotExist(err) {
+	if fs.osi.IsNotExist(err) {
 		return false
 	}
 
-	if os.IsExist(err) {
+	if fs.osi.IsExist(err) {
 		return false
 	}
 
-	// retry errors during file operations
-	var pe *os.PathError
-	if errors.As(err, &pe) {
+	if fs.osi.IsPathError(err) {
 		return true
 	}
 
-	// retry errors during rename
-	var le *os.LinkError
-	if errors.As(err, &le) {
+	if fs.osi.IsLinkError(err) {
 		return true
 	}
 
@@ -75,13 +73,13 @@ func (fs *fsImpl) GetBlobFromPath(ctx context.Context, dirPath, path string, off
 	err := retry.WithExponentialBackoffNoValue(ctx, "GetBlobFromPath:"+path, func() error {
 		output.Reset()
 
-		f, err := os.Open(path) //nolint:gosec
+		f, err := fs.osi.Open(path)
 		if err != nil {
 			//nolint:wrapcheck
 			return err
 		}
 
-		defer f.Close() //nolint:errcheck,gosec
+		defer f.Close() //nolint:errcheck
 
 		if length < 0 {
 			// nolint:wrapcheck
@@ -112,9 +110,9 @@ func (fs *fsImpl) GetBlobFromPath(ctx context.Context, dirPath, path string, off
 		}
 
 		return nil
-	}, isRetriable)
+	}, fs.isRetriable)
 	if err != nil {
-		if os.IsNotExist(err) {
+		if fs.osi.IsNotExist(err) {
 			return blob.ErrBlobNotFound
 		}
 
@@ -127,20 +125,28 @@ func (fs *fsImpl) GetBlobFromPath(ctx context.Context, dirPath, path string, off
 }
 
 func (fs *fsImpl) GetMetadataFromPath(ctx context.Context, dirPath, path string) (blob.Metadata, error) {
-	fi, err := os.Stat(path)
-	if err != nil {
-		if os.IsNotExist(err) {
-			return blob.Metadata{}, blob.ErrBlobNotFound
+	v, err := retry.WithExponentialBackoff(ctx, "GetMetadataFromPath:"+path, func() (interface{}, error) {
+		fi, err := fs.osi.Stat(path)
+		if err != nil {
+			if fs.osi.IsNotExist(err) {
+				return blob.Metadata{}, blob.ErrBlobNotFound
+			}
+
+			// nolint:wrapcheck
+			return blob.Metadata{}, err
 		}
 
+		return blob.Metadata{
+			Length:    fi.Size(),
+			Timestamp: fi.ModTime(),
+		}, nil
+	}, fs.isRetriable)
+	if err != nil {
 		// nolint:wrapcheck
 		return blob.Metadata{}, err
 	}
 
-	return blob.Metadata{
-		Length:    fi.Size(),
-		Timestamp: fi.ModTime(),
-	}, nil
+	return v.(blob.Metadata), nil
 }
 
 func (fs *fsImpl) PutBlobInPath(ctx context.Context, dirPath, path string, data blob.Bytes) error {
@@ -166,9 +172,9 @@ func (fs *fsImpl) PutBlobInPath(ctx context.Context, dirPath, path string, data
 			return errors.Wrap(err, "can't close temporary file")
 		}
 
-		err = os.Rename(tempFile, path)
+		err = fs.osi.Rename(tempFile, path)
 		if err != nil {
-			if removeErr := os.Remove(tempFile); removeErr != nil {
+			if removeErr := fs.osi.Remove(tempFile); removeErr != nil {
 				log(ctx).Errorf("can't remove temp file: %v", removeErr)
 			}
 
@@ -176,52 +182,54 @@ func (fs *fsImpl) PutBlobInPath(ctx context.Context, dirPath, path string, data
 			return err
 		}
 
-		if fs.FileUID != nil && fs.FileGID != nil && os.Geteuid() == 0 {
-			if chownErr := os.Chown(path, *fs.FileUID, *fs.FileGID); chownErr != nil {
+		if fs.FileUID != nil && fs.FileGID != nil && fs.osi.Geteuid() == 0 {
+			if chownErr := fs.osi.Chown(path, *fs.FileUID, *fs.FileGID); chownErr != nil {
 				log(ctx).Errorf("can't change file permissions: %v", chownErr)
 			}
 		}
 
 		return nil
-	}, isRetriable)
+	}, fs.isRetriable)
 }
 
-func (fs *fsImpl) createTempFileAndDir(tempFile string) (*os.File, error) {
-	flags := os.O_CREATE | os.O_WRONLY | os.O_EXCL
-
-	f, err := os.OpenFile(tempFile, flags, fs.fileMode()) //nolint:gosec
-	if os.IsNotExist(err) {
-		if err = os.MkdirAll(filepath.Dir(tempFile), fs.dirMode()); err != nil {
+func (fs *fsImpl) createTempFileAndDir(tempFile string) (osWriteFile, error) {
+	f, err := fs.osi.CreateNewFile(tempFile, fs.fileMode())
+	if fs.osi.IsNotExist(err) {
+		if err = fs.osi.MkdirAll(filepath.Dir(tempFile), fs.dirMode()); err != nil {
 			return nil, errors.Wrap(err, "cannot create directory")
 		}
 
 		// nolint:wrapcheck
-		return os.OpenFile(tempFile, flags, fs.fileMode()) //nolint:gosec
+		return fs.osi.CreateNewFile(tempFile, fs.fileMode())
 	}
 
-	// nolint:wrapcheck
-	return f, err
+	if err != nil {
+		// nolint:wrapcheck
+		return nil, err
+	}
+
+	return f, nil
 }
 
 func (fs *fsImpl) DeleteBlobInPath(ctx context.Context, dirPath, path string) error {
 	// nolint:wrapcheck
 	return retry.WithExponentialBackoffNoValue(ctx, "DeleteBlobInPath:"+path, func() error {
-		err := os.Remove(path)
-		if err == nil || os.IsNotExist(err) {
+		err := fs.osi.Remove(path)
+		if err == nil || fs.osi.IsNotExist(err) {
 			return nil
 		}
 
 		// nolint:wrapcheck
 		return err
-	}, isRetriable)
+	}, fs.isRetriable)
 }
 
 func (fs *fsImpl) ReadDir(ctx context.Context, dirname string) ([]os.FileInfo, error) {
 	v, err := retry.WithExponentialBackoff(ctx, "ReadDir:"+dirname, func() (interface{}, error) {
-		v, err := os.ReadDir(dirname)
+		v, err := fs.osi.ReadDir(dirname)
 		// nolint:wrapcheck
 		return v, err
-	}, isRetriable)
+	}, fs.isRetriable)
 	if err != nil {
 		// nolint:wrapcheck
 		return nil, err
@@ -232,7 +240,7 @@ func (fs *fsImpl) ReadDir(ctx context.Context, dirname string) ([]os.FileInfo, e
 	for _, e := range v.([]os.DirEntry) {
 		fi, err := e.Info()
 
-		if os.IsNotExist(err) {
+		if fs.osi.IsNotExist(err) {
 			// we lost the race, the file was deleted since it was listed.
 			continue
 		}
@@ -253,33 +261,38 @@ func (fs *fsImpl) SetTimeInPath(ctx context.Context, dirPath, filePath string, n
 	log(ctx).Debugf("updating timestamp on %v to %v", filePath, n)
 
 	// nolint:wrapcheck
-	return os.Chtimes(filePath, n, n)
+	return fs.osi.Chtimes(filePath, n, n)
 }
 
 // TouchBlob updates file modification time to current time if it's sufficiently old.
 func (fs *fsStorage) TouchBlob(ctx context.Context, blobID blob.ID, threshold time.Duration) error {
-	_, path, err := fs.Storage.GetShardedPathAndFilePath(ctx, blobID)
-	if err != nil {
-		return errors.Wrap(err, "error getting sharded path")
-	}
+	// nolint:wrapcheck
+	return retry.WithExponentialBackoffNoValue(ctx, "TouchBlob", func() error {
+		_, path, err := fs.Storage.GetShardedPathAndFilePath(ctx, blobID)
+		if err != nil {
+			return errors.Wrap(err, "error getting sharded path")
+		}
 
-	st, err := os.Stat(path)
-	if err != nil {
-		// nolint:wrapcheck
-		return err
-	}
+		osi := fs.Impl.(*fsImpl).osi
+
+		st, err := osi.Stat(path)
+		if err != nil {
+			// nolint:wrapcheck
+			return err
+		}
 
-	n := clock.Now()
+		n := clock.Now()
 
-	age := n.Sub(st.ModTime())
-	if age < threshold {
-		return nil
-	}
+		age := n.Sub(st.ModTime())
+		if age < threshold {
+			return nil
+		}
 
-	log(ctx).Debugf("updating timestamp on %v to %v", path, n)
+		log(ctx).Debugf("updating timestamp on %v to %v", path, n)
 
-	// nolint:wrapcheck
-	return os.Chtimes(path, n, n)
+		// nolint:wrapcheck
+		return osi.Chtimes(path, n, n)
+	}, fs.Impl.(*fsImpl).isRetriable)
 }
 
 func (fs *fsStorage) ConnectionInfo() blob.ConnectionInfo {
@@ -305,20 +318,25 @@ func (fs *fsStorage) FlushCaches(ctx context.Context) error {
 func New(ctx context.Context, opts *Options, isCreate bool) (blob.Storage, error) {
 	var err error
 
+	osi := opts.osInterfaceOverride
+	if osi == nil {
+		osi = realOS{}
+	}
+
 	if isCreate {
 		log(ctx).Debugf("creating directory: %v dir mode: %v", opts.Path, opts.dirMode())
 
-		if mkdirErr := os.MkdirAll(opts.Path, opts.dirMode()); mkdirErr != nil {
+		if mkdirErr := osi.MkdirAll(opts.Path, opts.dirMode()); mkdirErr != nil {
 			log(ctx).Errorf("unable to create directory: %v", mkdirErr)
 		}
 	}
 
-	if _, err = os.Stat(opts.Path); err != nil {
+	if _, err = osi.Stat(opts.Path); err != nil {
 		return nil, errors.Wrap(err, "cannot access storage path")
 	}
 
 	return &fsStorage{
-		sharded.New(&fsImpl{Options: *opts}, opts.Path, opts.Options, isCreate),
+		sharded.New(&fsImpl{*opts, osi}, opts.Path, opts.Options, isCreate),
 	}, nil
 }
 
diff --git a/repo/blob/filesystem/osinterface.go b/repo/blob/filesystem/osinterface.go
new file mode 100644
index 00000000..5a221082
--- /dev/null
+++ b/repo/blob/filesystem/osinterface.go
@@ -0,0 +1,36 @@
+package filesystem
+
+import (
+	"io"
+	"io/fs"
+	"os"
+	"time"
+)
+
+// osInterface is an operating system file interface, used by filesystemStorage to support mocking.
+type osInterface interface {
+	Open(fname string) (osReadFile, error)
+	IsNotExist(err error) bool
+	IsExist(err error) bool
+	IsPathError(err error) bool
+	IsLinkError(err error) bool
+	Remove(fname string) error
+	Rename(oldname, newname string) error
+	ReadDir(dirname string) ([]fs.DirEntry, error)
+	Stat(fname string) (os.FileInfo, error)
+	CreateNewFile(fname string, mode os.FileMode) (osWriteFile, error)
+	MkdirAll(fname string, mode os.FileMode) error
+	Chtimes(fname string, atime, mtime time.Time) error
+	Geteuid() int
+	Chown(fname string, uid, gid int) error
+}
+
+type osReadFile interface {
+	io.ReadSeekCloser
+
+	Stat() (os.FileInfo, error)
+}
+
+type osWriteFile interface {
+	io.WriteCloser
+}
diff --git a/repo/blob/filesystem/osinterface_realos.go b/repo/blob/filesystem/osinterface_realos.go
new file mode 100644
index 00000000..43f320d5
--- /dev/null
+++ b/repo/blob/filesystem/osinterface_realos.go
@@ -0,0 +1,83 @@
+package filesystem
+
+import (
+	"errors"
+	"io/fs"
+	"os"
+	"time"
+)
+
+// realOS is an implementation of osInterface that uses real operating system calls.
+type realOS struct{}
+
+func (realOS) Open(fname string) (osReadFile, error) {
+	f, err := os.Open(fname) //nolint:gosec
+	if err != nil {
+		// nolint:wrapcheck
+		return nil, err
+	}
+
+	return f, nil
+}
+
+func (realOS) IsNotExist(err error) bool { return os.IsNotExist(err) }
+
+func (realOS) IsExist(err error) bool { return os.IsExist(err) }
+
+func (realOS) Rename(oldname, newname string) error {
+	// nolint:wrapcheck
+	return os.Rename(oldname, newname)
+}
+
+func (realOS) ReadDir(dirname string) ([]fs.DirEntry, error) {
+	// nolint:wrapcheck
+	return os.ReadDir(dirname)
+}
+
+func (realOS) IsPathError(err error) bool {
+	var pe *os.PathError
+
+	return errors.As(err, &pe)
+}
+
+func (realOS) IsLinkError(err error) bool {
+	var pe *os.LinkError
+
+	return errors.As(err, &pe)
+}
+
+func (realOS) Remove(fname string) error {
+	// nolint:wrapcheck
+	return os.Remove(fname)
+}
+
+func (realOS) Stat(fname string) (os.FileInfo, error) {
+	// nolint:wrapcheck
+	return os.Stat(fname)
+}
+
+func (realOS) CreateNewFile(fname string, perm os.FileMode) (osWriteFile, error) {
+	// nolint:wrapcheck,gosec
+	return os.OpenFile(fname, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm)
+}
+
+func (realOS) MkdirAll(fname string, mode os.FileMode) error {
+	// nolint:wrapcheck
+	return os.MkdirAll(fname, mode)
+}
+
+func (realOS) Chtimes(fname string, atime, mtime time.Time) error {
+	// nolint:wrapcheck
+	return os.Chtimes(fname, atime, mtime)
+}
+
+func (realOS) Geteuid() int {
+	return os.Geteuid()
+}
+
+func (realOS) Chown(fname string, uid, gid int) error {
+	// nolint:wrapcheck
+	return os.Chown(fname, uid, gid)
+}
+
+var _ osInterface = realOS{}


Test patch:
diff --git a/repo/blob/filesystem/filesystem_storage_test.go b/repo/blob/filesystem/filesystem_storage_test.go
index 71ed8755..7a781a52 100644
--- a/repo/blob/filesystem/filesystem_storage_test.go
+++ b/repo/blob/filesystem/filesystem_storage_test.go
@@ -20,7 +20,6 @@
 
 func TestFileStorage(t *testing.T) {
 	t.Parallel()
-	testutil.ProviderTest(t)
 
 	ctx := testlogging.Context(t)
 
@@ -43,20 +42,40 @@ func TestFileStorage(t *testing.T) {
 			},
 		}, true)
 
-		if r == nil || err != nil {
-			t.Errorf("unexpected result: %v %v", r, err)
-		}
+		require.NoError(t, err)
+		require.NotNil(t, r)
 
 		blobtesting.VerifyStorage(ctx, t, r, blob.PutOptions{})
 		blobtesting.AssertConnectionInfoRoundTrips(ctx, t, r)
-		require.NoError(t, providervalidation.ValidateProvider(ctx, r, blobtesting.TestValidationOptions))
 
-		if err := r.Close(ctx); err != nil {
-			t.Fatalf("err: %v", err)
-		}
+		require.NoError(t, r.Close(ctx))
 	}
 }
 
+func TestFileStorageValidate(t *testing.T) {
+	t.Parallel()
+
+	testutil.ProviderTest(t)
+
+	ctx := testlogging.Context(t)
+
+	path := testutil.TempDirectory(t)
+
+	r, err := New(ctx, &Options{
+		Path:    path,
+		Options: sharded.Options{},
+	}, true)
+
+	require.NoError(t, err)
+	require.NotNil(t, r)
+
+	blobtesting.VerifyStorage(ctx, t, r, blob.PutOptions{})
+	blobtesting.AssertConnectionInfoRoundTrips(ctx, t, r)
+	require.NoError(t, providervalidation.ValidateProvider(ctx, r, blobtesting.TestValidationOptions))
+
+	require.NoError(t, r.Close(ctx))
+}
+
 const (
 	t1 = "392ee1bc299db9f235e046a62625afb84902"
 	t2 = "2a7ff4f29eddbcd4c18fa9e73fec20bbb71f"
@@ -65,7 +84,6 @@ func TestFileStorage(t *testing.T) {
 
 func TestFileStorageTouch(t *testing.T) {
 	t.Parallel()
-	testutil.ProviderTest(t)
 
 	ctx := testlogging.Context(t)
 
@@ -145,9 +163,7 @@ func TestFilesystemStorageDirectoryShards(t *testing.T) {
 			DirectoryShards: []int{5, 2},
 		},
 	}, true)
-	if err != nil {
-		t.Fatalf("unable to connect to rclone backend: %v", err)
-	}
+	require.NoError(t, err)
 
 	defer st.Close(ctx)
 
@@ -155,6 +171,283 @@ func TestFilesystemStorageDirectoryShards(t *testing.T) {
 	require.FileExists(t, filepath.Join(dataDir, "someb", "lo", "b1234567812345678.f"))
 }
 
+func TestFileStorage_GetBlob_RetriesOnReadError(t *testing.T) {
+	t.Parallel()
+
+	ctx := testlogging.Context(t)
+
+	dataDir := testutil.TempDirectory(t)
+
+	osi := &mockOS{
+		readFileRemainingErrors: 1,
+
+		osInterface: realOS{},
+	}
+
+	st, err := New(ctx, &Options{
+		Path: dataDir,
+		Options: sharded.Options{
+			DirectoryShards: []int{5, 2},
+		},
+		osInterfaceOverride: osi,
+	}, true)
+	require.NoError(t, err)
+
+	defer st.Close(ctx)
+
+	require.NoError(t, st.PutBlob(ctx, "someblob1234567812345678", gather.FromSlice([]byte{1, 2, 3}), blob.PutOptions{}))
+
+	var buf gather.WriteBuffer
+	defer buf.Close()
+
+	require.NoError(t, st.GetBlob(ctx, "someblob1234567812345678", 1, 2, &buf))
+	require.Equal(t, []byte{2, 3}, buf.ToByteSlice())
+}
+
+func TestFileStorage_GetMetadata_RetriesOnError(t *testing.T) {
+	t.Parallel()
+
+	ctx := testlogging.Context(t)
+
+	dataDir := testutil.TempDirectory(t)
+
+	osi := &mockOS{
+		statRemainingErrors: 1,
+
+		osInterface: realOS{},
+	}
+
+	st, err := New(ctx, &Options{
+		Path: dataDir,
+		Options: sharded.Options{
+			DirectoryShards: []int{5, 2},
+		},
+	}, true)
+	require.NoError(t, err)
+
+	defer st.Close(ctx)
+
+	require.NoError(t, st.PutBlob(ctx, "someblob1234567812345678", gather.FromSlice([]byte{1, 2, 3}), blob.PutOptions{}))
+
+	st.(*fsStorage).Impl.(*fsImpl).osi = osi
+
+	_, err = st.GetMetadata(ctx, "someblob1234567812345678")
+	require.NoError(t, err)
+}
+
+func TestFileStorage_PutBlob_RetriesOnErrors(t *testing.T) {
+	t.Parallel()
+
+	ctx := testlogging.Context(t)
+
+	dataDir := testutil.TempDirectory(t)
+
+	osi := &mockOS{
+		osInterface:                    realOS{},
+		createNewFileRemainingErrors:   3,
+		mkdirAllRemainingErrors:        2,
+		writeFileRemainingErrors:       3,
+		writeFileCloseRemainingErrors:  2,
+		renameRemainingErrors:          1,
+		removeRemainingRetriableErrors: 3,
+		chownRemainingErrors:           3,
+
+		effectiveUID: 0, // running as root
+	}
+
+	fileUID := 3
+	fileGID := 4
+
+	st, err := New(ctx, &Options{
+		Path:    dataDir,
+		FileUID: &fileUID,
+		FileGID: &fileGID,
+		Options: sharded.Options{
+			DirectoryShards: []int{5, 2},
+		},
+	}, true)
+	require.NoError(t, err)
+
+	st.(*fsStorage).Impl.(*fsImpl).osi = osi
+
+	defer st.Close(ctx)
+
+	require.NoError(t, st.PutBlob(ctx, "someblob1234567812345678", gather.FromSlice([]byte{1, 2, 3}), blob.PutOptions{}))
+
+	var buf gather.WriteBuffer
+	defer buf.Close()
+
+	require.NoError(t, st.GetBlob(ctx, "someblob1234567812345678", 1, 2, &buf))
+	require.Equal(t, []byte{2, 3}, buf.ToByteSlice())
+}
+
+func TestFileStorage_DeleteBlob_ErrorHandling(t *testing.T) {
+	t.Parallel()
+
+	ctx := testlogging.Context(t)
+
+	dataDir := testutil.TempDirectory(t)
+
+	osi := &mockOS{
+		osInterface:                       realOS{},
+		removeRemainingNonRetriableErrors: 1,
+	}
+
+	st, err := New(ctx, &Options{
+		Path: dataDir,
+		Options: sharded.Options{
+			DirectoryShards: []int{5, 2},
+		},
+	}, true)
+	require.NoError(t, err)
+
+	st.(*fsStorage).Impl.(*fsImpl).osi = osi
+
+	defer st.Close(ctx)
+
+	require.ErrorIs(t, st.DeleteBlob(ctx, "someblob1234567812345678"), errNonRetriable)
+}
+
+func TestFileStorage_New_MkdirAllFailureIsIgnored(t *testing.T) {
+	t.Parallel()
+
+	ctx := testlogging.Context(t)
+
+	dataDir := testutil.TempDirectory(t)
+
+	osi := &mockOS{
+		osInterface:             realOS{},
+		mkdirAllRemainingErrors: 1,
+	}
+
+	st, err := New(ctx, &Options{
+		Path: dataDir,
+		Options: sharded.Options{
+			DirectoryShards: []int{5, 2},
+		},
+		osInterfaceOverride: osi,
+	}, true)
+	require.NoError(t, err)
+
+	defer st.Close(ctx)
+}
+
+func TestFileStorage_New_ChecksDirectoryExistence(t *testing.T) {
+	t.Parallel()
+
+	ctx := testlogging.Context(t)
+
+	dataDir := testutil.TempDirectory(t)
+
+	osi := &mockOS{
+		osInterface:         realOS{},
+		statRemainingErrors: 1,
+	}
+
+	st, err := New(ctx, &Options{
+		Path: dataDir,
+		Options: sharded.Options{
+			DirectoryShards: []int{5, 2},
+		},
+		osInterfaceOverride: osi,
+	}, true)
+	require.Error(t, err)
+	require.Nil(t, st)
+}
+
+func TestFileStorage_ListBlobs_ErrorHandling(t *testing.T) {
+	t.Parallel()
+
+	ctx := testlogging.Context(t)
+
+	dataDir := testutil.TempDirectory(t)
+
+	osi := &mockOS{
+		osInterface:                         realOS{},
+		readDirRemainingErrors:              3,
+		readDirRemainingFileDeletedDirEntry: 3,
+	}
+
+	st, err := New(ctx, &Options{
+		Path: dataDir,
+		Options: sharded.Options{
+			DirectoryShards: []int{5, 2},
+		},
+	}, true)
+	require.NoError(t, err)
+
+	st.(*fsStorage).Impl.(*fsImpl).osi = osi
+
+	defer st.Close(ctx)
+
+	require.NoError(t, st.ListBlobs(ctx, "", func(bm blob.Metadata) error {
+		return nil
+	}))
+
+	osi.readDirRemainingNonRetriableErrors = 1
+
+	require.ErrorIs(t, st.ListBlobs(ctx, "", func(bm blob.Metadata) error {
+		return nil
+	}), errNonRetriable)
+
+	osi.readDirRemainingFatalDirEntry = 1
+
+	require.ErrorIs(t, st.ListBlobs(ctx, "", func(bm blob.Metadata) error {
+		return nil
+	}), errNonRetriable)
+}
+
+func TestFileStorage_TouchBlob_ErrorHandling(t *testing.T) {
+	t.Parallel()
+
+	ctx := testlogging.Context(t)
+
+	dataDir := testutil.TempDirectory(t)
+
+	osi := &mockOS{
+		osInterface: realOS{},
+	}
+
+	st, err := New(ctx, &Options{
+		Path: dataDir,
+		Options: sharded.Options{
+			DirectoryShards: []int{5, 2},
+		},
+	}, true)
+	require.NoError(t, err)
+
+	st.(*fsStorage).Impl.(*fsImpl).osi = osi
+
+	defer st.Close(ctx)
+
+	require.NoError(t, st.PutBlob(ctx, "someblob1234567812345678", gather.FromSlice([]byte{1, 2, 3}), blob.PutOptions{}))
+
+	osi.statRemainingErrors = 1
+
+	require.NoError(t, st.(*fsStorage).TouchBlob(ctx, "someblob1234567812345678", 0))
+}
+
+func TestFileStorage_Misc(t *testing.T) {
+	t.Parallel()
+
+	ctx := testlogging.Context(t)
+
+	dataDir := testutil.TempDirectory(t)
+
+	st, err := New(ctx, &Options{
+		Path: dataDir,
+		Options: sharded.Options{
+			DirectoryShards: []int{5, 2},
+		},
+	}, true)
+	require.NoError(t, err)
+
+	defer st.Close(ctx)
+
+	require.NoError(t, st.FlushCaches(ctx)) // this does nothing
+	require.Equal(t, st.DisplayName(), "Filesystem: "+dataDir)
+}
+
 func verifyBlobTimestampOrder(t *testing.T, st blob.Storage, want ...blob.ID) {
 	t.Helper()
 
diff --git a/repo/blob/filesystem/osinterface_mock_test.go b/repo/blob/filesystem/osinterface_mock_test.go
new file mode 100644
index 00000000..0256b79b
--- /dev/null
+++ b/repo/blob/filesystem/osinterface_mock_test.go
@@ -0,0 +1,174 @@
+package filesystem
+
+import (
+	"io/fs"
+	"os"
+	"sync/atomic"
+
+	"github.com/google/uuid"
+	"github.com/pkg/errors"
+)
+
+var errNonRetriable = errors.Errorf("some non-retriable error")
+
+type mockOS struct {
+	readFileRemainingErrors             int32
+	writeFileRemainingErrors            int32
+	writeFileCloseRemainingErrors       int32
+	createNewFileRemainingErrors        int32
+	mkdirAllRemainingErrors             int32
+	renameRemainingErrors               int32
+	removeRemainingRetriableErrors      int32
+	removeRemainingNonRetriableErrors   int32
+	chownRemainingErrors                int32
+	readDirRemainingErrors              int32
+	readDirRemainingNonRetriableErrors  int32
+	readDirRemainingFileDeletedDirEntry int32
+	readDirRemainingFatalDirEntry       int32
+	statRemainingErrors                 int32
+
+	effectiveUID int
+
+	osInterface
+}
+
+func (osi *mockOS) Open(fname string) (osReadFile, error) {
+	rf, err := osi.osInterface.Open(fname)
+	if err != nil {
+		return nil, err
+	}
+
+	if atomic.AddInt32(&osi.readFileRemainingErrors, -1) >= 0 {
+		return readFailureFile{rf}, nil
+	}
+
+	return rf, nil
+}
+
+func (osi *mockOS) Rename(oldname, newname string) error {
+	if atomic.AddInt32(&osi.renameRemainingErrors, -1) >= 0 {
+		return &os.LinkError{Op: "rename", Old: oldname, New: newname, Err: errors.Errorf("underlying problem")}
+	}
+
+	return osi.osInterface.Rename(oldname, newname)
+}
+
+func (osi *mockOS) ReadDir(dirname string) ([]fs.DirEntry, error) {
+	if atomic.AddInt32(&osi.readDirRemainingErrors, -1) >= 0 {
+		return nil, &os.PathError{Op: "readdir", Err: errors.Errorf("underlying problem")}
+	}
+
+	if atomic.AddInt32(&osi.readDirRemainingNonRetriableErrors, -1) >= 0 {
+		return nil, errNonRetriable
+	}
+
+	ent, err := osi.osInterface.ReadDir(dirname)
+
+	if atomic.AddInt32(&osi.readDirRemainingFileDeletedDirEntry, -1) >= 0 {
+		// add a dir entry which will fail at Info() time
+		_, noSuchFileErr := os.Open(uuid.NewString())
+
+		ent = append(ent, &mockDirEntryInfoError{nil, noSuchFileErr})
+	}
+
+	if atomic.AddInt32(&osi.readDirRemainingFatalDirEntry, -1) >= 0 {
+		ent = append(ent, &mockDirEntryInfoError{nil, errNonRetriable})
+	}
+
+	return ent, err
+}
+
+func (osi *mockOS) Remove(fname string) error {
+	if atomic.AddInt32(&osi.removeRemainingRetriableErrors, -1) >= 0 {
+		return &os.PathError{Op: "unlink", Err: errors.Errorf("underlying problem")}
+	}
+
+	if atomic.AddInt32(&osi.removeRemainingNonRetriableErrors, -1) >= 0 {
+		return errNonRetriable
+	}
+
+	return osi.osInterface.Remove(fname)
+}
+
+func (osi *mockOS) Stat(fname string) (fs.FileInfo, error) {
+	if atomic.AddInt32(&osi.statRemainingErrors, -1) >= 0 {
+		return nil, &os.PathError{Op: "stat", Err: errors.Errorf("underlying problem")}
+	}
+
+	return osi.osInterface.Stat(fname)
+}
+
+func (osi *mockOS) Chown(fname string, uid, gid int) error {
+	if atomic.AddInt32(&osi.chownRemainingErrors, -1) >= 0 {
+		return &os.PathError{Op: "chown", Err: errors.Errorf("underlying problem")}
+	}
+
+	return osi.osInterface.Chown(fname, uid, gid)
+}
+
+func (osi *mockOS) CreateNewFile(fname string, perm os.FileMode) (osWriteFile, error) {
+	if atomic.AddInt32(&osi.createNewFileRemainingErrors, -1) >= 0 {
+		return nil, &os.PathError{Op: "create", Err: errors.Errorf("underlying problem")}
+	}
+
+	wf, err := osi.osInterface.CreateNewFile(fname, perm)
+	if err != nil {
+		return nil, err
+	}
+
+	if atomic.AddInt32(&osi.writeFileRemainingErrors, -1) >= 0 {
+		return writeFailureFile{wf}, nil
+	}
+
+	if atomic.AddInt32(&osi.writeFileCloseRemainingErrors, -1) >= 0 {
+		return writeCloseFailureFile{wf}, nil
+	}
+
+	return wf, nil
+}
+
+func (osi *mockOS) MkdirAll(fname string, mode os.FileMode) error {
+	if atomic.AddInt32(&osi.mkdirAllRemainingErrors, -1) >= 0 {
+		return &os.PathError{Op: "mkdir", Err: errors.Errorf("underlying problem")}
+	}
+
+	return osi.osInterface.MkdirAll(fname, mode)
+}
+
+func (osi *mockOS) Geteuid() int {
+	return osi.effectiveUID
+}
+
+type readFailureFile struct {
+	osReadFile
+}
+
+func (f readFailureFile) Read(b []byte) (int, error) {
+	return 0, &os.PathError{Op: "read", Err: errors.Errorf("underlying problem")}
+}
+
+type writeFailureFile struct {
+	osWriteFile
+}
+
+func (f writeFailureFile) Write(b []byte) (int, error) {
+	return 0, &os.PathError{Op: "write", Err: errors.Errorf("underlying problem")}
+}
+
+type writeCloseFailureFile struct {
+	osWriteFile
+}
+
+func (f writeCloseFailureFile) Close() error {
+	return &os.PathError{Op: "close", Err: errors.Errorf("underlying problem")}
+}
+
+type mockDirEntryInfoError struct {
+	fs.DirEntry
+
+	err error
+}
+
+func (e mockDirEntryInfoError) Info() (fs.FileInfo, error) {
+	return nil, e.err
+}