Repository: kopia/kopia
Pull request #4218 (2024-10-31 14:48:13)

feat(cli): introduce adaptive estimation mechanism
Currently, to estimate amount of data to be processed, we need to iterate over all the data. This slows down hashing and also can take too much time when we have big amount of files.

This PR brings an improvement, now it's possible to use rough estimation, which is usable when we are snapshotting whole volume. In such case we can get filesystem stats (as `df` does) and use amount of data / number of files for estimation.

This new mechanism is available under the flag `./kopia snapshot create /some/path --progress-estimation-type=rough`

When using this flag, user needs to understand, that in such case any ignore rules are not taken into account and estimation could be a bit inaccurate.

In some cases it may worth to fallback to classic estimation to have more precise estimation of data to be uploaded.

This "adaptive" mechanism will also be available under the flag `kopia snapshot create /some-path --progress-estimation-type=adaptive --adaptive-estimation-threshold=10000`

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/kopia/kopia/pull/4218?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) Report
Attention: Patch coverage is `85.97561%` with `23 lines` in your changes missing coverage. Please review.
> Project coverage is 76.07%. Comparing base [(`cb455c6`)](https://app.codecov.io/gh/kopia/kopia/commit/cb455c699a04e54cb47e25a676d7d65063ce883e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) to head [(`0a818fc`)](https://app.codecov.io/gh/kopia/kopia/commit/0a818fc99af11680d68308b28bad06830174d4a0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia).
> Report is 327 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/kopia/kopia/pull/4218?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) | Patch % | Lines |
|---|---|---|
| [internal/volumesizeinfo/volume\_size\_info\_common.go](https://app.codecov.io/gh/kopia/kopia/pull/4218?src=pr&el=tree&filepath=internal%2Fvolumesizeinfo%2Fvolume_size_info_common.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvdm9sdW1lc2l6ZWluZm8vdm9sdW1lX3NpemVfaW5mb19jb21tb24uZ28=) | 0.00% | [13 Missing :warning: ](https://app.codecov.io/gh/kopia/kopia/pull/4218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) |
| [internal/volumesizeinfo/volume\_size\_info.go](https://app.codecov.io/gh/kopia/kopia/pull/4218?src=pr&el=tree&filepath=internal%2Fvolumesizeinfo%2Fvolume_size_info.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-aW50ZXJuYWwvdm9sdW1lc2l6ZWluZm8vdm9sdW1lX3NpemVfaW5mby5nbw==) | 44.44% | [5 Missing :warning: ](https://app.codecov.io/gh/kopia/kopia/pull/4218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) |
| [snapshot/snapshotfs/upload\_estimator.go](https://app.codecov.io/gh/kopia/kopia/pull/4218?src=pr&el=tree&filepath=snapshot%2Fsnapshotfs%2Fupload_estimator.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-c25hcHNob3Qvc25hcHNob3Rmcy91cGxvYWRfZXN0aW1hdG9yLmdv) | 96.47% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/kopia/kopia/pull/4218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) |
| [snapshot/snapshotfs/upload\_progress.go](https://app.codecov.io/gh/kopia/kopia/pull/4218?src=pr&el=tree&filepath=snapshot%2Fsnapshotfs%2Fupload_progress.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia#diff-c25hcHNob3Qvc25hcHNob3Rmcy91cGxvYWRfcHJvZ3Jlc3MuZ28=) | 81.81% | [2 Missing :warning: ](https://app.codecov.io/gh/kopia/kopia/pull/4218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #4218      +/-   ##
==========================================
+ Coverage   75.86%   76.07%   +0.20%     
==========================================
  Files         470      506      +36     
  Lines       37301    38828    +1527     
==========================================
+ Hits        28299    29537    +1238     
- Misses       7071     7329     +258     
- Partials     1931     1962      +31     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/kopia/kopia/pull/4218?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kopia).

jkowalski: I like the approach, I think we can merge this PR as-is, but we should follow-up with a PR that adds policy-level settings for this.

In general there are few more policy-level settings which could control the estimation. Some form of initial delay (to avoid estimation altogether on trivially small directories), throttling (to slow down estimation to give hashing more priority), etc come to mind.
jkowalski: @julio-lopez can you take a look and merge the PR if you're ok with that?

Patch:
diff --git a/.github/workflows/auto-merge.yml b/.github/workflows/auto-merge.yml
index 63cb0719..991cf954 100644
--- a/.github/workflows/auto-merge.yml
+++ b/.github/workflows/auto-merge.yml
@@ -7,7 +7,7 @@ jobs:
   auto-merge:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       - uses: ahmadnassri/action-dependabot-auto-merge@v2
         with:
           # auto-merge rules are in /.github/auto-merge.yml
diff --git a/.github/workflows/code-coverage.yml b/.github/workflows/code-coverage.yml
index 6d9db015..2cd33693 100644
--- a/.github/workflows/code-coverage.yml
+++ b/.github/workflows/code-coverage.yml
@@ -12,11 +12,11 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
@@ -28,7 +28,7 @@ jobs:
       with:
         files: coverage.txt
     - name: Upload Logs
-      uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+      uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
       with:
         name: logs
         path: .logs/**/*.log
diff --git a/.github/workflows/dependency-review.yml b/.github/workflows/dependency-review.yml
index 13b83009..f0ebeed4 100644
--- a/.github/workflows/dependency-review.yml
+++ b/.github/workflows/dependency-review.yml
@@ -15,6 +15,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: 'Checkout Repository'
-        uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       - name: 'Dependency Review'
         uses: actions/dependency-review-action@v4
diff --git a/.github/workflows/license-check.yml b/.github/workflows/license-check.yml
index d87b782e..6f857411 100644
--- a/.github/workflows/license-check.yml
+++ b/.github/workflows/license-check.yml
@@ -12,11 +12,11 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
index f5b92393..9f9bf0f7 100644
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
@@ -26,11 +26,11 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
diff --git a/.github/workflows/make.yml b/.github/workflows/make.yml
index f6c46d5a..9ecea9a3 100644
--- a/.github/workflows/make.yml
+++ b/.github/workflows/make.yml
@@ -40,11 +40,11 @@ jobs:
     continue-on-error: ${{ contains(matrix.os, 'self-hosted') }}
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
@@ -101,7 +101,7 @@ jobs:
         # macOS signing certificate (base64-encoded), used by Electron Builder
         MACOS_SIGNING_IDENTITY: ${{ secrets.MACOS_SIGNING_IDENTITY }}
     - name: Upload Kopia Artifacts
-      uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+      uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
       with:
         name: kopia-${{ matrix.os }}
         path: |
@@ -123,7 +123,7 @@ jobs:
         if-no-files-found: ignore
       if: ${{ !contains(matrix.os, 'self-hosted') }}
     - name: Upload Kopia Binary
-      uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+      uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
       with:
         name: kopia_binaries-${{ matrix.os }}
         path: |
@@ -139,7 +139,7 @@ jobs:
     needs: build
     if: github.event_name != 'pull_request' && github.repository == 'kopia/kopia'
     steps:
-    - uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
     - name: Set up QEMU
       uses: docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf # v3.2.0
     - name: Set up Docker Buildx
diff --git a/.github/workflows/ossf-scorecard.yml b/.github/workflows/ossf-scorecard.yml
index a9cd235e..c3212811 100644
--- a/.github/workflows/ossf-scorecard.yml
+++ b/.github/workflows/ossf-scorecard.yml
@@ -26,7 +26,7 @@ jobs:
     steps:
       -
         name: "Checkout repo"
-        uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
         with:
           persist-credentials: false
       -
@@ -39,12 +39,12 @@ jobs:
       -
         # Upload the results to GitHub's code scanning dashboard.
         name: "Upload to results to dashboard"
-        uses: github/codeql-action/upload-sarif@e2b3eafc8d227b0241d48be5f425d47c2d750a13 # v3.26.10
+        uses: github/codeql-action/upload-sarif@662472033e021d55d94146f66f6058822b0b39fd # v3.27.0
         with:
           sarif_file: results.sarif
       -
         name: "Upload analysis results as 'Job Artifact'"
-        uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+        uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
         with:
           name: SARIF file
           path: results.sarif
diff --git a/.github/workflows/providers-core.yml b/.github/workflows/providers-core.yml
index 189d467d..a96299ee 100644
--- a/.github/workflows/providers-core.yml
+++ b/.github/workflows/providers-core.yml
@@ -24,12 +24,12 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
         ref: ${{ github.event.inputs.ref_name || github.ref }}
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
diff --git a/.github/workflows/providers-extra.yml b/.github/workflows/providers-extra.yml
index 2c87ee00..79d09917 100644
--- a/.github/workflows/providers-extra.yml
+++ b/.github/workflows/providers-extra.yml
@@ -24,12 +24,12 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
         ref: ${{ github.event.inputs.ref_name || github.ref }}
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
diff --git a/.github/workflows/race-detector.yml b/.github/workflows/race-detector.yml
index af70976f..658c4ee1 100644
--- a/.github/workflows/race-detector.yml
+++ b/.github/workflows/race-detector.yml
@@ -12,11 +12,11 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
diff --git a/cli/cli_progress.go b/cli/cli_progress.go
index fa5fda5d..7ff6904d 100644
--- a/cli/cli_progress.go
+++ b/cli/cli_progress.go
@@ -2,6 +2,7 @@
 
 import (
 	"fmt"
+	"strconv"
 	"strings"
 	"sync"
 	"sync/atomic"
@@ -20,14 +21,19 @@
 )
 
 type progressFlags struct {
-	enableProgress         bool
-	progressUpdateInterval time.Duration
-	out                    textOutput
+	enableProgress              bool
+	progressEstimationType      string
+	adaptiveEstimationThreshold int64
+	progressUpdateInterval      time.Duration
+	out                         textOutput
 }
 
 func (p *progressFlags) setup(svc appServices, app *kingpin.Application) {
 	app.Flag("progress", "Enable progress bar").Hidden().Default("true").BoolVar(&p.enableProgress)
+	app.Flag("progress-estimation-type", "Set type of estimation of the data to be snapshotted").Hidden().Default(snapshotfs.EstimationTypeClassic).
+		EnumVar(&p.progressEstimationType, snapshotfs.EstimationTypeClassic, snapshotfs.EstimationTypeRough, snapshotfs.EstimationTypeAdaptive)
 	app.Flag("progress-update-interval", "How often to update progress information").Hidden().Default("300ms").DurationVar(&p.progressUpdateInterval)
+	app.Flag("adaptive-estimation-threshold", "Sets the threshold below which the classic estimation method will be used").Hidden().Default(strconv.FormatInt(snapshotfs.AdaptiveEstimationThreshold, 10)).Int64Var(&p.adaptiveEstimationThreshold)
 	p.out.setup(svc)
 }
 
@@ -57,7 +63,7 @@ type cliProgress struct {
 
 	uploadStartTime timetrack.Estimator // +checklocksignore
 
-	estimatedFileCount  int   // +checklocksignore
+	estimatedFileCount  int64 // +checklocksignore
 	estimatedTotalBytes int64 // +checklocksignore
 
 	// indicates shared instance that does not reset counters at the beginning of upload.
@@ -66,6 +72,11 @@ type cliProgress struct {
 	progressFlags
 }
 
+// Enabled returns true when progress is enabled.
+func (p *cliProgress) Enabled() bool {
+	return p.enableProgress
+}
+
 func (p *cliProgress) HashingFile(_ string) {
 	p.inProgressHashing.Add(1)
 }
@@ -226,7 +237,7 @@ func (p *cliProgress) UploadStarted() {
 	p.uploading.Store(true)
 }
 
-func (p *cliProgress) EstimatedDataSize(fileCount int, totalBytes int64) {
+func (p *cliProgress) EstimatedDataSize(fileCount, totalBytes int64) {
 	if p.shared {
 		// do nothing
 		return
@@ -259,4 +270,11 @@ func (p *cliProgress) Finish() {
 	}
 }
 
+func (p *cliProgress) EstimationParameters() snapshotfs.EstimationParameters {
+	return snapshotfs.EstimationParameters{
+		Type:              p.progressEstimationType,
+		AdaptiveThreshold: p.adaptiveEstimationThreshold,
+	}
+}
+
 var _ snapshotfs.UploadProgress = (*cliProgress)(nil)
diff --git a/go.mod b/go.mod
index d5154464..cc63157f 100644
--- a/go.mod
+++ b/go.mod
@@ -3,7 +3,7 @@ module github.com/kopia/kopia
 go 1.22.7
 
 require (
-	cloud.google.com/go/storage v1.45.0
+	cloud.google.com/go/storage v1.46.0
 	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0
 	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.8.0
 	github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.4.1
@@ -18,7 +18,7 @@ require (
 	github.com/fatih/color v1.18.0
 	github.com/foomo/htpasswd v0.0.0-20200116085101-e3a90e78da9c
 	github.com/gofrs/flock v0.12.1
-	github.com/golang-jwt/jwt/v4 v4.5.0
+	github.com/golang-jwt/jwt/v4 v4.5.1
 	github.com/google/fswalker v0.3.3
 	github.com/google/go-cmp v0.6.0
 	github.com/google/uuid v1.6.0
@@ -65,7 +65,7 @@ require (
 	golang.org/x/sys v0.26.0
 	golang.org/x/term v0.25.0
 	golang.org/x/text v0.19.0
-	google.golang.org/api v0.203.0
+	google.golang.org/api v0.204.0
 	google.golang.org/grpc v1.67.1
 	google.golang.org/protobuf v1.35.1
 	gopkg.in/kothar/go-backblaze.v0 v0.0.0-20210124194846-35409b867216
@@ -75,8 +75,8 @@ require (
 	al.essio.dev/pkg/shellescape v1.5.1 // indirect
 	cel.dev/expr v0.16.1 // indirect
 	cloud.google.com/go v0.116.0 // indirect
-	cloud.google.com/go/auth v0.9.9 // indirect
-	cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect
+	cloud.google.com/go/auth v0.10.0 // indirect
+	cloud.google.com/go/auth/oauth2adapt v0.2.5 // indirect
 	cloud.google.com/go/compute/metadata v0.5.2 // indirect
 	cloud.google.com/go/iam v1.2.1 // indirect
 	cloud.google.com/go/monitoring v1.21.1 // indirect
@@ -141,9 +141,9 @@ require (
 	go.opentelemetry.io/otel/sdk/metric v1.29.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.3.1 // indirect
 	golang.org/x/time v0.7.0 // indirect
-	google.golang.org/genproto v0.0.0-20241015192408-796eee8c2d53 // indirect
-	google.golang.org/genproto/googleapis/api v0.0.0-20241007155032-5fefd90f89a9 // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20241015192408-796eee8c2d53 // indirect
+	google.golang.org/genproto v0.0.0-20241021214115-324edc3d5d38 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20241015192408-796eee8c2d53 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20241021214115-324edc3d5d38 // indirect
 	google.golang.org/grpc/stats/opentelemetry v0.0.0-20240907200651-3ffb98b2c93a // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
diff --git a/go.sum b/go.sum
index da9e95d3..75299049 100644
--- a/go.sum
+++ b/go.sum
@@ -5,22 +5,22 @@ cel.dev/expr v0.16.1/go.mod h1:AsGA5zb3WruAEQeQng1RZdGEXmBj0jvMWh6l5SnNuC8=
 cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 cloud.google.com/go v0.116.0 h1:B3fRrSDkLRt5qSHWe40ERJvhvnQwdZiHu0bJOpldweE=
 cloud.google.com/go v0.116.0/go.mod h1:cEPSRWPzZEswwdr9BxE6ChEn01dWlTaF05LiC2Xs70U=
-cloud.google.com/go/auth v0.9.9 h1:BmtbpNQozo8ZwW2t7QJjnrQtdganSdmqeIBxHxNkEZQ=
-cloud.google.com/go/auth v0.9.9/go.mod h1:xxA5AqpDrvS+Gkmo9RqrGGRh6WSNKKOXhY3zNOr38tI=
-cloud.google.com/go/auth/oauth2adapt v0.2.4 h1:0GWE/FUsXhf6C+jAkWgYm7X9tK8cuEIfy19DBn6B6bY=
-cloud.google.com/go/auth/oauth2adapt v0.2.4/go.mod h1:jC/jOpwFP6JBxhB3P5Rr0a9HLMC/Pe3eaL4NmdvqPtc=
+cloud.google.com/go/auth v0.10.0 h1:tWlkvFAh+wwTOzXIjrwM64karR1iTBZ/GRr0S/DULYo=
+cloud.google.com/go/auth v0.10.0/go.mod h1:xxA5AqpDrvS+Gkmo9RqrGGRh6WSNKKOXhY3zNOr38tI=
+cloud.google.com/go/auth/oauth2adapt v0.2.5 h1:2p29+dePqsCHPP1bqDJcKj4qxRyYCcbzKpFyKGt3MTk=
+cloud.google.com/go/auth/oauth2adapt v0.2.5/go.mod h1:AlmsELtlEBnaNTL7jCj8VQFLy6mbZv0s4Q7NGBeQ5E8=
 cloud.google.com/go/compute/metadata v0.5.2 h1:UxK4uu/Tn+I3p2dYWTfiX4wva7aYlKixAHn3fyqngqo=
 cloud.google.com/go/compute/metadata v0.5.2/go.mod h1:C66sj2AluDcIqakBq/M8lw8/ybHgOZqin2obFxa/E5k=
 cloud.google.com/go/iam v1.2.1 h1:QFct02HRb7H12J/3utj0qf5tobFh9V4vR6h9eX5EBRU=
 cloud.google.com/go/iam v1.2.1/go.mod h1:3VUIJDPpwT6p/amXRC5GY8fCCh70lxPygguVtI0Z4/g=
-cloud.google.com/go/logging v1.11.0 h1:v3ktVzXMV7CwHq1MBF65wcqLMA7i+z3YxbUsoK7mOKs=
-cloud.google.com/go/logging v1.11.0/go.mod h1:5LDiJC/RxTt+fHc1LAt20R9TKiUTReDg6RuuFOZ67+A=
+cloud.google.com/go/logging v1.12.0 h1:ex1igYcGFd4S/RZWOCU51StlIEuey5bjqwH9ZYjHibk=
+cloud.google.com/go/logging v1.12.0/go.mod h1:wwYBt5HlYP1InnrtYI0wtwttpVU1rifnMT7RejksUAM=
 cloud.google.com/go/longrunning v0.6.1 h1:lOLTFxYpr8hcRtcwWir5ITh1PAKUD/sG2lKrTSYjyMc=
 cloud.google.com/go/longrunning v0.6.1/go.mod h1:nHISoOZpBcmlwbJmiVk5oDRz0qG/ZxPynEGs1iZ79s0=
 cloud.google.com/go/monitoring v1.21.1 h1:zWtbIoBMnU5LP9A/fz8LmWMGHpk4skdfeiaa66QdFGc=
 cloud.google.com/go/monitoring v1.21.1/go.mod h1:Rj++LKrlht9uBi8+Eb530dIrzG/cU/lB8mt+lbeFK1c=
-cloud.google.com/go/storage v1.45.0 h1:5av0QcIVj77t+44mV4gffFC/LscFRUhto6UBMB5SimM=
-cloud.google.com/go/storage v1.45.0/go.mod h1:wpPblkIuMP5jCB/E48Pz9zIo2S/zD8g+ITmxKkPCITE=
+cloud.google.com/go/storage v1.46.0 h1:OTXISBpFd8KaA2ClT3K3oRk8UGOcTHtrZ1bW88xKiic=
+cloud.google.com/go/storage v1.46.0/go.mod h1:lM+gMAW91EfXIeMTBmixRsKL/XCxysytoAgduVikjMk=
 cloud.google.com/go/trace v1.11.1 h1:UNqdP+HYYtnm6lb91aNA5JQ0X14GnxkABGlfz2PzPew=
 cloud.google.com/go/trace v1.11.1/go.mod h1:IQKNQuBzH72EGaXEodKlNJrWykGZxet2zgjtS60OtjA=
 github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 h1:JZg6HRh6W6U4OLl6lk7BZ7BLisIzM9dG1R50zUk9C/M=
@@ -136,8 +136,8 @@ github.com/godbus/dbus/v5 v5.1.0 h1:4KLkAxT3aOY8Li4FRJe/KvhoNFFxo0m6fNuFUO8QJUk=
 github.com/godbus/dbus/v5 v5.1.0/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
 github.com/gofrs/flock v0.12.1 h1:MTLVXXHf8ekldpJk3AKicLij9MdwOWkZ+a/jHHZby9E=
 github.com/gofrs/flock v0.12.1/go.mod h1:9zxTsyu5xtJ9DK+1tFZyibEV7y3uwDxPPfbxeeHCoD0=
-github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg=
-github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
+github.com/golang-jwt/jwt/v4 v4.5.1 h1:JdqV9zKUdtaa9gdPlywC3aeoEsR681PlKC+4F5gQgeo=
+github.com/golang-jwt/jwt/v4 v4.5.1/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
 github.com/golang-jwt/jwt/v5 v5.2.1 h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
 github.com/golang-jwt/jwt/v5 v5.2.1/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
@@ -429,19 +429,19 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc
 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-google.golang.org/api v0.203.0 h1:SrEeuwU3S11Wlscsn+LA1kb/Y5xT8uggJSkIhD08NAU=
-google.golang.org/api v0.203.0/go.mod h1:BuOVyCSYEPwJb3npWvDnNmFI92f3GeRnHNkETneT3SI=
+google.golang.org/api v0.204.0 h1:3PjmQQEDkR/ENVZZwIYB4W/KzYtN8OrqnNcHWpeR8E4=
+google.golang.org/api v0.204.0/go.mod h1:69y8QSoKIbL9F94bWgWAq6wGqGwyjBgi2y8rAK8zLag=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
 google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
 google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
-google.golang.org/genproto v0.0.0-20241015192408-796eee8c2d53 h1:Df6WuGvthPzc+JiQ/G+m+sNX24kc0aTBqoDN/0yyykE=
-google.golang.org/genproto v0.0.0-20241015192408-796eee8c2d53/go.mod h1:fheguH3Am2dGp1LfXkrvwqC/KlFq8F0nLq3LryOMrrE=
-google.golang.org/genproto/googleapis/api v0.0.0-20241007155032-5fefd90f89a9 h1:T6rh4haD3GVYsgEfWExoCZA2o2FmbNyKpTuAxbEFPTg=
-google.golang.org/genproto/googleapis/api v0.0.0-20241007155032-5fefd90f89a9/go.mod h1:wp2WsuBYj6j8wUdo3ToZsdxxixbvQNAHqVJrTgi5E5M=
-google.golang.org/genproto/googleapis/rpc v0.0.0-20241015192408-796eee8c2d53 h1:X58yt85/IXCx0Y3ZwN6sEIKZzQtDEYaBWrDvErdXrRE=
-google.golang.org/genproto/googleapis/rpc v0.0.0-20241015192408-796eee8c2d53/go.mod h1:GX3210XPVPUjJbTUbvwI8f2IpZDMZuPJWDzDuebbviI=
+google.golang.org/genproto v0.0.0-20241021214115-324edc3d5d38 h1:Q3nlH8iSQSRUwOskjbcSMcF2jiYMNiQYZ0c2KEJLKKU=
+google.golang.org/genproto v0.0.0-20241021214115-324edc3d5d38/go.mod h1:xBI+tzfqGGN2JBeSebfKXFSdBpWVQ7sLW40PTupVRm4=
+google.golang.org/genproto/googleapis/api v0.0.0-20241015192408-796eee8c2d53 h1:fVoAXEKA4+yufmbdVYv+SE73+cPZbbbe8paLsHfkK+U=
+google.golang.org/genproto/googleapis/api v0.0.0-20241015192408-796eee8c2d53/go.mod h1:riSXTwQ4+nqmPGtobMFyW5FqVAmIs0St6VPp4Ug7CE4=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20241021214115-324edc3d5d38 h1:zciRKQ4kBpFgpfC5QQCVtnnNAcLIqweL7plyZRQHVpI=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20241021214115-324edc3d5d38/go.mod h1:GX3210XPVPUjJbTUbvwI8f2IpZDMZuPJWDzDuebbviI=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
 google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
diff --git a/internal/server/source_manager.go b/internal/server/source_manager.go
index ec0dfbfd..4973c8e7 100644
--- a/internal/server/source_manager.go
+++ b/internal/server/source_manager.go
@@ -477,6 +477,11 @@ func (t *uitaskProgress) maybeReport() {
 	}
 }
 
+// Enabled implements UploadProgress, always returns true.
+func (t *uitaskProgress) Enabled() bool {
+	return true
+}
+
 // UploadStarted is emitted once at the start of an upload.
 func (t *uitaskProgress) UploadStarted() {
 	t.p.UploadStarted()
@@ -558,11 +563,16 @@ func (t *uitaskProgress) ExcludedDir(dirname string) {
 }
 
 // EstimatedDataSize is emitted whenever the size of upload is estimated.
-func (t *uitaskProgress) EstimatedDataSize(fileCount int, totalBytes int64) {
+func (t *uitaskProgress) EstimatedDataSize(fileCount, totalBytes int64) {
 	t.p.EstimatedDataSize(fileCount, totalBytes)
 	t.maybeReport()
 }
 
+// EstimationParameters returns parameters to be used for estimation.
+func (t *uitaskProgress) EstimationParameters() snapshotfs.EstimationParameters {
+	return t.p.EstimationParameters()
+}
+
 func newSourceManager(src snapshot.SourceInfo, server *Server, rep repo.Repository) *sourceManager {
 	m := &sourceManager{
 		src:              src,
diff --git a/internal/volumesizeinfo/volume_size_info.go b/internal/volumesizeinfo/volume_size_info.go
new file mode 100644
index 00000000..7cc3c938
--- /dev/null
+++ b/internal/volumesizeinfo/volume_size_info.go
@@ -0,0 +1,28 @@
+// Package volumesizeinfo contains helpers to obtain information about volume.
+package volumesizeinfo
+
+import (
+	"github.com/pkg/errors"
+)
+
+// VolumeSizeInfo keeps information about volume (total volume size, used size and number of files).
+type VolumeSizeInfo struct {
+	TotalSize  uint64
+	UsedSize   uint64
+	FilesCount uint64
+}
+
+// GetVolumeSizeInfo returns VolumeSizeInfo for given mount point.
+// FilesCount on Windows it always set to MaxInt64.
+func GetVolumeSizeInfo(volumeMountPoint string) (VolumeSizeInfo, error) {
+	if volumeMountPoint == "" {
+		return VolumeSizeInfo{}, errors.Errorf("volume mount point cannot be empty")
+	}
+
+	sizeInfo, err := getPlatformVolumeSizeInfo(volumeMountPoint)
+	if err != nil {
+		return VolumeSizeInfo{}, errors.Wrapf(err, "Unable to get volume size info for mount point %q", volumeMountPoint)
+	}
+
+	return sizeInfo, nil
+}
diff --git a/internal/volumesizeinfo/volume_size_info_common.go b/internal/volumesizeinfo/volume_size_info_common.go
new file mode 100644
index 00000000..fa40c144
--- /dev/null
+++ b/internal/volumesizeinfo/volume_size_info_common.go
@@ -0,0 +1,23 @@
+//go:build !openbsd && !windows
+
+package volumesizeinfo
+
+import (
+	"golang.org/x/sys/unix"
+)
+
+func getPlatformVolumeSizeInfo(volumeMountPoint string) (VolumeSizeInfo, error) {
+	stats := unix.Statfs_t{}
+
+	err := unix.Statfs(volumeMountPoint, &stats)
+	if err != nil {
+		return VolumeSizeInfo{}, err //nolint:wrapcheck
+	}
+
+	return VolumeSizeInfo{
+		TotalSize: stats.Blocks * uint64(stats.Bsize),                 //nolint:unconvert,nolintlint
+		UsedSize:  (stats.Blocks - stats.Bfree) * uint64(stats.Bsize), //nolint:unconvert,nolintlint
+		// Conversion to uint64 is needed for some arch/distrib combination.
+		FilesCount: stats.Files - uint64(stats.Ffree), //nolint:unconvert,nolintlint
+	}, nil
+}
diff --git a/internal/volumesizeinfo/volume_size_info_openbsd.go b/internal/volumesizeinfo/volume_size_info_openbsd.go
new file mode 100644
index 00000000..a4d767bb
--- /dev/null
+++ b/internal/volumesizeinfo/volume_size_info_openbsd.go
@@ -0,0 +1,22 @@
+//go:build openbsd
+
+package volumesizeinfo
+
+import (
+	"golang.org/x/sys/unix"
+)
+
+func getPlatformVolumeSizeInfo(volumeMountPoint string) (VolumeSizeInfo, error) {
+	stats := unix.Statfs_t{}
+
+	err := unix.Statfs(volumeMountPoint, &stats)
+	if err != nil {
+		return VolumeSizeInfo{}, err //nolint:wrapcheck
+	}
+
+	return VolumeSizeInfo{
+		TotalSize:  stats.F_blocks * uint64(stats.F_bsize),
+		UsedSize:   (stats.F_blocks - stats.F_bfree) * uint64(stats.F_bsize),
+		FilesCount: stats.F_files - stats.F_ffree,
+	}, nil
+}
diff --git a/internal/volumesizeinfo/volume_size_info_windows.go b/internal/volumesizeinfo/volume_size_info_windows.go
new file mode 100644
index 00000000..133299c8
--- /dev/null
+++ b/internal/volumesizeinfo/volume_size_info_windows.go
@@ -0,0 +1,31 @@
+//go:build windows
+
+package volumesizeinfo
+
+import (
+	"math"
+
+	"golang.org/x/sys/windows"
+
+	"github.com/kopia/kopia/repo/blob"
+)
+
+func getPlatformVolumeSizeInfo(volumeMountPoint string) (VolumeSizeInfo, error) {
+	var c blob.Capacity
+
+	pathPtr, err := windows.UTF16PtrFromString(volumeMountPoint)
+	if err != nil {
+		return VolumeSizeInfo{}, err //nolint:wrapcheck
+	}
+
+	err = windows.GetDiskFreeSpaceEx(pathPtr, nil, &c.SizeB, &c.FreeB)
+	if err != nil {
+		return VolumeSizeInfo{}, err //nolint:wrapcheck
+	}
+
+	return VolumeSizeInfo{
+		TotalSize:  c.SizeB,
+		UsedSize:   c.SizeB - c.FreeB,
+		FilesCount: uint64(math.MaxInt64), // On Windows it's not possible to get / estimate number of files on volume
+	}, nil
+}
diff --git a/snapshot/snapshotfs/estimate.go b/snapshot/snapshotfs/estimate.go
index 1f79c793..4950b159 100644
--- a/snapshot/snapshotfs/estimate.go
+++ b/snapshot/snapshotfs/estimate.go
@@ -9,10 +9,13 @@
 	"github.com/kopia/kopia/fs"
 	"github.com/kopia/kopia/fs/ignorefs"
 	"github.com/kopia/kopia/internal/units"
+	"github.com/kopia/kopia/repo/logging"
 	"github.com/kopia/kopia/snapshot"
 	"github.com/kopia/kopia/snapshot/policy"
 )
 
+var estimateLog = logging.Module("estimate")
+
 // SampleBucket keeps track of count and total size of files above in certain size range and
 // includes small number of examples of such files.
 type SampleBucket struct {
diff --git a/snapshot/snapshotfs/upload.go b/snapshot/snapshotfs/upload.go
index 8293d596..87528864 100644
--- a/snapshot/snapshotfs/upload.go
+++ b/snapshot/snapshotfs/upload.go
@@ -10,7 +10,6 @@
 	"path"
 	"path/filepath"
 	"runtime"
-	"sync"
 	"sync/atomic"
 	"time"
 
@@ -37,9 +36,8 @@
 const DefaultCheckpointInterval = 45 * time.Minute
 
 var (
-	uploadLog   = logging.Module("uploader")
-	estimateLog = logging.Module("estimate")
-	repoFSLog   = logging.Module("repofs")
+	uploadLog = logging.Module("uploader")
+	repoFSLog = logging.Module("repofs")
 
 	uploadTracer = otel.Tracer("upload")
 )
@@ -1279,37 +1277,9 @@ func (u *Uploader) Upload(
 
 	s.StartTime = fs.UTCTimestampFromTime(u.repo.Time())
 
-	var scanWG sync.WaitGroup
-
-	scanctx, cancelScan := context.WithCancel(ctx)
-
-	defer cancelScan()
-
 	switch entry := source.(type) {
 	case fs.Directory:
-		var previousDirs []fs.Directory
-
-		for _, m := range previousManifests {
-			if d := u.maybeOpenDirectoryFromManifest(ctx, m); d != nil {
-				previousDirs = append(previousDirs, d)
-			}
-		}
-
-		scanWG.Add(1)
-
-		go func() {
-			defer scanWG.Done()
-
-			wrapped := u.wrapIgnorefs(estimateLog(ctx), entry, policyTree, false /* reportIgnoreStats */)
-
-			ds, _ := u.scanDirectory(scanctx, wrapped, policyTree)
-
-			u.Progress.EstimatedDataSize(ds.numFiles, ds.totalFileSize)
-		}()
-
-		wrapped := u.wrapIgnorefs(uploadLog(ctx), entry, policyTree, true /* reportIgnoreStats */)
-
-		s.RootEntry, err = u.uploadDirWithCheckpointing(ctx, wrapped, policyTree, previousDirs, sourceInfo)
+		s.RootEntry, err = u.uploadDir(ctx, previousManifests, entry, policyTree, sourceInfo)
 
 	case fs.File:
 		u.Progress.EstimatedDataSize(1, entry.Size())
@@ -1323,9 +1293,6 @@ func (u *Uploader) Upload(
 		return nil, rootCauseError(err)
 	}
 
-	cancelScan()
-	scanWG.Wait()
-
 	s.IncompleteReason = u.incompleteReason()
 	s.EndTime = fs.UTCTimestampFromTime(u.repo.Time())
 	s.Stats = *u.stats
@@ -1333,6 +1300,54 @@ func (u *Uploader) Upload(
 	return s, nil
 }
 
+func (u *Uploader) uploadDir(
+	ctx context.Context,
+	previousManifests []*snapshot.Manifest,
+	entry fs.Directory,
+	policyTree *policy.Tree,
+	sourceInfo snapshot.SourceInfo,
+) (*snapshot.DirEntry, error) {
+	var previousDirs []fs.Directory
+
+	for _, m := range previousManifests {
+		if d := u.maybeOpenDirectoryFromManifest(ctx, m); d != nil {
+			previousDirs = append(previousDirs, d)
+		}
+	}
+
+	estimationCtl := u.startDataSizeEstimation(ctx, entry, policyTree)
+	defer func() {
+		estimationCtl.Cancel()
+		estimationCtl.Wait()
+	}()
+
+	wrapped := u.wrapIgnorefs(uploadLog(ctx), entry, policyTree, true /* reportIgnoreStats */)
+
+	return u.uploadDirWithCheckpointing(ctx, wrapped, policyTree, previousDirs, sourceInfo)
+}
+
+func (u *Uploader) startDataSizeEstimation(
+	ctx context.Context,
+	entry fs.Directory,
+	policyTree *policy.Tree,
+) EstimationController {
+	logger := estimateLog(ctx)
+	wrapped := u.wrapIgnorefs(logger, entry, policyTree, false /* reportIgnoreStats */)
+
+	if u.disableEstimation || !u.Progress.Enabled() {
+		logger.Debug("Estimation disabled")
+		return noOpEstimationCtrl
+	}
+
+	estimator := NewEstimator(wrapped, policyTree, u.Progress.EstimationParameters(), logger)
+
+	estimator.StartEstimation(ctx, func(filesCount, totalFileSize int64) {
+		u.Progress.EstimatedDataSize(filesCount, totalFileSize)
+	})
+
+	return estimator
+}
+
 func (u *Uploader) wrapIgnorefs(logger logging.Logger, entry fs.Directory, policyTree *policy.Tree, reportIgnoreStats bool) fs.Directory {
 	if u.DisableIgnoreRules {
 		return entry
diff --git a/snapshot/snapshotfs/upload_estimator.go b/snapshot/snapshotfs/upload_estimator.go
new file mode 100644
index 00000000..ee3f3264
--- /dev/null
+++ b/snapshot/snapshotfs/upload_estimator.go
@@ -0,0 +1,183 @@
+package snapshotfs
+
+import (
+	"context"
+	"sync"
+
+	"github.com/kopia/kopia/fs"
+	vsi "github.com/kopia/kopia/internal/volumesizeinfo"
+	"github.com/kopia/kopia/repo/logging"
+	"github.com/kopia/kopia/snapshot/policy"
+
+	"github.com/pkg/errors"
+)
+
+// EstimationDoneFn represents the signature of the callback function which will be invoked when an estimation is done.
+type EstimationDoneFn func(int64, int64)
+
+// EstimationStarter defines an interface that is used to start an estimation of the size of data to be uploaded.
+type EstimationStarter interface {
+	StartEstimation(ctx context.Context, cb EstimationDoneFn)
+}
+
+// EstimationController defines an interface which has to be used to cancel or wait for running estimation.
+type EstimationController interface {
+	Cancel()
+	Wait()
+}
+
+// Estimator interface combines EstimationStarter and EstimationController interfaces.
+// It represents the objects that can both initiate and control an estimation process.
+type Estimator interface {
+	EstimationStarter
+	EstimationController
+}
+
+// NoOpEstimationController is a default implementation of the EstimationController interface.
+// It's used in cases where no estimation operation is running and hence, its methods are no-ops.
+type NoOpEstimationController struct{}
+
+// Cancel is a no-op function to satisfy the EstimationController interface.
+func (c *NoOpEstimationController) Cancel() {}
+
+// Wait is a no-op function to satisfy the EstimationController interface.
+func (c *NoOpEstimationController) Wait() {}
+
+// noOpEstimationCtrl is an instance of NoOpEstimationController.
+// It's a singleton instance used to handle operations when no estimation is running.
+var noOpEstimationCtrl EstimationController = &NoOpEstimationController{} //nolint:gochecknoglobals
+
+type estimator struct {
+	estimationParameters EstimationParameters
+	logger               logging.Logger
+	entry                fs.Directory
+	policyTree           *policy.Tree
+
+	scanWG              sync.WaitGroup
+	cancelCtx           context.CancelFunc
+	getVolumeSizeInfoFn func(string) (vsi.VolumeSizeInfo, error)
+}
+
+// EstimatorOption is an option which could be used to customize estimator behavior.
+type EstimatorOption func(Estimator)
+
+// VolumeSizeInfoFn represents a function type which is used to retrieve volume size information.
+type VolumeSizeInfoFn func(string) (vsi.VolumeSizeInfo, error)
+
+// WithVolumeSizeInfoFn returns EstimatorOption which allows to pass custom GetVolumeSizeInfo implementation.
+func WithVolumeSizeInfoFn(fn VolumeSizeInfoFn) EstimatorOption {
+	return func(e Estimator) {
+		roughEst, _ := e.(*estimator)
+		roughEst.getVolumeSizeInfoFn = fn
+	}
+}
+
+// NewEstimator returns instance of estimator.
+func NewEstimator(
+	entry fs.Directory,
+	policyTree *policy.Tree,
+	estimationParams EstimationParameters,
+	logger logging.Logger,
+	options ...EstimatorOption,
+) Estimator {
+	est := &estimator{
+		estimationParameters: estimationParams,
+		logger:               logger,
+		entry:                entry,
+		policyTree:           policyTree,
+		getVolumeSizeInfoFn:  vsi.GetVolumeSizeInfo,
+	}
+
+	for _, option := range options {
+		option(est)
+	}
+
+	return est
+}
+
+// StartEstimation starts estimation of data to be uploaded.
+// Terminates early as soon as the provided context is canceled.
+func (e *estimator) StartEstimation(ctx context.Context, cb EstimationDoneFn) {
+	if e.cancelCtx != nil {
+		return // Estimation already started, do nothing
+	}
+
+	scanCtx, cancelScan := context.WithCancel(ctx)
+
+	e.cancelCtx = cancelScan
+	e.scanWG.Add(1)
+
+	go func() {
+		defer e.scanWG.Done()
+
+		logger := estimateLog(ctx)
+
+		var filesCount, totalFileSize int64
+
+		var err error
+
+		et := e.estimationParameters.Type
+		useClassic := false
+
+		if et == EstimationTypeAdaptive || et == EstimationTypeRough {
+			filesCount, totalFileSize, err = e.doRoughEstimation()
+			if err != nil {
+				logger.Debugf("Unable to do rough estimation, fallback to classic one. %v", err)
+
+				useClassic = true
+			}
+
+			if et == EstimationTypeAdaptive && filesCount < e.estimationParameters.AdaptiveThreshold {
+				logger.Debugf("Small number of files (%d) on volume, falling back to classic estimation.", filesCount)
+
+				useClassic = true
+			}
+		}
+
+		if useClassic || et == EstimationTypeClassic {
+			filesCount, totalFileSize, err = e.doClassicEstimation(scanCtx)
+			if err != nil {
+				if errors.Is(err, context.Canceled) {
+					logger.Debugf("Estimation has been interrupted")
+				} else {
+					logger.Debugf("Estimation failed: %v", err)
+					logger.Warn("Unable to estimate")
+				}
+			}
+		}
+
+		cb(filesCount, totalFileSize)
+	}()
+}
+
+func (e *estimator) Wait() {
+	e.scanWG.Wait()
+	e.cancelCtx = nil
+}
+
+func (e *estimator) Cancel() {
+	if e.cancelCtx != nil {
+		e.cancelCtx()
+		e.cancelCtx = nil
+	}
+}
+
+func (e *estimator) doRoughEstimation() (filesCount, totalFileSize int64, err error) {
+	volumeSizeInfo, err := e.getVolumeSizeInfoFn(e.entry.LocalFilesystemPath())
+	if err != nil {
+		return 0, 0, errors.Wrap(err, "Unable to get volume size info")
+	}
+
+	return int64(volumeSizeInfo.FilesCount), int64(volumeSizeInfo.UsedSize), nil //nolint:gosec
+}
+
+func (e *estimator) doClassicEstimation(ctx context.Context) (filesCount, totalFileSize int64, err error) {
+	var res scanResults
+
+	err = Estimate(ctx, e.entry, e.policyTree, &res, 1)
+	if err != nil {
+		return 0, 0, errors.Wrap(err, "Unable to scan directory")
+	}
+
+	return int64(res.numFiles), res.totalFileSize, nil
+}
diff --git a/snapshot/snapshotfs/upload_progress.go b/snapshot/snapshotfs/upload_progress.go
index f3352f93..4f690f03 100644
--- a/snapshot/snapshotfs/upload_progress.go
+++ b/snapshot/snapshotfs/upload_progress.go
@@ -7,10 +7,32 @@
 	"github.com/kopia/kopia/internal/uitask"
 )
 
+const (
+	// EstimationTypeClassic represents old way of estimation, which assumes iterating over all files.
+	EstimationTypeClassic = "classic"
+	// EstimationTypeRough represents new way of estimation, which looks into filesystem stats to get amount of data.
+	EstimationTypeRough = "rough"
+	// EstimationTypeAdaptive is a combination of new and old approaches. If the estimated file count is high,
+	// it will use a rough estimation. If the count is low, it will switch to the classic method.
+	EstimationTypeAdaptive = "adaptive"
+
+	// AdaptiveEstimationThreshold is the point at which the classic estimation is used instead of the rough estimation.
+	AdaptiveEstimationThreshold = 300000
+)
+
+// EstimationParameters represents parameters to be used for estimation.
+type EstimationParameters struct {
+	Type              string
+	AdaptiveThreshold int64
+}
+
 // UploadProgress is invoked by uploader to report status of file and directory uploads.
 //
 //nolint:interfacebloat
 type UploadProgress interface {
+	// Enabled returns true when progress is enabled, false otherwise.
+	Enabled() bool
+
 	// UploadStarted is emitted once at the start of an upload
 	UploadStarted()
 
@@ -53,20 +75,28 @@ type UploadProgress interface {
 	// FinishedDirectory is emitted whenever a directory is finished uploading.
 	FinishedDirectory(dirname string)
 
+	// EstimationParameters returns settings to be used for estimation
+	EstimationParameters() EstimationParameters
+
 	// EstimatedDataSize is emitted whenever the size of upload is estimated.
-	EstimatedDataSize(fileCount int, totalBytes int64)
+	EstimatedDataSize(fileCount int64, totalBytes int64)
 }
 
 // NullUploadProgress is an implementation of UploadProgress that does not produce any output.
 type NullUploadProgress struct{}
 
+// Enabled implements UploadProgress, always returns false.
+func (p *NullUploadProgress) Enabled() bool {
+	return false
+}
+
 // UploadStarted implements UploadProgress.
 func (p *NullUploadProgress) UploadStarted() {}
 
 // EstimatedDataSize implements UploadProgress.
 //
 //nolint:revive
-func (p *NullUploadProgress) EstimatedDataSize(fileCount int, totalBytes int64) {}
+func (p *NullUploadProgress) EstimatedDataSize(fileCount, totalBytes int64) {}
 
 // UploadFinished implements UploadProgress.
 func (p *NullUploadProgress) UploadFinished() {}
@@ -126,6 +156,13 @@ func (p *NullUploadProgress) FinishedDirectory(dirname string) {}
 //nolint:revive
 func (p *NullUploadProgress) Error(path string, err error, isIgnored bool) {}
 
+// EstimationParameters implements UploadProgress.
+func (p *NullUploadProgress) EstimationParameters() EstimationParameters {
+	return EstimationParameters{
+		Type: EstimationTypeClassic,
+	}
+}
+
 var _ UploadProgress = (*NullUploadProgress)(nil)
 
 // UploadCounters represents a snapshot of upload counters.
@@ -155,7 +192,7 @@ type UploadCounters struct {
 	// +checkatomic
 	IgnoredErrorCount int32 `json:"ignoredErrors"`
 	// +checkatomic
-	EstimatedFiles int32 `json:"estimatedFiles"`
+	EstimatedFiles int64 `json:"estimatedFiles"`
 
 	CurrentDirectory string `json:"directory"`
 
@@ -184,9 +221,9 @@ func (p *CountingUploadProgress) UploadedBytes(numBytes int64) {
 }
 
 // EstimatedDataSize implements UploadProgress.
-func (p *CountingUploadProgress) EstimatedDataSize(numFiles int, numBytes int64) {
+func (p *CountingUploadProgress) EstimatedDataSize(numFiles, numBytes int64) {
 	atomic.StoreInt64(&p.counters.EstimatedBytes, numBytes)
-	atomic.StoreInt32(&p.counters.EstimatedFiles, int32(numFiles)) //nolint:gosec
+	atomic.StoreInt64(&p.counters.EstimatedFiles, numFiles)
 }
 
 // HashedBytes implements UploadProgress.
@@ -262,7 +299,7 @@ func (p *CountingUploadProgress) Snapshot() UploadCounters {
 		TotalCachedBytes:  atomic.LoadInt64(&p.counters.TotalCachedBytes),
 		TotalHashedBytes:  atomic.LoadInt64(&p.counters.TotalHashedBytes),
 		EstimatedBytes:    atomic.LoadInt64(&p.counters.EstimatedBytes),
-		EstimatedFiles:    atomic.LoadInt32(&p.counters.EstimatedFiles),
+		EstimatedFiles:    atomic.LoadInt64(&p.counters.EstimatedFiles),
 		IgnoredErrorCount: atomic.LoadInt32(&p.counters.IgnoredErrorCount),
 		FatalErrorCount:   atomic.LoadInt32(&p.counters.FatalErrorCount),
 		CurrentDirectory:  p.counters.CurrentDirectory,
@@ -298,7 +335,7 @@ func (p *CountingUploadProgress) UITaskCounters(final bool) map[string]uitask.Co
 	}
 
 	if !final {
-		m["Estimated Files"] = uitask.SimpleCounter(int64(atomic.LoadInt32(&p.counters.EstimatedFiles)))
+		m["Estimated Files"] = uitask.SimpleCounter(atomic.LoadInt64(&p.counters.EstimatedFiles))
 		m["Estimated Bytes"] = uitask.BytesCounter(atomic.LoadInt64(&p.counters.EstimatedBytes))
 	}
 
diff --git a/snapshot/snapshotfs/upload_scan.go b/snapshot/snapshotfs/upload_scan.go
index daf3f490..9874d656 100644
--- a/snapshot/snapshotfs/upload_scan.go
+++ b/snapshot/snapshotfs/upload_scan.go
@@ -4,9 +4,7 @@
 	"context"
 	"sync/atomic"
 
-	"github.com/kopia/kopia/fs"
 	"github.com/kopia/kopia/snapshot"
-	"github.com/kopia/kopia/snapshot/policy"
 )
 
 type scanResults struct {
@@ -27,17 +25,3 @@ func (e *scanResults) Stats(ctx context.Context, s *snapshot.Stats, includedFile
 }
 
 var _ EstimateProgress = (*scanResults)(nil)
-
-// scanDirectory computes the number of files and their total size in a given directory recursively descending
-// into subdirectories. The scan teminates early as soon as the provided context is canceled.
-func (u *Uploader) scanDirectory(ctx context.Context, dir fs.Directory, policyTree *policy.Tree) (scanResults, error) {
-	var res scanResults
-
-	if u.disableEstimation {
-		return res, nil
-	}
-
-	err := Estimate(ctx, dir, policyTree, &res, 1)
-
-	return res, err
-}


Test patch:
diff --git a/.github/workflows/compat-test.yml b/.github/workflows/compat-test.yml
index 5786c2a8..cf77d52b 100644
--- a/.github/workflows/compat-test.yml
+++ b/.github/workflows/compat-test.yml
@@ -14,11 +14,11 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
@@ -26,7 +26,7 @@ jobs:
     - name: Compat Test
       run: make compat-tests
     - name: Upload Logs
-      uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+      uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
       with:
         name: logs
         path: .logs/**/*.log
diff --git a/.github/workflows/endurance-test.yml b/.github/workflows/endurance-test.yml
index 46f656b9..ad22bf22 100644
--- a/.github/workflows/endurance-test.yml
+++ b/.github/workflows/endurance-test.yml
@@ -19,11 +19,11 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
@@ -31,7 +31,7 @@ jobs:
     - name: Endurance Tests
       run: make endurance-tests
     - name: Upload Logs
-      uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+      uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
       with:
         name: logs
         path: .logs/**/*.log
diff --git a/.github/workflows/htmlui-tests.yml b/.github/workflows/htmlui-tests.yml
index bcdf8585..37f5fdb5 100644
--- a/.github/workflows/htmlui-tests.yml
+++ b/.github/workflows/htmlui-tests.yml
@@ -27,11 +27,11 @@ jobs:
     runs-on: macos-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
@@ -39,7 +39,7 @@ jobs:
     - name: Run Tests
       run: make htmlui-e2e-test
     - name: Upload Screenshots
-      uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+      uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
       with:
         path: .screenshots/**/*.png
         if-no-files-found: ignore
diff --git a/.github/workflows/stress-test.yml b/.github/workflows/stress-test.yml
index 455083d9..ec710e5a 100644
--- a/.github/workflows/stress-test.yml
+++ b/.github/workflows/stress-test.yml
@@ -18,11 +18,11 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
@@ -30,7 +30,7 @@ jobs:
     - name: Stress Test
       run: make stress-test
     - name: Upload Logs
-      uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+      uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
       with:
         name: logs
         path: .logs/**/*.log
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 78469486..d4fffae2 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -38,11 +38,11 @@ jobs:
     continue-on-error: ${{ contains(matrix.os, 'self-hosted') }}
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
@@ -66,7 +66,7 @@ jobs:
     - name: Integration Tests
       run: make -j2 ci-integration-tests
     - name: Upload Logs
-      uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+      uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
       with:
         name: logs-${{ matrix.os }}
         path: .logs/**/*.log
diff --git a/.github/workflows/volume-shadow-copy-test.yml b/.github/workflows/volume-shadow-copy-test.yml
index b975a6ac..ddbafc71 100644
--- a/.github/workflows/volume-shadow-copy-test.yml
+++ b/.github/workflows/volume-shadow-copy-test.yml
@@ -15,11 +15,11 @@ jobs:
     runs-on: windows-latest
     steps:
     - name: Check out repository
-      uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0
+      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         fetch-depth: 0
     - name: Set up Go
-      uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
+      uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
       with:
         go-version-file: 'go.mod'
         check-latest: true
@@ -34,7 +34,7 @@ jobs:
     - name: Non-Admin Test
       run: gsudo -i Medium make os-snapshot-tests
     - name: Upload Logs
-      uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+      uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
       with:
         name: logs
         path: .logs/**/*.log
diff --git a/snapshot/snapshotfs/upload_estimator_test.go b/snapshot/snapshotfs/upload_estimator_test.go
new file mode 100644
index 00000000..083b6483
--- /dev/null
+++ b/snapshot/snapshotfs/upload_estimator_test.go
@@ -0,0 +1,255 @@
+package snapshotfs_test
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/kopia/kopia/internal/mockfs"
+	vsi "github.com/kopia/kopia/internal/volumesizeinfo"
+	"github.com/kopia/kopia/repo/logging"
+	"github.com/kopia/kopia/snapshot/policy"
+	"github.com/kopia/kopia/snapshot/snapshotfs"
+
+	"github.com/stretchr/testify/require"
+	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
+)
+
+var errSimulated = errors.New("simulated error")
+
+type mockLogger struct{}
+
+func (w *mockLogger) Write(p []byte) (int, error) {
+	return len(p), nil
+}
+
+func (w *mockLogger) Sync() error {
+	return nil
+}
+
+func getMockLogger() logging.Logger {
+	ml := &mockLogger{}
+	return zap.New(
+		zapcore.NewCore(
+			zapcore.NewConsoleEncoder(zapcore.EncoderConfig{
+				// Keys can be anything except the empty string.
+				TimeKey:        zapcore.OmitKey,
+				LevelKey:       zapcore.OmitKey,
+				NameKey:        zapcore.OmitKey,
+				CallerKey:      zapcore.OmitKey,
+				FunctionKey:    zapcore.OmitKey,
+				MessageKey:     "M",
+				StacktraceKey:  "S",
+				LineEnding:     zapcore.DefaultLineEnding,
+				EncodeLevel:    zapcore.CapitalLevelEncoder,
+				EncodeTime:     zapcore.ISO8601TimeEncoder,
+				EncodeDuration: zapcore.StringDurationEncoder,
+				EncodeCaller:   zapcore.ShortCallerEncoder,
+			}),
+			ml,
+			zapcore.DebugLevel,
+		),
+	).Sugar()
+}
+
+// withFailedVolumeSizeInfo returns EstimatorOption which ensures that GetVolumeSizeInfo will fail with provided error.
+// Purposed for tests.
+func withFailedVolumeSizeInfo(err error) snapshotfs.EstimatorOption {
+	return snapshotfs.WithVolumeSizeInfoFn(func(_ string) (vsi.VolumeSizeInfo, error) {
+		return vsi.VolumeSizeInfo{}, err
+	})
+}
+
+// withVolumeSizeInfo returns EstimatorOption which provides fake volume size.
+func withVolumeSizeInfo(filesCount, usedFileSize, totalFileSize uint64) snapshotfs.EstimatorOption {
+	return snapshotfs.WithVolumeSizeInfoFn(func(_ string) (vsi.VolumeSizeInfo, error) {
+		return vsi.VolumeSizeInfo{
+			TotalSize:  totalFileSize,
+			UsedSize:   usedFileSize,
+			FilesCount: filesCount,
+		}, nil
+	})
+}
+
+func expectSuccessfulEstimation(
+	ctx context.Context,
+	t *testing.T,
+	estimator snapshotfs.Estimator,
+	expectedNumberOfFiles,
+	expectedDataSize int64,
+) {
+	t.Helper()
+	var filesCount, totalFileSize int64
+
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		estimator.StartEstimation(ctx, func(fc, ts int64) {
+			filesCount = fc
+			totalFileSize = ts
+		})
+
+		estimator.Wait()
+	}()
+
+	select {
+	case <-done:
+		require.Equal(t, expectedNumberOfFiles, filesCount)
+		require.Equal(t, expectedDataSize, totalFileSize)
+	case <-time.After(time.Second):
+		t.Fatal("timed out waiting for estimation")
+	}
+}
+
+func TestUploadEstimator(t *testing.T) {
+	dir1 := mockfs.NewDirectory()
+
+	file1Content := []byte{1, 2, 3}
+	file2Content := []byte{4, 5, 6, 7}
+	file3Content := []byte{8, 9, 10, 11, 12}
+
+	dir1.AddFile("file1", file1Content, 0o644)
+	dir1.AddFile("file2", file2Content, 0o644)
+	dir1.AddFile("file3", file3Content, 0o644)
+
+	expectedNumberOfFiles := int64(3)
+	expectedDataSize := int64(len(file1Content) + len(file2Content) + len(file3Content))
+
+	t.Run("Classic estimation", func(t *testing.T) {
+		logger := getMockLogger()
+
+		policyTree := policy.BuildTree(nil, policy.DefaultPolicy)
+		estimator := snapshotfs.NewEstimator(dir1, policyTree, snapshotfs.EstimationParameters{Type: snapshotfs.EstimationTypeClassic}, logger)
+
+		estimationCtx := context.Background()
+		expectSuccessfulEstimation(estimationCtx, t, estimator, expectedNumberOfFiles, expectedDataSize)
+	})
+	t.Run("Rough estimation", func(t *testing.T) {
+		logger := getMockLogger()
+
+		expectedNumberOfFiles := int64(1000)
+		expectedDataSize := int64(2000)
+
+		policyTree := policy.BuildTree(nil, policy.DefaultPolicy)
+		estimator := snapshotfs.NewEstimator(
+			dir1, policyTree, snapshotfs.EstimationParameters{Type: snapshotfs.EstimationTypeRough}, logger,
+			withVolumeSizeInfo(uint64(expectedNumberOfFiles), uint64(expectedDataSize), 3000))
+
+		estimationCtx := context.Background()
+
+		expectSuccessfulEstimation(estimationCtx, t, estimator, expectedNumberOfFiles, expectedDataSize)
+	})
+	t.Run("Rough estimation - GetVolumeSizeInfo failed", func(t *testing.T) {
+		logger := getMockLogger()
+
+		policyTree := policy.BuildTree(nil, policy.DefaultPolicy)
+		estimator := snapshotfs.NewEstimator(
+			dir1, policyTree, snapshotfs.EstimationParameters{Type: snapshotfs.EstimationTypeRough}, logger,
+			withFailedVolumeSizeInfo(errSimulated))
+
+		estimationCtx := context.Background()
+
+		// We expect that estimation will succeed even when GetVolumeSizeInfo will fail
+		// fallback to classical estimation should handle this case
+		expectSuccessfulEstimation(estimationCtx, t, estimator, expectedNumberOfFiles, expectedDataSize)
+	})
+	t.Run("Adaptive estimation - rough estimation path", func(t *testing.T) {
+		logger := getMockLogger()
+
+		expectedNumberOfFiles := int64(1000)
+		expectedDataSize := int64(2000)
+
+		policyTree := policy.BuildTree(nil, policy.DefaultPolicy)
+		estimator := snapshotfs.NewEstimator(
+			dir1, policyTree,
+			snapshotfs.EstimationParameters{Type: snapshotfs.EstimationTypeAdaptive, AdaptiveThreshold: 100}, logger,
+			withVolumeSizeInfo(uint64(expectedNumberOfFiles), uint64(expectedDataSize), 3000))
+
+		estimationCtx := context.Background()
+
+		expectSuccessfulEstimation(estimationCtx, t, estimator, expectedNumberOfFiles, expectedDataSize)
+	})
+	t.Run("Adaptive estimation - classic estimation path", func(t *testing.T) {
+		logger := getMockLogger()
+
+		policyTree := policy.BuildTree(nil, policy.DefaultPolicy)
+		estimator := snapshotfs.NewEstimator(
+			dir1, policyTree,
+			snapshotfs.EstimationParameters{Type: snapshotfs.EstimationTypeAdaptive, AdaptiveThreshold: 10000}, logger,
+			withVolumeSizeInfo(uint64(1000), uint64(2000), 3000))
+
+		estimationCtx := context.Background()
+
+		expectSuccessfulEstimation(estimationCtx, t, estimator, expectedNumberOfFiles, expectedDataSize)
+	})
+	t.Run("Adaptive estimation - getVolumeSizeInfo failed", func(t *testing.T) {
+		logger := getMockLogger()
+
+		policyTree := policy.BuildTree(nil, policy.DefaultPolicy)
+		estimator := snapshotfs.NewEstimator(
+			dir1, policyTree, snapshotfs.EstimationParameters{Type: snapshotfs.EstimationTypeAdaptive, AdaptiveThreshold: 1}, logger,
+			withFailedVolumeSizeInfo(errSimulated))
+
+		estimationCtx := context.Background()
+
+		// We expect that estimation will succeed even when getVolumeSizeInfo will fail
+		// fallback to classical estimation should handle this case
+		expectSuccessfulEstimation(estimationCtx, t, estimator, expectedNumberOfFiles, expectedDataSize)
+	})
+
+	t.Run("Classic estimation stops on context cancel", func(t *testing.T) {
+		testCtx, cancel := context.WithCancel(context.Background())
+		dir2 := mockfs.NewDirectory()
+
+		dir2.AddFile("file1", file1Content, 0o644)
+		dir2.AddFile("file2", file2Content, 0o644)
+		dir2.AddFile("file3", file3Content, 0o644)
+		dir2.AddDir("d1", 0o777)
+
+		dir2.Subdir("d1").OnReaddir(func() {
+			cancel()
+		})
+
+		logger := getMockLogger()
+		policyTree := policy.BuildTree(nil, policy.DefaultPolicy)
+		estimator := snapshotfs.NewEstimator(dir2, policyTree, snapshotfs.EstimationParameters{Type: snapshotfs.EstimationTypeRough}, logger)
+
+		// In case of canceled context, we should get zeroes instead of estimated numbers
+		expectSuccessfulEstimation(testCtx, t, estimator, 0, 0)
+	})
+	t.Run("EstimationStarter stops on request", func(t *testing.T) {
+		dir2 := mockfs.NewDirectory()
+
+		dir2.AddFile("file1", file1Content, 0o644)
+		dir2.AddFile("file2", file2Content, 0o644)
+		dir2.AddFile("file3", file3Content, 0o644)
+		dir2.AddDir("d1", 0o777)
+
+		logger := getMockLogger()
+		policyTree := policy.BuildTree(nil, policy.DefaultPolicy)
+		estimator := snapshotfs.NewEstimator(dir2, policyTree, snapshotfs.EstimationParameters{Type: snapshotfs.EstimationTypeClassic}, logger)
+
+		dir2.Subdir("d1").OnReaddir(func() {
+			estimator.Cancel()
+		})
+
+		// In case interrupted estimation, we should get zeroes instead of estimated numbers
+		expectSuccessfulEstimation(context.Background(), t, estimator, 0, 0)
+	})
+	t.Run("Classic estimation respects ignores from policy tree", func(t *testing.T) {
+		policyTree := policy.BuildTree(map[string]*policy.Policy{
+			".": {
+				FilesPolicy: policy.FilesPolicy{
+					IgnoreRules: []string{"file1"},
+				},
+			},
+		}, policy.DefaultPolicy)
+
+		logger := getMockLogger()
+		estimator := snapshotfs.NewEstimator(dir1, policyTree, snapshotfs.EstimationParameters{Type: snapshotfs.EstimationTypeClassic}, logger)
+
+		expectSuccessfulEstimation(context.Background(), t, estimator, expectedNumberOfFiles-1, expectedDataSize-int64(len(file1Content)))
+	})
+}
diff --git a/snapshot/snapshotfs/upload_test.go b/snapshot/snapshotfs/upload_test.go
index 936a2fdc..5f109762 100644
--- a/snapshot/snapshotfs/upload_test.go
+++ b/snapshot/snapshotfs/upload_test.go
@@ -854,59 +854,6 @@ func randomBytes(n int64) []byte {
 	return b
 }
 
-func TestUploadScanStopsOnContextCancel(t *testing.T) {
-	ctx := testlogging.Context(t)
-	th := newUploadTestHarness(ctx, t)
-
-	defer th.cleanup()
-
-	u := NewUploader(th.repo)
-
-	scanctx, cancel := context.WithCancel(ctx)
-
-	th.sourceDir.Subdir("d1").Subdir("d2").OnReaddir(func() {
-		cancel()
-	})
-
-	result, err := u.scanDirectory(scanctx, th.sourceDir, nil)
-	require.ErrorIs(t, err, scanctx.Err())
-
-	if result.numFiles == 0 && result.totalFileSize == 0 {
-		t.Fatalf("should have returned partial results, got zeros")
-	}
-}
-
-func TestUploadScanIgnoresFiles(t *testing.T) {
-	ctx := testlogging.Context(t)
-	th := newUploadTestHarness(ctx, t)
-
-	defer th.cleanup()
-
-	u := NewUploader(th.repo)
-
-	// set up a policy tree where that ignores some files.
-	policyTree := policy.BuildTree(map[string]*policy.Policy{
-		".": {
-			FilesPolicy: policy.FilesPolicy{
-				IgnoreRules: []string{"f1"},
-			},
-		},
-	}, policy.DefaultPolicy)
-
-	// no policy
-	result1, err := u.scanDirectory(ctx, th.sourceDir, nil)
-	require.NoError(t, err)
-
-	result2, err := u.scanDirectory(ctx, th.sourceDir, policyTree)
-	require.NoError(t, err)
-
-	require.NotEqual(t, 0, result1.numFiles)
-	require.NotEqual(t, 0, result2.numFiles)
-
-	require.Less(t, result2.numFiles, result1.numFiles)
-	require.Less(t, result2.totalFileSize, result1.totalFileSize)
-}
-
 func TestUpload_VirtualDirectoryWithStreamingFile(t *testing.T) {
 	ctx := testlogging.Context(t)
 	th := newUploadTestHarness(ctx, t)