opened
JMoVS
username_0
As requested on IRC, here's my use case:

I have a laptop with an internal disk and zpool. This zpool is zrepl configured as a push job, pushing to my USB connected 2-way mirror zpool. Now I primarily care about the fact that incremental backup is always possible, so that means that a bookmark from which to resume from is always available.

With the new holding logic, I would need the pruner to also clear all holds (as a setting). 

The reason is as follows:
If replication happens, it creates the holds. Now when I disconnect and export the external mirror zpool, zrepl will still hold those snapshots. I have configured zrepl to only ever hold 20 snapshots on my laptop to conserve space. And it happens that I'm away for a week or two.

In this use case, I would prefer for zrepl to release the hold, destroy the snapshots and then when the mirror targt pool becomes availble again, delete the resumable send state and send an incremental backup from the bookmark. This way, my local machine would always only ever have 20 snapshots.
Use case and the need for pruning to destroy snapshots that have not been fully sent
issue
created
problame
username_1
I figure the best way to address this need is an option on the pruner that converts any existing replication holds to bookmarks. Converting means

* create a step-bookmark of the step-hold's snapshot
* release the hold
* retry the prune-operation

```
  pruning:
    hold_handling:
      step_holds: convert_to_step_bookmark
    keep_sender: ...
    keep_receiver: ...
```

Note that the `last_received_hold` is on the receiving side and cannot be converted to a bookmark because it is used as `from` for subsequent replications.
comment
created
problame
username_1
Would this address your needs?
(note to myself: rename the issue once it is clear that above proposal is what we want)
comment
created
JMoVS
username_0
it is used as from and it might also simply be not accessible and offline by the time. Now is it possible to resume a resumable send from a bookmark?
comment
created
problame
username_1
Yes, even if the transfer that got interrupted used a snapshot as `from`.
(The resume token just contains the `fromguid`, the resume-send code looks up snapshots, then bookmarks with that guid).
comment
created
JMoVS
username_0
Hmm ok, if resuming a send from a bookmark works, then your strategy of creating a bookmark for every hold and then releasing that hold, enabling the pruner to delete the snapshot would solve this problem. It might even make sense to make this the default option (I don't really see any downsides).

"it" was referring to snapshots/resume tokens that are residing on the sink pool. As I regularly detach my sink pool, "it" would not be accessible most of the time.
comment
created
problame
username_1
One more idea about this feature, but possibly, this is a separate feature:
**Why can't we just use deferred destroys (`zfs destroy -d`)?**

The implementation would
*  filter out all snapshots that have `defer_destroy=on` before evaluating the keep filters.
* run `zfs destroy -d` on those snapshots that are not kept
* be done.

zrepl only ever keeps at most two holds per send-side dataset.
Any other situation is a (separate) bug in zrepl.

---
But, if I understand you correctly, deferred destroys wouldn't address your problem, since you don't want the snapshot to stick around any longer, right?
comment
created
JMoVS
username_0
zrepl currently holds like 200 snapshots per dataset for me currently. So this might also be a separate bug. 

But yes, you understood correctly, I effectively want to make an "offsite" backup which I connect to semi-regularly and because I only connect semiregularly, I don't want to waste the space on my main machine. That's why I want the snapshots to be destroyed and only bookmarks to be left.
comment
created
problame
username_1
All right, wrt to the 200 holds, keep an eye on #282  - I'll be pushing some code there soon, although @username_2 's WIP implementation in that branch already works.
comment
created
problame
username_1
@username_0 in #293, there are now subcommands for removing stale holds `zrepl holds release-stale`.

There should also be some minor performance improvements (less zfs CLI invocations). 

Would you mind testing that PR so we can get it merged soon? Please provide any feedback on the PR in the comments _there_, unless it relates to this issue.
comment
created
problame
username_1
We discussed this in voice chat tonight:

- **Absence** of `keep: not_replicated` should be taken literally: no exceptions for snapshots that are not replicated, including those that are part of an ongonig send and thus have a step hold. Thus, in **absence** of `keep: not_replicated`, we should convert the step holds into step bookmarks, regardless of whether initial or incremental send. It's what the user wants if they don't specify this option.
- Introduce `keep: step_holds` which keeps step holds (and step bookmarks, but we don't prune those).
  - Step holds are **per definition** included in `keep: not_replicated since a step hold is always younger than the replication cursor used to determine whether a snapshot is replicated or not.
  - Howeve,r this keep rule can be specified separately as well if a user just wants to keep step holds for resumability, but avoid stacking up an infinite amount of snapshots on the sender as they'd do with `keep: not_replicated`.

- [ ] Boils down to the following decision tree that should be part of the docs:

```
<username_0> Decision tree:
<username_0> Q1: Do you want ALL SNAPSHOTS that are created ON YOUR SOURCE to ALL end up on the destination?
<username_0> => Activate keep-non-replicated and STOP here
<username_0> Q2: Do you want backups started to resume at all times? (eg spotty network connection) This means that we would keep the snapshot that is currently replicated on the source and resume the backup when the connection is available again.
<username_0> => Activate keep-step-holds and STOP here
<username_0> Q3: You really don't care and just want the last 20 snapshots to be retained on your source no matter what?
<username_0> => Just leave the config as is without any of the other options mentioned above
```
comment
created
problame
username_1
To clarify: the current situation is a bug: if "keep-non-replicated" is not set, step-holds created by zrepl must be destroyed by zrepl, because the user really didn't want snapshots to accumulate on the sending side.

A single hold violating this policy might not be all that bad, but hold leaks such as #316 exacerbate the problem.
comment
created
InsanePrawn
username_2
I think ideally we need something better than 'keep all not replicated' snapshots: a different set of pruning rules.
(Otherwise a remote down for maintenance might cause massive amounts of snapshots to build up if short snap intervals are configured, which is not something zfs handles too well)

Also, not_replicated or whatever succeeds it should optionally take a list of JobIDs to support unconvential/unsupported setups, e.g. pruning in snap jobs or multiple jobs replicating from one source dataset.
comment
created
problame
username_1
- [ ] Yes, that's very important, and the current state of master (i.e. per-job replication cursors) would indeed break existing snapjob + replication-only-job setups like yours.
comment
created
problame
username_1
Huh, actually, I'm not sure how your (@username_2  ) setup looks like. Which job does the pruning? The snap job or the replicating job?
comment
created
InsanePrawn
username_2
Both. It depends. It's complicated!

My abbreviated experimental configs for the three instances I'm using to test the 0.3 work is this, as is, on disk right now. I've played with the interval lengths a couple of times, these might be stupid for a number of reasons, I can't remember.

node 1 -> node 2 -> node 3

- node 1:
  - source job:
     - no pruning, snapshotting: manual
  - snap job:
    - 3m snap interval
    - prunes locally with a grid + keeps non-zrepl snaps
- node 2:
  - pull (from node 1) job:
    - 5m interval
    - pruning: keep everything on the sender, grid on receiver aka locally (hint: keep=all for newest 30m to smooth races, my zfs is slow af) (matches all snapshots, not just `zrepl_.*`) <- could technically be split into a snap job I guess, but that's job-unaware right now, I see no benefit at the cost of added raciness.
  - source job:
    - snapshotting: "manual" 
    - offers FS with snaps pulled from node 1
- node 3:
  - pull (from node 2) job:
    - pruning: like 2: don't touch sender, just prune receiver with a grid, this grid is in much lower time resolution though, think second level backup. Could also be split out, but again, why?


(Yes, this replicates from the source to the primary backup and from the primary to the secondary. Some blogs might tell you that's a bad thing to do, because corruption occurring on the primary replicates onto the secondary, etc. Still unsure whether you can have one of those nasty uncorrectable blocks be successfully sent and received by ZFS.)
Now we're going a little OT, but if I were to make up one example for pruning rules optionally taking __multiple__ JobIDs, it would help if you were to serve one source FS to a number of clients, such as your primary _and_ secondary backup server[s?]. (Or maybe your replication strategy is 'a little more horizontally')
Then you might define a source job per client and keep those pruned Just Right by offloading their pruning to a central snap job that applies its specific rule sets for replicated and non-replicated snapshots each, accounting for all those JobIDs in Just The Right Way.
For the first iteration of this feature, form the semantic union of those jobs while evaluating keep rules.
Stretch goal for zrepl 2025.04 Cluster Edition: allow some form of logical AND and OR groupings between jobs? Complexity to the max!
comment
created
problame
username_1
@username_0 : I was talking to @username_2 yesterday and he suggested that, for the 0.3 release, it might be sufficient to have a per-job option that disables step holds for incremental sends:

```
jobs:
- type: push
  ...
  send:
    step_holds:
      disable_incremental: true # defaults to false
```
Suppose you yank the external drive during an incremental `@from -> @to` step:
* restarting that step or future incrementals `@from -> `@to_later` will be possible
  * because the replication cursor bookmark points to `@from` until the step is complete
* *resuming* `@from -> @to` will work as long as the pruner on your internal pool doesn't come around to destroy `@to`.
  * in that case, the replication algorithm should determine that the resumable state on the receiving side is useless because `@to` no longer exists on the sending side, and consequently clear it, and restart an incremental step `@from -> @to_later`

Do you agree?
The advantage of this solution is that it doesn't require (substantial) changes to the pruner at this point, which I would appreciate given that I wanted to release 0.3 (= what is in `username_1/develop` about two months ago :D )
comment
created
JMoVS
username_0
yes, that would work I think. As long as at all times there will be a bookmark to increment from. ;-)
comment
created
problame
username_1
@username_0  `username_1/develop` `bb9444a` now contains a commit with the config syntax proposed above.
The new option has platformtest coverage and I tested your use-case manually.
I think it is safe to test on your setup.
Please report back ASAP whether the code works for you, I'll 
- [ ] document the option and use case then
comment
created
problame
username_1
Side note on the actual topic of this issue: the WIP branch `username_1/pruner_rewrite_step_holds` contains a pruner rewrite that is
* local to a given side of a replication setup
* supports `keep: step_holds`
  * [ ] that keep rule should actually support an option `younger_than: ` or sth for the following use case:
    * resumability is desirable, but only for a day or so, afterwards it'd be better to drop the hold and restart in the future
    * useful for the laptop + external HDD use case ("I plug in the HDD for backup every other day => set the option to 4 days as a compromise")
    * useful for flaky networks: replication every 4h, mid-way network failures 50% of the time => set it to 32h and have a greatly reduced probability of needed to re-start a transfer
comment