Repository: vshn/k8up (go)

Issue:
Support "auto" schedules
## Summary

**As** K8up user\
**I want** to specify smart schedules\
**So that** I can let K8up figure out optimal schedules for optimal resource usage

## Context

The scheduler should be able accept "auto" or "smart" schedules.

These can be like for crons:
* daily
* weekly
* monthly
* etc.

They should behave in such a way that the jobs will be run at least once during that defined time. The old cron syntax should still be supported for use cases where specific time is necessary.

The idea for the auto schedules would be to be triggered at any time in the given frame. For example daily should mean that the job should run at least once every day sometime between 00:00 and 23:59. When exactly should be determined by the operator.

This feature is intended mostly for jobs that need exclusive access to the backup repository like prune and check. They don't have any impact on the applications and can thus run whenever no backups are running.

For prune and check jobs the operator has to figure out the best time between backups to a repository when the jobs can run. One idea could be that the prune could be triggered right after all backups have finished. That would eliminate the need for a separate prune schedule completely.

Additional from #118:

Also the cron library used by k8up has some predefined schedules like `@daily` or `@weekly` so for the auto schedules we'd have to define them without an @ as not to break those pre-defined schedules. https://pkg.go.dev/github.com/robfig/cron#hdr-Predefined_schedules

And finally, the cron library supports intervals which could come in very handy for this feature.

## Out of Scope

* Implement a smart scheduling strategy with evenly distributed schedules (as discussed in this issue, this poses more problems than it may solve)

## Further links

* [Cron Standard](https://en.wikipedia.org/wiki/Cron)
* #118 
* [Intervals](https://pkg.go.dev/github.com/robfig/cron#hdr-Intervals)

## Acceptance criteria

*Given* a schedulable K8up object\
*When* a [standardized cron syntax](https://en.wikipedia.org/wiki/Cron) is specified\
*Then* schedule the resource at the specified times
(this keeps existing functionality)

*Given* a schedulable K8up object\
*When* a non-standardized predefined cron syntax is specified\
*Then* schedule the resource at randomized times within the given timeframe, with a stable randomization seed
(e.g. when `@hourly-randomized` this could result in `<random number between 0 and 59> * * * *` schedule, contrast to predefined `0 * * * *`, which would defeat the purpose of smart scheduling)

*Given* a schedulable K8up object\
*When* a non-standardized predefined cron syntax with `@every` is specified\
*Then* schedule the resource at randomized times with the given interval
(e.g. when `@every 2h` this could result in a schedule that runs every 2h with a random start time)

*Given* a schedulable K8up object\
*When* any sort of non-standardized predefined cron syntax is specified\
*Then* store the resulting schedule in the status field of the object
(to make the actual schedule transparent for the user)

## Implementation Ideas

* Examples of non-standard cron syntax:
  * `@hourly` -> same as https://pkg.go.dev/github.com/robfig/cron#hdr-Predefined_schedules
  * `@hourly-randomized` -> this will trigger randomization of the schedule and then generate the actual, standard cron syntax
* stable random seed could be generated from the Name + Namespace combination, create a unique number out of it (checksum?). For the minute, calculate this seed modulo 60, for hours modulo 24 (in case of `@daily-randomized`) and fill that until we get a stable, random schedule like `23 5 * * *`






Pull request:
Add new smart schedules
Implements new schedules:
* `@hourly-random`
* `@daily-random`
* `@weekly-random`
* `@monthly-random`
* `@yearly-random` or `@annually-random`

Special case: Monthly. The `@monthly-random` will produce a schedule with a day-of-month between 1 and 27, even though some months have 30 and 31 days. This is to prevent schedules being skipped if the current month has fewer days than specified in the day-of-month field in the schedule.

Implemented status field for random schedules, e.g.
```
metadata:
  ...
status:
  effectiveSchedules:
    archive: 0 * * * *
    check: 44 * * * *
```
Once the status field contains a schedule for a specific job type, it will use that and leave the `.spec.[type].schedule` as-is, so that the schedule like `@hourly-random` is not lost/overwritten in the spec with the generated schedule. If the generated schedule is missing in the status, it will be created and status updated. That means users can potentially adjust the generated schedule if that is not a good randomized schedule for some reason. It skips non-random schedules (e.g. `*/2 * * * *`) from putting them into the status. However, if the user provides another non-random schedule in the status still, that one will be used.

Closes #116 

Patch:
diff --git a/api/v1alpha1/schedule_types.go b/api/v1alpha1/schedule_types.go
index 8958d111..f778e925 100644
--- a/api/v1alpha1/schedule_types.go
+++ b/api/v1alpha1/schedule_types.go
@@ -56,6 +56,8 @@ type PruneSchedule struct {
 
 // ScheduleStatus defines the observed state of Schedule
 type ScheduleStatus struct {
+	// EffectiveSchedules displays the final schedule for each type (useful when using smart schedules).
+	EffectiveSchedules map[JobType]string `json:"effectiveSchedules,omitempty"`
 }
 
 // +kubebuilder:object:root=true
diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go
index 01897a22..4f874269 100644
--- a/api/v1alpha1/zz_generated.deepcopy.go
+++ b/api/v1alpha1/zz_generated.deepcopy.go
@@ -963,7 +963,7 @@ func (in *Schedule) DeepCopyInto(out *Schedule) {
 	out.TypeMeta = in.TypeMeta
 	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
 	in.Spec.DeepCopyInto(&out.Spec)
-	out.Status = in.Status
+	in.Status.DeepCopyInto(&out.Status)
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Schedule.
@@ -1085,6 +1085,13 @@ func (in *ScheduleSpec) DeepCopy() *ScheduleSpec {
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *ScheduleStatus) DeepCopyInto(out *ScheduleStatus) {
 	*out = *in
+	if in.EffectiveSchedules != nil {
+		in, out := &in.EffectiveSchedules, &out.EffectiveSchedules
+		*out = make(map[JobType]string, len(*in))
+		for key, val := range *in {
+			(*out)[key] = val
+		}
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ScheduleStatus.
diff --git a/config/crd/apiextensions.k8s.io/v1/backup.appuio.ch_schedules.yaml b/config/crd/apiextensions.k8s.io/v1/backup.appuio.ch_schedules.yaml
index f287ae0a..d09844b8 100644
--- a/config/crd/apiextensions.k8s.io/v1/backup.appuio.ch_schedules.yaml
+++ b/config/crd/apiextensions.k8s.io/v1/backup.appuio.ch_schedules.yaml
@@ -1641,6 +1641,12 @@ spec:
             type: object
           status:
             description: ScheduleStatus defines the observed state of Schedule
+            properties:
+              effectiveSchedules:
+                additionalProperties:
+                  type: string
+                description: EffectiveSchedules displays the final schedule for each type (useful when using smart schedules).
+                type: object
             type: object
         type: object
     served: true
diff --git a/config/crd/apiextensions.k8s.io/v1beta1/backup.appuio.ch_schedules.yaml b/config/crd/apiextensions.k8s.io/v1beta1/backup.appuio.ch_schedules.yaml
index 0b16d6fc..74323ca9 100644
--- a/config/crd/apiextensions.k8s.io/v1beta1/backup.appuio.ch_schedules.yaml
+++ b/config/crd/apiextensions.k8s.io/v1beta1/backup.appuio.ch_schedules.yaml
@@ -1641,6 +1641,12 @@ spec:
           type: object
         status:
           description: ScheduleStatus defines the observed state of Schedule
+          properties:
+            effectiveSchedules:
+              additionalProperties:
+                type: string
+              description: EffectiveSchedules displays the final schedule for each type (useful when using smart schedules).
+              type: object
           type: object
       type: object
   version: v1alpha1
diff --git a/config/samples/k8up_v1alpha1_schedule.yaml b/config/samples/k8up_v1alpha1_schedule.yaml
index d5bcb189..7ade56da 100644
--- a/config/samples/k8up_v1alpha1_schedule.yaml
+++ b/config/samples/k8up_v1alpha1_schedule.yaml
@@ -52,7 +52,7 @@ spec:
       requests:
         memory: "64Mi"
         cpu: "250m"
-    schedule: '*/5 * * * *'
+    schedule: '@hourly-random'
     promURL: http://10.144.1.224:9000
   prune:
     schedule: '*/2 * * * *'
diff --git a/docs/modules/ROOT/examples/schedule-status.yaml b/docs/modules/ROOT/examples/schedule-status.yaml
new file mode 100644
index 00000000..0503d96e
--- /dev/null
+++ b/docs/modules/ROOT/examples/schedule-status.yaml
@@ -0,0 +1,10 @@
+apiVersion: backup.appuio.ch/v1alpha1
+kind: Schedule
+metadata:
+  name: schedule-test
+spec:
+  check:
+    schedule: '@hourly-random'
+status:
+  effectiveSchedules:
+    check: 44 * * * *
diff --git a/docs/modules/ROOT/pages/explanations/missing-docs.adoc b/docs/modules/ROOT/pages/explanations/missing-docs.adoc
index 18e66a4a..58e6b3f3 100644
--- a/docs/modules/ROOT/pages/explanations/missing-docs.adoc
+++ b/docs/modules/ROOT/pages/explanations/missing-docs.adoc
@@ -6,7 +6,6 @@ For the following keywords documentation is missing and will be added over time:
 * Webhooks with backup information
 * Prometheus Metrics of Operator
 * Prometheus Pushgateway Metrics of Wrestic
-* Smart Schedules
 * K8up HA (leader election)
 * Index of possible annotations
 * Generated CRD API Doc just like in Lieutenant Operator
diff --git a/docs/modules/ROOT/pages/how-tos/optimize-schedules.adoc b/docs/modules/ROOT/pages/how-tos/optimize-schedules.adoc
new file mode 100644
index 00000000..e4a31fed
--- /dev/null
+++ b/docs/modules/ROOT/pages/how-tos/optimize-schedules.adoc
@@ -0,0 +1,24 @@
+= How to Optimize schedules across the cluster
+
+K8up features advanced scheduling mechanisms that allow you to optimize backup and other schedules.
+
+When you start having hundreds of backup, check, prune or archive schedules it can become a bad idea to run them all at the same time.
+However, manually trying to balance the schedules to different times is no solution either.
+
+Enter smart schedules.
+In addition to the standard Cron syntax (e.g. `* */12 * * *`) K8up features stable randomization of schedules.
+Enter a special `-random`-suffixed schedule to your spec in order to let K8up generate a schedule for you.
+
+A schedule of `@weekly-random` generates an effective schedule like `52 4 * * 4`.
+The effective schedule per job type can be inspected in the `.status.effectiveSchedule` field of a `Schedule` object.
+
+[source,yaml]
+----
+include::example$schedule-status.yaml[]
+----
+
+TIP: The full specification of the schedules can be found in the xref:references/schedule-specification.adoc[Schedule Specifications]
+
+The generated schedules are randomized with a stable seed based on the namespace and name of the schedule object.
+That means that every schedule object will have different schedules, but deleting and recreating the same schedule will have the same effective schedules.
+This behaviour allows portability between clusters.
diff --git a/docs/modules/ROOT/pages/references/schedule-specification.adoc b/docs/modules/ROOT/pages/references/schedule-specification.adoc
new file mode 100644
index 00000000..b5a3d55f
--- /dev/null
+++ b/docs/modules/ROOT/pages/references/schedule-specification.adoc
@@ -0,0 +1,63 @@
+= Schedule Specification
+
+K8up comes with its own integrated scheduler and supports more advanced schedule specifications.
+The standard Cron syntax (e.g. `0 */12 * * *`) is supported (see https://en.wikipedia.org/wiki/Cron[Wikipedia])
+
+All interpretation and scheduling is done in the machine's local time zone, as provided by the http://www.golang.org/pkg/time[Go time package].
+
+The following non-standard schedules are supported:
+
+.Non-Standard schedule specification
+|===
+|Entry in Spec|Description|Equivalent to
+
+|`@yearly` or `@annually`
+|Run once a year, midnight, Jan. 1st
+|`0 0 1 1 *`
+
+|`@monthly`
+|Run once a month, midnight, first of month
+|`0 0 1 * *`
+
+|`@weekly`
+|Run once a week, midnight between Sat/Sun
+|`0 0 * * 0`
+
+|`@daily` or `@midnight`
+|Run once a day, midnight
+|`0 0 * * *`
+
+|`@hourly`
+|Run once an hour, beginning of hour
+|`0 * * * *`
+
+|`@every <duration>`
+|Run once at fixed intervals, starting at the time it's added or K8up is started up.
+ `<duration>` is a string accepted by http://golang.org/pkg/time/#ParseDuration[time.ParseDuration] (e.g. `1h30m`).
+|
+
+|`@yearly-random` or `@annually-random`
+|Run once a year, randomized start time and date
+|`52 4 26 5 *` (example)
+
+|`@monthly-random`
+|Run once a month, randomized start time and date
+|`52 4 26 * *` (example)
+
+|`@weekly-random`
+|Run once a week, randomized start time and weekday
+|`52 4 * * 4` (example)
+
+|`@daily-random` or `@midnight`
+|Run once a day, randomized start time
+|`52 4 * * *` (example)
+
+|`@hourly-random`
+|Run once an hour, randomized minute
+|`52 * * * *` (example)
+|===
+
+NOTE: The day-of-month in the `@monthly` or `@yearly` is capped at the number 27.
+      This is because there are months that have between 27 and 31 days.
+      Cron skips a job when the day-of-month is higher than the current month has  days in it.
+      This is to prevent skipping backups and alerts from being fired.
diff --git a/docs/modules/ROOT/partials/nav-howtos.adoc b/docs/modules/ROOT/partials/nav-howtos.adoc
index b529316c..d7250744 100644
--- a/docs/modules/ROOT/partials/nav-howtos.adoc
+++ b/docs/modules/ROOT/partials/nav-howtos.adoc
@@ -2,6 +2,7 @@
 * xref:k8up:ROOT:how-tos/upgrade.adoc[Upgrade]
 * xref:k8up:ROOT:how-tos/backup.adoc[Backup]
 * xref:k8up:ROOT:how-tos/schedules.adoc[Schedules]
+* xref:k8up:ROOT:how-tos/optimize-schedules.adoc[Optimize Schedules]
 * xref:k8up:ROOT:how-tos/restore.adoc[Restore]
 * xref:k8up:ROOT:how-tos/check-status.adoc[Check Status]
 * xref:k8up:ROOT:how-tos/application-aware-backups.adoc[Application-Aware Backups]
diff --git a/docs/modules/ROOT/partials/nav-reference.adoc b/docs/modules/ROOT/partials/nav-reference.adoc
index 169cf7bb..238856f3 100644
--- a/docs/modules/ROOT/partials/nav-reference.adoc
+++ b/docs/modules/ROOT/partials/nav-reference.adoc
@@ -1,2 +1,3 @@
 * xref:k8up:ROOT:references/config-reference.adoc[Configuration]
 * xref:k8up:ROOT:references/object-specifications.adoc[Object Specifications]
+* xref:k8up:ROOT:references/schedule-specification.adoc[Schedule Specifications]
diff --git a/handler/job.go b/handler/job.go
index d98da90d..6b175f92 100644
--- a/handler/job.go
+++ b/handler/job.go
@@ -1,7 +1,6 @@
 package handler
 
 import (
-	"context"
 	"fmt"
 	"strconv"
 
@@ -10,7 +9,6 @@ import (
 	"github.com/vshn/k8up/observer"
 	batchv1 "k8s.io/api/batch/v1"
 	"k8s.io/apimachinery/pkg/api/errors"
-	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 )
 
@@ -23,6 +21,7 @@ const (
 type JobHandler struct {
 	job *batchv1.Job
 	job.Config
+	requireSpecUpdate bool
 }
 
 // NewJobHandler returns a new JobHandler.
@@ -45,20 +44,13 @@ func (j *JobHandler) Handle() error {
 
 	if j.job.GetDeletionTimestamp() != nil && contains(j.job.GetFinalizers(), jobFinalizerName) {
 		jobEvent = observer.Delete
-		err := removeFinalizer(j.CTX, j.job, jobFinalizerName, j.Client)
-		if err != nil {
-			if errors.IsNotFound(err) {
-				return nil
-			}
-			return fmt.Errorf("error removing finalizer: %w", err)
-		}
+		controllerutil.RemoveFinalizer(j.job, jobFinalizerName)
+		j.requireSpecUpdate = true
 	} else {
 		if j.job.Status.Active > 0 {
 			jobEvent = observer.Running
-			err := addFinalizer(j.CTX, j.job, jobFinalizerName, j.Client)
-			if err != nil {
-				return err
-			}
+			controllerutil.AddFinalizer(j.job, jobFinalizerName)
+			j.requireSpecUpdate = true
 		}
 
 		if j.job.Status.Succeeded > 0 {
@@ -88,27 +80,12 @@ func (j *JobHandler) Handle() error {
 		Event:     jobEvent,
 	}
 
-	observer.GetObserver().GetUpdateChannel() <- oj
-	return nil
-}
-
-func addFinalizer(ctx context.Context, obj controllerutil.Object, name string, client client.Client) error {
-	controllerutil.AddFinalizer(obj, name)
-
-	// Update CR
-	err := client.Update(ctx, obj)
-	if err != nil {
-		return err
-	}
-	return nil
-}
-
-func removeFinalizer(ctx context.Context, obj controllerutil.Object, name string, client client.Client) error {
-	controllerutil.RemoveFinalizer(obj, name)
-	err := client.Update(ctx, obj)
-	if err != nil {
-		return err
+	if j.requireSpecUpdate {
+		if err := j.Client.Update(j.CTX, j.job); err != nil && !errors.IsNotFound(err) {
+			return fmt.Errorf("error updating resource: %w", err)
+		}
 	}
+	observer.GetObserver().GetUpdateChannel() <- oj
 	return nil
 }
 
diff --git a/handler/randomizer.go b/handler/randomizer.go
new file mode 100644
index 00000000..18929843
--- /dev/null
+++ b/handler/randomizer.go
@@ -0,0 +1,67 @@
+package handler
+
+import (
+	"crypto/sha1"
+	"encoding/hex"
+	"fmt"
+	k8upv1alpha1 "github.com/vshn/k8up/api/v1alpha1"
+	"math/big"
+)
+
+const (
+	ScheduleHourlyRandom   = "@hourly-random"
+	ScheduleDailyRandom    = "@daily-random"
+	ScheduleYearlyRandom   = "@yearly-random"
+	ScheduleAnnuallyRandom = "@annually-random"
+	ScheduleMonthlyRandom  = "@monthly-random"
+	ScheduleWeeklyRandom   = "@weekly-random"
+)
+
+func (s *ScheduleHandler) createSeed(schedule *k8upv1alpha1.Schedule, jobType k8upv1alpha1.JobType) string {
+	return schedule.Namespace + "/" + schedule.Name + "@" + string(jobType)
+}
+
+// randomizeSchedule randomizes the given originalSchedule with a seed. The originalSchedule has to be one of the supported
+// '@x-random' predefined schedules, otherwise it returns an error with the original schedule unmodified.
+func randomizeSchedule(seed, originalSchedule string) (string, error) {
+	checksum := calculateChecksumFromSeed(seed)
+
+	minute := remainderFromModulo(checksum, 60, 0)
+	hour := remainderFromModulo(checksum, 24, 0)
+	// A month can have between 27 and 31 days. Cron does not automatically schedule at the end of the month if 31 in February.
+	// To not cause a skip of a scheduled backup that could potentially raise alerts, we cap the day-of-month to 27 so it fits in all months.
+	dayOfMonth := remainderFromModulo(checksum, 27, 1)
+
+	switch originalSchedule {
+	case ScheduleHourlyRandom:
+		return fmt.Sprintf("%d * * * *", minute), nil
+	case ScheduleDailyRandom:
+		return fmt.Sprintf("%d %d * * *", minute, hour), nil
+	case ScheduleMonthlyRandom:
+		return fmt.Sprintf("%d %d %d * *", minute, hour, dayOfMonth), nil
+	case ScheduleAnnuallyRandom:
+	case ScheduleYearlyRandom:
+		month := remainderFromModulo(checksum, 12, 1)
+		return fmt.Sprintf("%d %d %d %d *", minute, hour, dayOfMonth, month), nil
+	case ScheduleWeeklyRandom:
+		weekday := remainderFromModulo(checksum, 6, 0)
+		return fmt.Sprintf("%d %d * * %d", minute, hour, weekday), nil
+	default:
+		return originalSchedule, fmt.Errorf("unrecognized random schedule: '%s'", originalSchedule)
+	}
+	return originalSchedule, nil
+}
+
+// calculateChecksumFromSeed calculates a SHA1 hexadecimal checksum from the given seed.
+func calculateChecksumFromSeed(seed string) *big.Int {
+	hash := sha1.New()
+	hash.Write([]byte(seed))
+	sum := hex.EncodeToString(hash.Sum(nil))
+	sumBase16, _ := new(big.Int).SetString(sum, 16)
+	return sumBase16
+}
+
+// remainderFromModulo calculates the remainder from (dividend % divisor), then adds the offset.
+func remainderFromModulo(dividend *big.Int, divisor, offset int64) *big.Int {
+	return new(big.Int).Add(big.NewInt(offset), new(big.Int).Mod(dividend, big.NewInt(divisor)))
+}
diff --git a/handler/schedule.go b/handler/schedule.go
index da4c39b1..d0af8e1f 100644
--- a/handler/schedule.go
+++ b/handler/schedule.go
@@ -8,7 +8,10 @@ import (
 	"github.com/vshn/k8up/job"
 	"github.com/vshn/k8up/scheduler"
 	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/errors"
 	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
+	"strings"
 )
 
 const (
@@ -20,6 +23,8 @@ const (
 type ScheduleHandler struct {
 	schedule *k8upv1alpha1.Schedule
 	job.Config
+	requireSpecUpdate   bool
+	requireStatusUpdate bool
 }
 
 // NewScheduleHandler will return a new ScheduleHandler.
@@ -37,14 +42,10 @@ func (s *ScheduleHandler) Handle() error {
 	namespacedName := types.NamespacedName{Name: s.schedule.GetName(), Namespace: s.schedule.GetNamespace()}
 
 	if s.schedule.GetDeletionTimestamp() != nil {
-		err := removeFinalizer(s.CTX, s.schedule, scheduleFinalizerName, s.Client)
-		if err != nil {
-			return fmt.Errorf("error while removing the finalizer: %w", err)
-		}
-
+		controllerutil.RemoveFinalizer(s.schedule, scheduleFinalizerName)
 		scheduler.GetScheduler().RemoveSchedules(namespacedName)
 
-		return nil
+		return s.updateSchedule()
 	}
 
 	var err error
@@ -56,16 +57,19 @@ func (s *ScheduleHandler) Handle() error {
 		return fmt.Errorf("cannot add to cron: %w", err)
 	}
 
-	if !contains(s.schedule.GetFinalizers(), scheduleFinalizerName) {
-		err = addFinalizer(s.CTX, s.schedule, scheduleFinalizerName, s.Client)
-		if err != nil {
-			return fmt.Errorf("error while adding finalizer: %w", err)
-		}
+	if !controllerutil.ContainsFinalizer(s.schedule, scheduleFinalizerName) {
+		controllerutil.AddFinalizer(s.schedule, scheduleFinalizerName)
+		s.requireSpecUpdate = true
 	}
 
+	if s.requireSpecUpdate {
+		return s.updateSchedule()
+	}
+	if s.requireStatusUpdate {
+		return s.updateStatus()
+	}
 	return nil
 }
-
 func (s *ScheduleHandler) createJobList() scheduler.JobList {
 	jobList := scheduler.JobList{
 		Config: s.Config,
@@ -76,7 +80,7 @@ func (s *ScheduleHandler) createJobList() scheduler.JobList {
 		s.schedule.Spec.Archive.ArchiveSpec.Resources = s.mergeResourcesWithDefaults(s.schedule.Spec.Archive.Resources)
 		jobList.Jobs = append(jobList.Jobs, scheduler.Job{
 			Type:     scheduler.ArchiveType,
-			Schedule: s.schedule.Spec.Archive.Schedule,
+			Schedule: s.getEffectiveSchedule(k8upv1alpha1.ArchiveType, s.schedule.Spec.Archive.Schedule),
 			Object:   s.schedule.Spec.Archive.ArchiveSpec,
 		})
 	}
@@ -84,7 +88,7 @@ func (s *ScheduleHandler) createJobList() scheduler.JobList {
 		s.schedule.Spec.Backup.BackupSpec.Resources = s.mergeResourcesWithDefaults(s.schedule.Spec.Backup.Resources)
 		jobList.Jobs = append(jobList.Jobs, scheduler.Job{
 			Type:     scheduler.BackupType,
-			Schedule: s.schedule.Spec.Backup.Schedule,
+			Schedule: s.getEffectiveSchedule(k8upv1alpha1.BackupType, s.schedule.Spec.Backup.Schedule),
 			Object:   s.schedule.Spec.Backup.BackupSpec,
 		})
 	}
@@ -92,7 +96,7 @@ func (s *ScheduleHandler) createJobList() scheduler.JobList {
 		s.schedule.Spec.Check.CheckSpec.Resources = s.mergeResourcesWithDefaults(s.schedule.Spec.Check.Resources)
 		jobList.Jobs = append(jobList.Jobs, scheduler.Job{
 			Type:     scheduler.CheckType,
-			Schedule: s.schedule.Spec.Check.Schedule,
+			Schedule: s.getEffectiveSchedule(k8upv1alpha1.CheckType, s.schedule.Spec.Check.Schedule),
 			Object:   s.schedule.Spec.Check.CheckSpec,
 		})
 	}
@@ -100,7 +104,7 @@ func (s *ScheduleHandler) createJobList() scheduler.JobList {
 		s.schedule.Spec.Restore.RestoreSpec.Resources = s.mergeResourcesWithDefaults(s.schedule.Spec.Restore.Resources)
 		jobList.Jobs = append(jobList.Jobs, scheduler.Job{
 			Type:     scheduler.RestoreType,
-			Schedule: s.schedule.Spec.Restore.Schedule,
+			Schedule: s.getEffectiveSchedule(k8upv1alpha1.RestoreType, s.schedule.Spec.Restore.Schedule),
 			Object:   s.schedule.Spec.Restore.RestoreSpec,
 		})
 	}
@@ -108,7 +112,7 @@ func (s *ScheduleHandler) createJobList() scheduler.JobList {
 		s.schedule.Spec.Prune.PruneSpec.Resources = s.mergeResourcesWithDefaults(s.schedule.Spec.Prune.Resources)
 		jobList.Jobs = append(jobList.Jobs, scheduler.Job{
 			Type:     scheduler.PruneType,
-			Schedule: s.schedule.Spec.Prune.Schedule,
+			Schedule: s.getEffectiveSchedule(k8upv1alpha1.PruneType, s.schedule.Spec.Prune.Schedule),
 			Object:   s.schedule.Spec.Prune.PruneSpec,
 		})
 	}
@@ -125,3 +129,43 @@ func (s *ScheduleHandler) mergeResourcesWithDefaults(resources corev1.ResourceRe
 	}
 	return resources
 }
+
+func (s *ScheduleHandler) updateSchedule() error {
+	if err := s.Client.Update(s.CTX, s.schedule); err != nil && !errors.IsNotFound(err) {
+		return fmt.Errorf("error updating resource %s/%s: %w", s.schedule.Namespace, s.schedule.Name, err)
+	}
+	return nil
+}
+
+func (s *ScheduleHandler) updateStatus() error {
+	err := s.Client.Status().Update(s.CTX, s.schedule)
+	if err != nil {
+		s.Log.Error(err, "Could not update SyncConfig.", "name", s.schedule)
+		return err
+	}
+	s.Log.Info("Updated SyncConfig status.")
+	return nil
+}
+
+func (s *ScheduleHandler) getEffectiveSchedule(jobType k8upv1alpha1.JobType, originalSchedule string) string {
+	if s.schedule.Status.EffectiveSchedules == nil {
+		s.schedule.Status.EffectiveSchedules = make(map[k8upv1alpha1.JobType]string)
+	}
+	if existingSchedule, found := s.schedule.Status.EffectiveSchedules[jobType]; found {
+		return existingSchedule
+	}
+	if !strings.HasSuffix(originalSchedule, "-random") {
+		return originalSchedule
+	}
+	schedule := originalSchedule
+	seed := s.createSeed(s.schedule, jobType)
+	schedule, err := randomizeSchedule(seed, originalSchedule)
+	if err != nil {
+		s.Log.Info("Could not randomize schedule, ignoring and try original schedule", "error", err.Error())
+	} else {
+		s.Log.V(1).Info("Randomized schedule", "seed", seed, "from_schedule", originalSchedule, "effective_schedule", schedule)
+	}
+	s.schedule.Status.EffectiveSchedules[jobType] = schedule
+	s.requireStatusUpdate = true
+	return schedule
+}
diff --git a/scheduler/scheduler.go b/scheduler/scheduler.go
index 2c1498af..4872419c 100644
--- a/scheduler/scheduler.go
+++ b/scheduler/scheduler.go
@@ -104,17 +104,13 @@ func (s *Scheduler) AddSchedules(jobs JobList) error {
 
 	jobIDs := make([]int, len(jobs.Jobs))
 
-	for i, job := range jobs.Jobs {
+	for i, jb := range jobs.Jobs {
 
-		jobs.Config.Log.Info("registering schedule for", "type", job.Type)
+		jobs.Config.Log.Info("registering schedule for", "type", jb.Type)
 
-		jobType := job.Type
-		obj := job.Object
-		config := jobs.Config
-
-		id, err := s.cron.AddFunc(job.Schedule, func() {
-			jobs.Config.Log.Info("running schedule for", "job", jobType)
-			s.createObject(jobType, namespacedName.Namespace, obj, config)
+		id, err := s.cron.AddFunc(jb.Schedule, func() {
+			jobs.Config.Log.Info("running schedule for", "jb", jb.Type)
+			s.createObject(jb.Type, namespacedName.Namespace, jb.Object, jobs.Config)
 		})
 		if err != nil {
 			return err


Test patch:
diff --git a/handler/randomizer_test.go b/handler/randomizer_test.go
new file mode 100644
index 00000000..16bf0246
--- /dev/null
+++ b/handler/randomizer_test.go
@@ -0,0 +1,80 @@
+package handler
+
+import (
+	"github.com/stretchr/testify/assert"
+	"strconv"
+	"strings"
+	"testing"
+)
+
+func Test_randomizeSchedule_VerifyCronSyntax(t *testing.T) {
+	formatString := "%d in '%s' is not between %d and %d"
+	arr := []int{0, 0, 0, 0, 0}
+	t.Run("YearlySyntax", func(t *testing.T) {
+		for i := 0; i < 200; i++ {
+			seed := "namespace/name-" + strconv.Itoa(i) + "@backup"
+			schedule, err := randomizeSchedule(seed, "@yearly-random")
+			assert.NoError(t, err)
+			fields := strings.Split(schedule, " ")
+			for j, f := range fields {
+				number, _ := strconv.Atoi(f)
+				arr[j] = number
+			}
+			assert.InDelta(t, 0, arr[0], 59.0, formatString, arr[0], "minute", 0, 59)
+			assert.InDelta(t, 0, arr[1], 23.0, formatString, arr[1], "hour", 0, 23)
+			assert.InDelta(t, 1, arr[2], 26.0, formatString, arr[2], "dayOfMonth", 1, 27)
+			assert.InDelta(t, 1, arr[3], 11.0, formatString, arr[3], "month", 1, 12)
+		}
+	})
+	t.Run("WeeklySyntax", func(t *testing.T) {
+		for i := 0; i < 100; i++ {
+			seed := "namespace/name-" + strconv.Itoa(i) + "@backup"
+			schedule, err := randomizeSchedule(seed, "@weekly-random")
+			assert.NoError(t, err)
+			fields := strings.Split(schedule, " ")
+			for j, f := range fields {
+				number, _ := strconv.Atoi(f)
+				arr[j] = number
+			}
+			assert.InDelta(t, 0, arr[0], 59.0, formatString, arr[0], "minute", 0, 59)
+			assert.InDelta(t, 0, arr[1], 23.0, formatString, arr[1], "hour", 0, 23)
+			assert.InDelta(t, 1, arr[4], 5.0, formatString, arr[4], "weekday", 0, 6)
+		}
+	})
+}
+
+func Test_randomizeSchedule_VerifySchedules(t *testing.T) {
+	seed := "k8up-system/my-scheduled-backup@backup"
+	tests := map[string]struct {
+		schedule         string
+		expectedSchedule string
+	}{
+		"WhenScheduleRandomHourlyGiven_ThenReturnStableRandomizedSchedule": {
+			schedule:         "@hourly-random",
+			expectedSchedule: "52 * * * *",
+		},
+		"WhenScheduleRandomDailyGiven_ThenReturnStableRandomizedSchedule": {
+			schedule:         "@daily-random",
+			expectedSchedule: "52 4 * * *",
+		},
+		"WhenScheduleRandomWeeklyGiven_ThenReturnStableRandomizedSchedule": {
+			schedule:         "@weekly-random",
+			expectedSchedule: "52 4 * * 4",
+		},
+		"WhenScheduleRandomMonthlyGiven_ThenReturnStableRandomizedSchedule": {
+			schedule:         "@monthly-random",
+			expectedSchedule: "52 4 26 * *",
+		},
+		"WhenScheduleRandomYearlyGiven_ThenReturnStableRandomizedSchedule": {
+			schedule:         "@yearly-random",
+			expectedSchedule: "52 4 26 5 *",
+		},
+	}
+	for name, tt := range tests {
+		t.Run(name, func(t *testing.T) {
+			result, err := randomizeSchedule(seed, tt.schedule)
+			assert.Equal(t, tt.expectedSchedule, result)
+			assert.NoError(t, err)
+		})
+	}
+}
diff --git a/handler/schedule_test.go b/handler/schedule_test.go
index 51bf576a..e66a9d65 100644
--- a/handler/schedule_test.go
+++ b/handler/schedule_test.go
@@ -5,8 +5,10 @@ import (
 	"github.com/stretchr/testify/require"
 	"github.com/vshn/k8up/api/v1alpha1"
 	"github.com/vshn/k8up/cfg"
+	"github.com/vshn/k8up/job"
 	v1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
+	"sigs.k8s.io/controller-runtime/pkg/log/zap"
 	"testing"
 )
 
@@ -150,3 +152,51 @@ func TestScheduleHandler_mergeResourcesWithDefaults(t *testing.T) {
 		})
 	}
 }
+
+func TestScheduleHandler_getEffectiveSchedule(t *testing.T) {
+	tests := map[string]struct {
+		schedule             *v1alpha1.Schedule
+		originalSchedule     string
+		expectedStatusUpdate bool
+		expectedSchedule     string
+	}{
+		"GivenScheduleWithoutStatus_WhenUsingRandomSchedule_ThenPutGeneratedScheduleInStatus": {
+			schedule: &v1alpha1.Schedule{
+				Spec: v1alpha1.ScheduleSpec{
+					Backup: &v1alpha1.BackupSchedule{},
+				},
+			},
+			originalSchedule:     "@hourly-random",
+			expectedSchedule:     "26 * * * *",
+			expectedStatusUpdate: true,
+		},
+		"GivenScheduleWithStatus_WhenUsingRandomSchedule_ThenUseGeneratedScheduleFromStatus": {
+			schedule: &v1alpha1.Schedule{
+				Spec: v1alpha1.ScheduleSpec{
+					Backup: &v1alpha1.BackupSchedule{},
+				},
+				Status: v1alpha1.ScheduleStatus{
+					EffectiveSchedules: map[v1alpha1.JobType]string{
+						v1alpha1.BackupType: "26 * 3 * *",
+					},
+				},
+			},
+			originalSchedule: "@hourly-random",
+			expectedSchedule: "26 * 3 * *",
+		},
+	}
+	for name, tt := range tests {
+		t.Run(name, func(t *testing.T) {
+			s := &ScheduleHandler{
+				schedule: tt.schedule,
+				Config:   job.Config{Log: zap.New(zap.UseDevMode(true))},
+			}
+			result := s.getEffectiveSchedule(v1alpha1.BackupType, tt.originalSchedule)
+			assert.Equal(t, tt.expectedSchedule, result)
+			assert.Equal(t, tt.expectedStatusUpdate, s.requireStatusUpdate)
+			if tt.expectedStatusUpdate {
+				assert.Equal(t, tt.expectedSchedule, tt.schedule.Status.EffectiveSchedules[v1alpha1.BackupType])
+			}
+		})
+	}
+}