Repository: prometheus/prometheus (language: go)

Issue:
Support reload config file automatically
<!--

    Please do *NOT* ask support questions in Github issues.

    If your issue is not a feature request or bug report use our
    community support.

    https://prometheus.io/community/

    There is also commercial support available.

    https://prometheus.io/support-training/

-->
## Proposal
Support reload config file automatically without calling the reload API, when the file is changed.



Discussion / hints:
Thanks for your proposal!

We decided that only allowing SIGHUP (and by extension the reload API) would be the best when reloading was first implemented. See the comment thread: https://github.com/prometheus/prometheus/issues/108#issuecomment-104318203.
Hello,

Actually while the consensus 6 years ago was not to have this, since then many people implemented their own reloaders.

I think we should revisit this decision.

The new reloader would:

- be enabled behind a feature flag to start with
- use inotify where possible, manual reload otherwise (and look at checksum to see if there are differences)
- throttle config reloads (e.g. not more than one/minute)

Let's see if we can get consensus on this: @beorn7 @brancz @discordianfish @superq
Rather than a feature flag, a simple normal boolean flag would be sufficient. Not every new feature we add needs a feature flag.

```
--config.auto-reload
```
> Rather than a feature flag, a simple normal boolean flag would be sufficient. Not every new feature we add needs a feature flag.
> 
> ```
> --config.auto-reload
> ```

ack, should it take the minimum interval ?
Agree that normal flag should be sufficient, but I think we'll need two flags/options: Poll interval if not using inotify and max reload per minute. So I'd keep it a boolean flag and have a separate one for the other options.
I don't remember the arguments against auto-reloading from back then (and confusingly, I said myself I would be able to make a point against it in that [mysterious comment](https://github.com/prometheus/prometheus/issues/108#issuecomment-104318203), but I really cannot remember).

In general, as an optional feature, I don't have objections right now, but knowing how the discussion back then went would be better.
https://github.com/prometheus/prometheus/issues/108#issuecomment-92288938 and https://github.com/prometheus/prometheus/issues/108#issuecomment-104318819 would be the arguments, so it ended up done for everything except the config+rules files.

Something might update the rules/configs in any order, or briefly have partial files on disk.
So the main argument would be that the config to reload is inherently multiple files (because rules are in a separate file, and then we even plan to support .d style config diretories).

Avoiding partial files is fairly easy with a single file (via renaming).

The question is if the increased difficulty to update the set of files is prohibitive for providing auto-reload as an optional feature. (I guess we all agree that it should not become the default behavior, not even in 3.x.)
https://github.com/gmiroshnykov/prometheus-reloader
https://github.com/GoogleCloudPlatform/prometheus-engine/tree/main/cmd/config-reloader
https://github.com/prometheus-operator/prometheus-operator/tree/main/cmd/prometheus-config-reloader
https://github.com/thanos-io/thanos/blob/main/pkg/reloader/reloader.go


That problem has been solved at least 4 times by the community. I suggest we look at those implementations, maybe write down how they work in a small design doc, and adopt the best behaviour.
> https://github.com/gmiroshnykov/prometheus-reloader https://github.com/GoogleCloudPlatform/prometheus-engine/tree/main/cmd/config-reloader https://github.com/prometheus-operator/prometheus-operator/tree/main/cmd/prometheus-config-reloader https://github.com/thanos-io/thanos/blob/main/pkg/reloader/reloader.go
> 
> That problem has been solved at least 4 times by the community. I suggest we look at those implementations, maybe write down how they work in a small design doc, and adopt the best behaviour.

+1
@roidelapluie just FYI the prometheus-operator is wrapping https://github.com/thanos-io/thanos/blob/main/pkg/reloader/reloader.go under the hood
..and I'm using https://github.com/jimmidyson/configmap-reload :)
At GL, we are using https://github.com/weaveworks/Watch , which is a fork of https://github.com/eaburns/Watch, and which isn't even doing anything fancy. It still seems to work just fine (but perhaps that's because configmap updates are (almost) atomic?).
👋🏽  It would be indeed epic to revisit this. I would love to propose (list done with the help from @roidelapluie):

- [ ] `--config.watch-interval` (default 0) duration flag which, when non zero, watches changes of Prometheus config file, previously provided rule files, as well as previously provided scrape_config_files. When change is noticed, Prometheus will self-reload.
- [ ] Generally we propose to add core watching code similar to already mentioned [thanos reloader](https://github.com/thanos-io/thanos/blob/main/pkg/reloader/reloader.go#L540), but without compression or environment substitution features. It could be slightly generic e.g. in `util/fswatch` and used later for [file SD](https://github.com/prometheus/prometheus/blob/main/discovery/file/file.go#L242) if we want. It would:
  * Watch given files using inotify (inotify will be on dirs), with regular interval to catch inotify flaky cases
  * Check if specified file really changed using checksums
  * It has be able to remove files from watching, not only adding (new feature of reloader)
  * It will notify the callers through Go channel when any of the given file changed.
  * It will have throttling behaviour. Perhaps it deserves additional `--config.watch-delay-duration` duration flag. We need that to only provide one notification after bigger update, instead of multiple updates within seconds for multiple files changed at once. One simple solution is [delay functionality](https://github.com/thanos-io/thanos/blob/main/pkg/reloader/reloader.go#L123) reload offers.
- [ ] Such watcher will be instantiated in the main Prometheus loop (after reload is ready). Change notifications will trigger reload like SIGHUP/reload API would (aka "self-reload")

Since the feature will be optional, we don't envision polluting feature flags for this, but we would still mark it as experimental. If we prefer feature flag, let me know.

We (Prometheus Managed Team at Google) are happy to contribute/lead this.

## Motivation

No extra sidecar will be required for common file watching logic,  especially popular with Kubernetes and ConfigMaps. This is especially important for managed setups (like ours) where we have strict alerting and SLIs on each component. The problem with config reloader sidecar is that it's has strong dependency on Prometheus being up, thus the alerts and SLIs has to be even move complex (to avoid noise), which is overkill for such a simple (yet important) functionality

## Background

Did some research, but most of it was mentioned here already:

Back in 2015 https://github.com/prometheus/prometheus/issues/108, we agreed
that SIGHUP only reloading is solid enough. The main argument against file
watching was that it's not implemented by all file systems. Further more, not
all filesystem file changes are atomic. Finally, it was noticed that inotify can
be flaky, even on modern Linux OSes.

However, despite the above decision Prometheus started to use file watchers for [file SD](https://github.com/prometheus/prometheus/blob/02277bbe0dd2d89fd3fe5f508399d54da9c61827/discovery/file/file.go#L242) (with some safety intervals)

Furthermore, with those flaws acknowledged, some advanced setups of Prometheus uses watch mechanisms
with some safety re-read interval (and extra features beyond this PR):

* Thanos Sidecar uses [reload](https://github.com/thanos-io/thanos/blob/main/pkg/reloader/reloader.go) package.
* Prometheus Operator adds a [config-reloader sidecar](https://github.com/prometheus-operator/prometheus-operator/blob/main/pkg/operator/config_reloader.go#L153) build from [here](https://github.com/prometheus-operator/prometheus-operator/blob/main/cmd/prometheus-config-reloader/main.go) based on Thanos reloader.
* Similarly, our Google operator adds [a config-reloader sidecar](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/main/manifests/operator.yaml#L562) build from [here](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/main/cmd/config-reloader/main.go) and also based on Thanos reloader.


Yeah I'm (still) very much in favor of supporting this.
> watches changes of Prometheus config file, previously provided rule files, as well as previously provided scrape_config_files

Note: This is not enough to replace side-cars. If that is a goal, then you would need to watch all file configurations "_files" within all scrape configs. For example, credentials files such as those used for mTLS `cert_file` where a certificate or key would expire or be rotated need to be watched for changes. The filepaths may also change dynamically so Prometheus would have to watch and unwatch files, as needed.
_files like TLS certificates do not require reloads. They are read on every request.
Thanks for the clarification!
Hello from the bug scrub!

We think this is still relevant and would welcome contributions.

Fix patch:
diff --git a/cmd/prometheus/main.go b/cmd/prometheus/main.go
index 65ffd7de5aa..3840c941557 100644
--- a/cmd/prometheus/main.go
+++ b/cmd/prometheus/main.go
@@ -154,6 +154,9 @@ type flagConfig struct {
 	RemoteFlushDeadline model.Duration
 	nameEscapingScheme  string
 
+	enableAutoReload   bool
+	autoReloadInterval model.Duration
+
 	featureList   []string
 	memlimitRatio float64
 	// These options are extracted from featureList
@@ -212,6 +215,12 @@ func (c *flagConfig) setFeatureListOptions(logger log.Logger) error {
 			case "auto-gomaxprocs":
 				c.enableAutoGOMAXPROCS = true
 				level.Info(logger).Log("msg", "Automatically set GOMAXPROCS to match Linux container CPU quota")
+			case "auto-reload-config":
+				c.enableAutoReload = true
+				if s := time.Duration(c.autoReloadInterval).Seconds(); s > 0 && s < 1 {
+					c.autoReloadInterval, _ = model.ParseDuration("1s")
+				}
+				level.Info(logger).Log("msg", fmt.Sprintf("Enabled automatic configuration file reloading. Checking for configuration changes every %s.", c.autoReloadInterval))
 			case "auto-gomemlimit":
 				c.enableAutoGOMEMLIMIT = true
 				level.Info(logger).Log("msg", "Automatically set GOMEMLIMIT to match Linux container or system memory limit")
@@ -302,6 +311,9 @@ func main() {
 	a.Flag("config.file", "Prometheus configuration file path.").
 		Default("prometheus.yml").StringVar(&cfg.configFile)
 
+	a.Flag("config.auto-reload-interval", "Specifies the interval for checking and automatically reloading the Prometheus configuration file upon detecting changes.").
+		Default("30s").SetValue(&cfg.autoReloadInterval)
+
 	a.Flag("web.listen-address", "Address to listen on for UI, API, and telemetry. Can be repeated.").
 		Default("0.0.0.0:9090").StringsVar(&cfg.web.ListenAddresses)
 
@@ -492,7 +504,7 @@ func main() {
 
 	a.Flag("scrape.name-escaping-scheme", `Method for escaping legacy invalid names when sending to Prometheus that does not support UTF-8. Can be one of "values", "underscores", or "dots".`).Default(scrape.DefaultNameEscapingScheme.String()).StringVar(&cfg.nameEscapingScheme)
 
-	a.Flag("enable-feature", "Comma separated feature names to enable. Valid options: agent, auto-gomaxprocs, auto-gomemlimit, concurrent-rule-eval, created-timestamp-zero-ingestion, delayed-compaction, exemplar-storage, expand-external-labels, extra-scrape-metrics, memory-snapshot-on-shutdown, native-histograms, new-service-discovery-manager, no-default-scrape-port, otlp-write-receiver, promql-experimental-functions, promql-delayed-name-removal, promql-per-step-stats, remote-write-receiver (DEPRECATED), utf8-names. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details.").
+	a.Flag("enable-feature", "Comma separated feature names to enable. Valid options: agent, auto-gomaxprocs, auto-gomemlimit, auto-reload-config, concurrent-rule-eval, created-timestamp-zero-ingestion, delayed-compaction, exemplar-storage, expand-external-labels, extra-scrape-metrics, memory-snapshot-on-shutdown, native-histograms, new-service-discovery-manager, no-default-scrape-port, otlp-write-receiver, promql-experimental-functions, promql-delayed-name-removal, promql-per-step-stats, remote-write-receiver (DEPRECATED), utf8-names. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details.").
 		Default("").StringsVar(&cfg.featureList)
 
 	promlogflag.AddFlags(a, &cfg.promlogConfig)
@@ -1120,6 +1132,15 @@ func main() {
 		hup := make(chan os.Signal, 1)
 		signal.Notify(hup, syscall.SIGHUP)
 		cancel := make(chan struct{})
+
+		var checksum string
+		if cfg.enableAutoReload {
+			checksum, err = config.GenerateChecksum(cfg.configFile)
+			if err != nil {
+				level.Error(logger).Log("msg", "Failed to generate initial checksum for configuration file", "err", err)
+			}
+		}
+
 		g.Add(
 			func() error {
 				<-reloadReady.C
@@ -1129,6 +1150,12 @@ func main() {
 					case <-hup:
 						if err := reloadConfig(cfg.configFile, cfg.enableExpandExternalLabels, cfg.tsdb.EnableExemplarStorage, logger, noStepSubqueryInterval, reloaders...); err != nil {
 							level.Error(logger).Log("msg", "Error reloading config", "err", err)
+						} else if cfg.enableAutoReload {
+							if currentChecksum, err := config.GenerateChecksum(cfg.configFile); err == nil {
+								checksum = currentChecksum
+							} else {
+								level.Error(logger).Log("msg", "Failed to generate checksum during configuration reload", "err", err)
+							}
 						}
 					case rc := <-webHandler.Reload():
 						if err := reloadConfig(cfg.configFile, cfg.enableExpandExternalLabels, cfg.tsdb.EnableExemplarStorage, logger, noStepSubqueryInterval, reloaders...); err != nil {
@@ -1136,6 +1163,32 @@ func main() {
 							rc <- err
 						} else {
 							rc <- nil
+							if cfg.enableAutoReload {
+								if currentChecksum, err := config.GenerateChecksum(cfg.configFile); err == nil {
+									checksum = currentChecksum
+								} else {
+									level.Error(logger).Log("msg", "Failed to generate checksum during configuration reload", "err", err)
+								}
+							}
+						}
+					case <-time.Tick(time.Duration(cfg.autoReloadInterval)):
+						if !cfg.enableAutoReload {
+							continue
+						}
+						currentChecksum, err := config.GenerateChecksum(cfg.configFile)
+						if err != nil {
+							level.Error(logger).Log("msg", "Failed to generate checksum during configuration reload", "err", err)
+							continue
+						}
+						if currentChecksum == checksum {
+							continue
+						}
+						level.Info(logger).Log("msg", "Configuration file change detected, reloading the configuration.")
+
+						if err := reloadConfig(cfg.configFile, cfg.enableExpandExternalLabels, cfg.tsdb.EnableExemplarStorage, logger, noStepSubqueryInterval, reloaders...); err != nil {
+							level.Error(logger).Log("msg", "Error reloading config", "err", err)
+						} else {
+							checksum = currentChecksum
 						}
 					case <-cancel:
 						return nil
diff --git a/config/reload.go b/config/reload.go
new file mode 100644
index 00000000000..8be1b28d8ab
--- /dev/null
+++ b/config/reload.go
@@ -0,0 +1,92 @@
+// Copyright 2024 The Prometheus Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package config
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"os"
+	"path/filepath"
+
+	"gopkg.in/yaml.v2"
+)
+
+type ExternalFilesConfig struct {
+	RuleFiles         []string `yaml:"rule_files"`
+	ScrapeConfigFiles []string `yaml:"scrape_config_files"`
+}
+
+// GenerateChecksum generates a checksum of the YAML file and the files it references.
+func GenerateChecksum(yamlFilePath string) (string, error) {
+	hash := sha256.New()
+
+	yamlContent, err := os.ReadFile(yamlFilePath)
+	if err != nil {
+		return "", fmt.Errorf("error reading YAML file: %w", err)
+	}
+	_, err = hash.Write(yamlContent)
+	if err != nil {
+		return "", fmt.Errorf("error writing YAML file to hash: %w", err)
+	}
+
+	var config ExternalFilesConfig
+	if err := yaml.Unmarshal(yamlContent, &config); err != nil {
+		return "", fmt.Errorf("error unmarshalling YAML: %w", err)
+	}
+
+	dir := filepath.Dir(yamlFilePath)
+
+	for i, file := range config.RuleFiles {
+		config.RuleFiles[i] = filepath.Join(dir, file)
+	}
+	for i, file := range config.ScrapeConfigFiles {
+		config.ScrapeConfigFiles[i] = filepath.Join(dir, file)
+	}
+
+	files := map[string][]string{
+		"r": config.RuleFiles,         // "r" for rule files
+		"s": config.ScrapeConfigFiles, // "s" for scrape config files
+	}
+
+	for _, prefix := range []string{"r", "s"} {
+		for _, pattern := range files[prefix] {
+			matchingFiles, err := filepath.Glob(pattern)
+			if err != nil {
+				return "", fmt.Errorf("error finding files with pattern %q: %w", pattern, err)
+			}
+
+			for _, file := range matchingFiles {
+				// Write prefix to the hash ("r" or "s") followed by \0, then
+				// the file path.
+				_, err = hash.Write([]byte(prefix + "\x00" + file + "\x00"))
+				if err != nil {
+					return "", fmt.Errorf("error writing %q path to hash: %w", file, err)
+				}
+
+				// Read and hash the content of the file.
+				content, err := os.ReadFile(file)
+				if err != nil {
+					return "", fmt.Errorf("error reading file %s: %w", file, err)
+				}
+				_, err = hash.Write(append(content, []byte("\x00")...))
+				if err != nil {
+					return "", fmt.Errorf("error writing %q content to hash: %w", file, err)
+				}
+			}
+		}
+	}
+
+	return hex.EncodeToString(hash.Sum(nil)), nil
+}
diff --git a/docs/command-line/prometheus.md b/docs/command-line/prometheus.md
index 7d9e5a3c809..8fefa8ecc91 100644
--- a/docs/command-line/prometheus.md
+++ b/docs/command-line/prometheus.md
@@ -15,6 +15,7 @@ The Prometheus monitoring server
 | <code class="text-nowrap">-h</code>, <code class="text-nowrap">--help</code> | Show context-sensitive help (also try --help-long and --help-man). |  |
 | <code class="text-nowrap">--version</code> | Show application version. |  |
 | <code class="text-nowrap">--config.file</code> | Prometheus configuration file path. | `prometheus.yml` |
+| <code class="text-nowrap">--config.auto-reload-interval</code> | Specifies the interval for checking and automatically reloading the Prometheus configuration file upon detecting changes. | `30s` |
 | <code class="text-nowrap">--web.listen-address</code> <code class="text-nowrap">...<code class="text-nowrap"> | Address to listen on for UI, API, and telemetry. Can be repeated. | `0.0.0.0:9090` |
 | <code class="text-nowrap">--auto-gomemlimit.ratio</code> | The ratio of reserved GOMEMLIMIT memory to the detected maximum container or system memory | `0.9` |
 | <code class="text-nowrap">--web.config.file</code> | [EXPERIMENTAL] Path to configuration file that can enable TLS or authentication. |  |
@@ -57,7 +58,7 @@ The Prometheus monitoring server
 | <code class="text-nowrap">--query.max-concurrency</code> | Maximum number of queries executed concurrently. Use with server mode only. | `20` |
 | <code class="text-nowrap">--query.max-samples</code> | Maximum number of samples a single query can load into memory. Note that queries will fail if they try to load more samples than this into memory, so this also limits the number of samples a query can return. Use with server mode only. | `50000000` |
 | <code class="text-nowrap">--scrape.name-escaping-scheme</code> | Method for escaping legacy invalid names when sending to Prometheus that does not support UTF-8. Can be one of "values", "underscores", or "dots". | `values` |
-| <code class="text-nowrap">--enable-feature</code> <code class="text-nowrap">...<code class="text-nowrap"> | Comma separated feature names to enable. Valid options: agent, auto-gomaxprocs, auto-gomemlimit, concurrent-rule-eval, created-timestamp-zero-ingestion, delayed-compaction, exemplar-storage, expand-external-labels, extra-scrape-metrics, memory-snapshot-on-shutdown, native-histograms, new-service-discovery-manager, no-default-scrape-port, otlp-write-receiver, promql-experimental-functions, promql-delayed-name-removal, promql-per-step-stats, remote-write-receiver (DEPRECATED), utf8-names. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details. |  |
+| <code class="text-nowrap">--enable-feature</code> <code class="text-nowrap">...<code class="text-nowrap"> | Comma separated feature names to enable. Valid options: agent, auto-gomaxprocs, auto-gomemlimit, auto-reload-config, concurrent-rule-eval, created-timestamp-zero-ingestion, delayed-compaction, exemplar-storage, expand-external-labels, extra-scrape-metrics, memory-snapshot-on-shutdown, native-histograms, new-service-discovery-manager, no-default-scrape-port, otlp-write-receiver, promql-experimental-functions, promql-delayed-name-removal, promql-per-step-stats, remote-write-receiver (DEPRECATED), utf8-names. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details. |  |
 | <code class="text-nowrap">--log.level</code> | Only log messages with the given severity or above. One of: [debug, info, warn, error] | `info` |
 | <code class="text-nowrap">--log.format</code> | Output format of log messages. One of: [logfmt, json] | `logfmt` |
 
diff --git a/docs/feature_flags.md b/docs/feature_flags.md
index 7b07a04d0e2..1ef9efb9b11 100644
--- a/docs/feature_flags.md
+++ b/docs/feature_flags.md
@@ -265,3 +265,15 @@ This allows optionally preserving the `__name__` label via the `label_replace` a
 When enabled, changes the metric and label name validation scheme inside Prometheus to allow the full UTF-8 character set.
 By itself, this flag does not enable the request of UTF-8 names via content negotiation.
 Users will also have to set `metric_name_validation_scheme` in scrape configs to enable the feature either on the global config or on a per-scrape config basis.
+
+## Auto Reload Config
+
+`--enable-feature=auto-reload-config`
+
+When enabled, Prometheus will automatically reload its configuration file at a
+specified interval. The interval is defined by the
+`--config.auto-reload-interval` flag, which defaults to `30s`.
+
+Configuration reloads are triggered by detecting changes in the checksum of the
+main configuration file or any referenced files, such as rule and scrape
+configurations.


Test patch:
diff --git a/config/reload_test.go b/config/reload_test.go
new file mode 100644
index 00000000000..f0f44f35880
--- /dev/null
+++ b/config/reload_test.go
@@ -0,0 +1,222 @@
+// Copyright 2024 The Prometheus Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package config
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestGenerateChecksum(t *testing.T) {
+	tmpDir := t.TempDir()
+
+	// Define paths for the temporary files.
+	yamlFilePath := filepath.Join(tmpDir, "test.yml")
+	ruleFilePath := filepath.Join(tmpDir, "rule_file.yml")
+	scrapeConfigFilePath := filepath.Join(tmpDir, "scrape_config.yml")
+
+	// Define initial and modified content for the files.
+	originalRuleContent := "groups:\n- name: example\n  rules:\n  - alert: ExampleAlert"
+	modifiedRuleContent := "groups:\n- name: example\n  rules:\n  - alert: ModifiedAlert"
+
+	originalScrapeConfigContent := "scrape_configs:\n- job_name: example"
+	modifiedScrapeConfigContent := "scrape_configs:\n- job_name: modified_example"
+
+	// Define YAML content referencing the rule and scrape config files.
+	yamlContent := `
+rule_files:
+  - rule_file.yml
+scrape_config_files:
+  - scrape_config.yml
+`
+
+	// Write initial content to files.
+	require.NoError(t, os.WriteFile(ruleFilePath, []byte(originalRuleContent), 0o644))
+	require.NoError(t, os.WriteFile(scrapeConfigFilePath, []byte(originalScrapeConfigContent), 0o644))
+	require.NoError(t, os.WriteFile(yamlFilePath, []byte(yamlContent), 0o644))
+
+	// Generate the original checksum.
+	originalChecksum := calculateChecksum(t, yamlFilePath)
+
+	t.Run("Rule File Change", func(t *testing.T) {
+		// Modify the rule file.
+		require.NoError(t, os.WriteFile(ruleFilePath, []byte(modifiedRuleContent), 0o644))
+
+		// Checksum should change.
+		modifiedChecksum := calculateChecksum(t, yamlFilePath)
+		require.NotEqual(t, originalChecksum, modifiedChecksum)
+
+		// Revert the rule file.
+		require.NoError(t, os.WriteFile(ruleFilePath, []byte(originalRuleContent), 0o644))
+
+		// Checksum should return to the original.
+		revertedChecksum := calculateChecksum(t, yamlFilePath)
+		require.Equal(t, originalChecksum, revertedChecksum)
+	})
+
+	t.Run("Scrape Config Change", func(t *testing.T) {
+		// Modify the scrape config file.
+		require.NoError(t, os.WriteFile(scrapeConfigFilePath, []byte(modifiedScrapeConfigContent), 0o644))
+
+		// Checksum should change.
+		modifiedChecksum := calculateChecksum(t, yamlFilePath)
+		require.NotEqual(t, originalChecksum, modifiedChecksum)
+
+		// Revert the scrape config file.
+		require.NoError(t, os.WriteFile(scrapeConfigFilePath, []byte(originalScrapeConfigContent), 0o644))
+
+		// Checksum should return to the original.
+		revertedChecksum := calculateChecksum(t, yamlFilePath)
+		require.Equal(t, originalChecksum, revertedChecksum)
+	})
+
+	t.Run("Rule File Deletion", func(t *testing.T) {
+		// Delete the rule file.
+		require.NoError(t, os.Remove(ruleFilePath))
+
+		// Checksum should change.
+		deletedChecksum := calculateChecksum(t, yamlFilePath)
+		require.NotEqual(t, originalChecksum, deletedChecksum)
+
+		// Restore the rule file.
+		require.NoError(t, os.WriteFile(ruleFilePath, []byte(originalRuleContent), 0o644))
+
+		// Checksum should return to the original.
+		revertedChecksum := calculateChecksum(t, yamlFilePath)
+		require.Equal(t, originalChecksum, revertedChecksum)
+	})
+
+	t.Run("Scrape Config Deletion", func(t *testing.T) {
+		// Delete the scrape config file.
+		require.NoError(t, os.Remove(scrapeConfigFilePath))
+
+		// Checksum should change.
+		deletedChecksum := calculateChecksum(t, yamlFilePath)
+		require.NotEqual(t, originalChecksum, deletedChecksum)
+
+		// Restore the scrape config file.
+		require.NoError(t, os.WriteFile(scrapeConfigFilePath, []byte(originalScrapeConfigContent), 0o644))
+
+		// Checksum should return to the original.
+		revertedChecksum := calculateChecksum(t, yamlFilePath)
+		require.Equal(t, originalChecksum, revertedChecksum)
+	})
+
+	t.Run("Main File Change", func(t *testing.T) {
+		// Modify the main YAML file.
+		modifiedYamlContent := `
+global:
+  scrape_interval: 3s
+rule_files:
+  - rule_file.yml
+scrape_config_files:
+  - scrape_config.yml
+`
+		require.NoError(t, os.WriteFile(yamlFilePath, []byte(modifiedYamlContent), 0o644))
+
+		// Checksum should change.
+		modifiedChecksum := calculateChecksum(t, yamlFilePath)
+		require.NotEqual(t, originalChecksum, modifiedChecksum)
+
+		// Revert the main YAML file.
+		require.NoError(t, os.WriteFile(yamlFilePath, []byte(yamlContent), 0o644))
+
+		// Checksum should return to the original.
+		revertedChecksum := calculateChecksum(t, yamlFilePath)
+		require.Equal(t, originalChecksum, revertedChecksum)
+	})
+
+	t.Run("Rule File Removed from YAML Config", func(t *testing.T) {
+		// Modify the YAML content to remove the rule file.
+		modifiedYamlContent := `
+scrape_config_files:
+  - scrape_config.yml
+`
+		require.NoError(t, os.WriteFile(yamlFilePath, []byte(modifiedYamlContent), 0o644))
+
+		// Checksum should change.
+		modifiedChecksum := calculateChecksum(t, yamlFilePath)
+		require.NotEqual(t, originalChecksum, modifiedChecksum)
+
+		// Revert the YAML content.
+		require.NoError(t, os.WriteFile(yamlFilePath, []byte(yamlContent), 0o644))
+
+		// Checksum should return to the original.
+		revertedChecksum := calculateChecksum(t, yamlFilePath)
+		require.Equal(t, originalChecksum, revertedChecksum)
+	})
+
+	t.Run("Scrape Config Removed from YAML Config", func(t *testing.T) {
+		// Modify the YAML content to remove the scrape config file.
+		modifiedYamlContent := `
+rule_files:
+  - rule_file.yml
+`
+		require.NoError(t, os.WriteFile(yamlFilePath, []byte(modifiedYamlContent), 0o644))
+
+		// Checksum should change.
+		modifiedChecksum := calculateChecksum(t, yamlFilePath)
+		require.NotEqual(t, originalChecksum, modifiedChecksum)
+
+		// Revert the YAML content.
+		require.NoError(t, os.WriteFile(yamlFilePath, []byte(yamlContent), 0o644))
+
+		// Checksum should return to the original.
+		revertedChecksum := calculateChecksum(t, yamlFilePath)
+		require.Equal(t, originalChecksum, revertedChecksum)
+	})
+
+	t.Run("Empty Rule File", func(t *testing.T) {
+		// Write an empty rule file.
+		require.NoError(t, os.WriteFile(ruleFilePath, []byte(""), 0o644))
+
+		// Checksum should change.
+		emptyChecksum := calculateChecksum(t, yamlFilePath)
+		require.NotEqual(t, originalChecksum, emptyChecksum)
+
+		// Restore the rule file.
+		require.NoError(t, os.WriteFile(ruleFilePath, []byte(originalRuleContent), 0o644))
+
+		// Checksum should return to the original.
+		revertedChecksum := calculateChecksum(t, yamlFilePath)
+		require.Equal(t, originalChecksum, revertedChecksum)
+	})
+
+	t.Run("Empty Scrape Config File", func(t *testing.T) {
+		// Write an empty scrape config file.
+		require.NoError(t, os.WriteFile(scrapeConfigFilePath, []byte(""), 0o644))
+
+		// Checksum should change.
+		emptyChecksum := calculateChecksum(t, yamlFilePath)
+		require.NotEqual(t, originalChecksum, emptyChecksum)
+
+		// Restore the scrape config file.
+		require.NoError(t, os.WriteFile(scrapeConfigFilePath, []byte(originalScrapeConfigContent), 0o644))
+
+		// Checksum should return to the original.
+		revertedChecksum := calculateChecksum(t, yamlFilePath)
+		require.Equal(t, originalChecksum, revertedChecksum)
+	})
+}
+
+// calculateChecksum generates a checksum for the given YAML file path.
+func calculateChecksum(t *testing.T, yamlFilePath string) string {
+	checksum, err := GenerateChecksum(yamlFilePath)
+	require.NoError(t, err)
+	require.NotEmpty(t, checksum)
+	return checksum
+}