Repository: prometheus/prometheus (language: go)

Issue:
Prometheus 3.0.0 OOMing when query logging is enabled
### What did you do?

Initially, I thought this was an issue on my end, because I mistakenly upgraded from Prometheus v2.54.1 to Prometheus 3.0.0 (skipping Prometheus 2.55.0, [which was called out in the migration guide](https://prometheus.io/docs/prometheus/3.0/migration/#tsdb-format-and-downgrade)).

Going off of that thought, I decided to do the following:
- Delete my existing TSDB (dev env, don't care)
- Uninstall Prometheus v3.0.0
- Install Prometheus v2.54.1
- Update to Prometheus v2.55.1
- Update to Prometheus v3.0.0

Unfortunately, the behavior persisted, so it can't have been the breaking TSDB change.

After all of that, I started running `pprof` against the heap, and found that `slog` was using a large amount of memory.

pprof for Prometheus v3.0.0:
```console
~# go tool pprof -top http://localhost:9090/prometheus/prometheus/debug/pprof/heap | head -n 20
Fetching profile over HTTP from http://localhost:9090/prometheus/prometheus/debug/pprof/heap
Saved profile in /root/pprof/pprof.prometheus.alloc_objects.alloc_space.inuse_objects.inuse_space.1063.pb.gz
File: prometheus
Type: inuse_space
Time: Nov 21, 2024 at 12:22am (UTC)
Showing nodes accounting for 25187.56MB, 97.01% of 25963.43MB total
Dropped 601 nodes (cum <= 129.82MB)
      flat  flat%   sum%        cum   cum%
13262.16MB 51.08% 51.08% 13262.16MB 51.08%  log/slog/internal/buffer.(*Buffer).WriteString (inline)
 8913.35MB 34.33% 85.41%  8913.35MB 34.33%  log/slog/internal/buffer.(*Buffer).Write (inline)
 1339.76MB  5.16% 90.57%  1339.76MB  5.16%  github.com/prometheus/prometheus/model/labels.(*Builder).Labels
  425.86MB  1.64% 92.21%  1716.61MB  6.61%  github.com/prometheus/prometheus/storage/remote.(*QueueManager).StoreSeries
  340.18MB  1.31% 93.52%   340.18MB  1.31%  github.com/prometheus/prometheus/tsdb/index.(*MemPostings).Delete.func1 (inlin
e)
  296.58MB  1.14% 94.66%   296.58MB  1.14%  github.com/klauspost/compress/flate.(*dictDecoder).init (inline)
  221.89MB  0.85% 95.52%   221.89MB  0.85%  github.com/prometheus/prometheus/tsdb.(*seriesHashmap).set
  172.58MB  0.66% 96.18%   459.49MB  1.77%  github.com/prometheus/prometheus/tsdb.(*stripeSeries).getOrSet
  169.53MB  0.65% 96.84%   509.70MB  1.96%  github.com/prometheus/prometheus/tsdb.(*Head).gc
   31.57MB  0.12% 96.96%   328.15MB  1.26%  github.com/klauspost/compress/flate.NewReaderOpts
    7.60MB 0.029% 96.99% 22321.83MB 85.97%  github.com/prometheus/prometheus/rules.(*Group).Eval.func1
    4.50MB 0.017% 97.00%   332.65MB  1.28%  github.com/klauspost/compress/gzip.NewReader (inline)
    1.50MB 0.0058% 97.01%   712.55MB  2.74%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).run
    0.50MB 0.0019% 97.01%  6056.14MB 23.33%  github.com/prometheus/prometheus/rules.(*AlertingRule).Eval
```

pprof for Prometheus v2.54.1
```console
~# go tool pprof -top http://localhost:9090/prometheus/prometheus/debug/pprof/heap | head -n 20
Fetching profile over HTTP from http://localhost:9090/prometheus/prometheus/debug/pprof/heap
Saved profile in /root/pprof/pprof.prometheus.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
File: prometheus
Type: inuse_space
Time: Nov 21, 2024 at 12:23am (UTC)
Showing nodes accounting for 9960.11MB, 95.07% of 10477.14MB total
Dropped 558 nodes (cum <= 52.39MB)
      flat  flat%   sum%        cum   cum%
 1800.33MB 17.18% 17.18%  1800.33MB 17.18%  github.com/prometheus/prometheus/model/labels.(*Builder).Labels
  869.15MB  8.30% 25.48%  1032.15MB  9.85%  github.com/prometheus/prometheus/tsdb.newMemSeries (inline)
  772.45MB  7.37% 32.85%   815.20MB  7.78%  github.com/prometheus/prometheus/tsdb.(*headAppender).Append
     768MB  7.33% 40.18%      768MB  7.33%  github.com/klauspost/compress/zstd.(*fastBase).ensureHist (inline)
  750.62MB  7.16% 47.35%   750.62MB  7.16%  github.com/prometheus/prometheus/model/labels.(*ScratchBuilder).Labels
  702.07MB  6.70% 54.05%   702.07MB  6.70%  github.com/prometheus/prometheus/tsdb/chunkenc.NewXORChunk (inline)
  695.47MB  6.64% 60.69%   695.47MB  6.64%  github.com/prometheus/prometheus/scrape.(*scrapeCache).addRef (inline)
  410.46MB  3.92% 64.60%   410.46MB  3.92%  github.com/prometheus/prometheus/scrape.(*scrapeCache).trackStaleness (inline)
  380.95MB  3.64% 68.24%  1371.14MB 13.09%  github.com/prometheus/prometheus/storage/remote.(*QueueManager).StoreSeries
  339.90MB  3.24% 71.48%   339.90MB  3.24%  github.com/klauspost/compress/flate.(*dictDecoder).init (inline)
  287.01MB  2.74% 74.22%   292.53MB  2.79%  github.com/prometheus/prometheus/tsdb.(*memSeries).mmapChunks
  265.73MB  2.54% 76.76%   265.73MB  2.54%  github.com/prometheus/prometheus/tsdb/index.(*MemPostings).Delete.func1 (inline)
  240.51MB  2.30% 79.05%  1181.59MB 11.28%  github.com/prometheus/prometheus/tsdb.(*memSeries).cutNewHeadChunk
  239.01MB  2.28% 81.34%   239.01MB  2.28%  github.com/prometheus/prometheus/tsdb/chunkenc.(*XORChunk).Appender
```

As you can see, `slog` is using a significant amount of memory comparatively. Both of these Prometheus instances are exactly the same, except for the version. Same scrape targets, same scrape jobs, same recording and alerting rules being evaluated.

### What did you expect to see?

_No response_

### What did you see instead? Under which circumstances?

After the WAL replayed, I had roughly 10 to 15 minutes before the prometheus process would be oom killed. Additionally, I noticed that `/var/lib/prometheus/prometheus_query.log` was growing at exponential rates (1GB per 2s - using `watch`). The highest I saw it was 1.1TB, which is obviously quite concerning. When testing earlier in the week when I skipped v2.55.1, I noticed the same behavior. 

### System information

Linux 5.10.0-33-amd64 x86_64

### Prometheus version

```text
prometheus, version 3.0.0 (branch: HEAD, revision: c5d009d57fcccb7247e1191a0b10d74b06295388)
  build user:       root@aa286d0eb00a
  build date:       20241114-16:40:43
  go version:       go1.23.3
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels
```


### Prometheus configuration file

```yaml
alerting:
  alertmanagers:
  - path_prefix: /prometheus/alertmanager
    scheme: https
    static_configs:
    - targets:
      - $REDACTED:9999
global:
  external_labels:
    datacenter: $REDACTED
    domain: $REDACTED
    replica: prometheus
  query_log_file: /var/lib/prometheus/prometheus_query.log
  scrape_interval: 30s
remote_write:
- queue_config:
    batch_send_deadline: 5s
    capacity: 5000
    max_samples_per_send: 1000
    max_shards: 2000
  tls_config:
    cert_file: /etc/prometheus/$REDACTED.crt
    key_file: /etc/prometheus/$REDACTED.key
  url: $REDACTED
  write_relabel_configs:
  - action: labeldrop
    regex: replica
rule_files:
- /etc/prometheus/rules/infra/common/*.yaml
- /etc/prometheus/rules/infra/datacenter/$REDACTED/*.yaml
scrape_config_files:
- /etc/prometheus/scrape_config_files/*.yaml
tracing:
  client_type: grpc
  endpoint: localhost:4317
  insecure: true
  sampling_fraction: 1.0
```


### Alertmanager version

_No response_

### Alertmanager configuration file

_No response_

### Logs

_No response_

Relevant CNCF Slack thread with @tjhop: https://cloud-native.slack.com/archives/C167KFM6C/p1732131830045239

So you don't have to sift through Slack, here's TJ's theory:
>ok, I think I see what's happening. we set the query logger here:
https://github.com/prometheus/prometheus/blob/main/cmd/prometheus/main.go#L894-L898
https://github.com/prometheus/prometheus/blob/main/promql/engine.go#L454-L474
we're passing in JSONFileLogger as the query logger which does satisfy the querylogger interface, but the important bit is in the implementation of .With, in that it modifies it's internal logger in place and keeps appending over and over
https://github.com/prometheus/prometheus/blob/main/promql/engine.go#L126-L135
https://github.com/prometheus/prometheus/blob/main/util/logging/file.go#L56-L58


Discussion / hints:
I know we like graphs here, so this is a graph of how quickly the `prometheus_query.log` is filling up.
![image](https://github.com/user-attachments/assets/89669f79-d9fa-488a-bfdf-a8419041b468)

My system is now completely out of disk space:
```
~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             48G     0   48G   0% /dev
tmpfs           9.5G  772K  9.5G   1% /run
/dev/sda        1.9T  1.8T     0 100% /
tmpfs            48G     0   48G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           9.5G     0  9.5G   0% /run/user/1100800327
~# du -sh /var/lib/prometheus/prometheus_query.log
1.7T	/var/lib/prometheus/prometheus_query.log
```

This isn't production for me, so no actual harm done. Just wanted to provide some info to demostrate.
Is it true that this only happens when query logging is enabled?

I haven't tested that, but I can if you'd like.
These are my current args:
```
--config.file /etc/prometheus/prometheus.yml --query.max-concurrency 150 --query.max-samples 500000000 --query.timeout 10m --storage.tsdb.max-block-duration 2h --storage.tsdb.path /var/lib/prometheus/ --storage.tsdb.retention.time 2d --storage.tsdb.wal-compression-type zstd --web.enable-admin-api   --web.external-url https://$REDACTED:9999/prometheus/prometheus/
```
> Is it true that this only happens when query logging is enabled?

Yes -- it's specifically related to the usage of the `.With()` method and how it's used within `promql.Engine.exec()`. Scrape failure loggers also use the json file logger, but they don't follow the same bad pattern mentioned (more details in slack [here](https://cloud-native.slack.com/archives/C01AUBA4PFE/p1732174008622169?thread_ts=1732169864.149909&cid=C01AUBA4PFE)). 

As far as I can tell, this occurs IFF query logging is enabled.
Thanks.  I changed the title because there are many issues sounding like "OOM after WAL replay", which are more of a general point about memory use in Prometheus.  This is something quite different.

Fix patch:
diff --git a/promql/engine.go b/promql/engine.go
index 13f8b069722..00a142b0c0d 100644
--- a/promql/engine.go
+++ b/promql/engine.go
@@ -47,6 +47,7 @@ import (
 	"github.com/prometheus/prometheus/storage"
 	"github.com/prometheus/prometheus/tsdb/chunkenc"
 	"github.com/prometheus/prometheus/util/annotations"
+	"github.com/prometheus/prometheus/util/logging"
 	"github.com/prometheus/prometheus/util/stats"
 	"github.com/prometheus/prometheus/util/zeropool"
 )
@@ -123,12 +124,13 @@ type QueryEngine interface {
 	NewRangeQuery(ctx context.Context, q storage.Queryable, opts QueryOpts, qs string, start, end time.Time, interval time.Duration) (Query, error)
 }
 
+var _ QueryLogger = (*logging.JSONFileLogger)(nil)
+
 // QueryLogger is an interface that can be used to log all the queries logged
 // by the engine.
 type QueryLogger interface {
-	Log(context.Context, slog.Level, string, ...any)
-	With(args ...any)
-	Close() error
+	slog.Handler
+	io.Closer
 }
 
 // A Query is derived from an a raw query string and can be run against an engine
@@ -626,6 +628,9 @@ func (ng *Engine) exec(ctx context.Context, q *query) (v parser.Value, ws annota
 	defer func() {
 		ng.queryLoggerLock.RLock()
 		if l := ng.queryLogger; l != nil {
+			logger := slog.New(l)
+			f := make([]slog.Attr, 0, 16) // Probably enough up front to not need to reallocate on append.
+
 			params := make(map[string]interface{}, 4)
 			params["query"] = q.q
 			if eq, ok := q.Statement().(*parser.EvalStmt); ok {
@@ -634,20 +639,20 @@ func (ng *Engine) exec(ctx context.Context, q *query) (v parser.Value, ws annota
 				// The step provided by the user is in seconds.
 				params["step"] = int64(eq.Interval / (time.Second / time.Nanosecond))
 			}
-			f := []interface{}{"params", params}
+			f = append(f, slog.Any("params", params))
 			if err != nil {
-				f = append(f, "error", err)
+				f = append(f, slog.Any("error", err))
 			}
-			f = append(f, "stats", stats.NewQueryStats(q.Stats()))
+			f = append(f, slog.Any("stats", stats.NewQueryStats(q.Stats())))
 			if span := trace.SpanFromContext(ctx); span != nil {
-				f = append(f, "spanID", span.SpanContext().SpanID())
+				f = append(f, slog.Any("spanID", span.SpanContext().SpanID()))
 			}
 			if origin := ctx.Value(QueryOrigin{}); origin != nil {
 				for k, v := range origin.(map[string]interface{}) {
-					f = append(f, k, v)
+					f = append(f, slog.Any(k, v))
 				}
 			}
-			l.Log(context.Background(), slog.LevelInfo, "promql query logged", f...)
+			logger.LogAttrs(context.Background(), slog.LevelInfo, "promql query logged", f...)
 			// TODO: @tjhop -- do we still need this metric/error log if logger doesn't return errors?
 			// ng.metrics.queryLogFailures.Inc()
 			// ng.logger.Error("can't log query", "err", err)
diff --git a/scrape/manager.go b/scrape/manager.go
index 04da3162e62..5ef5dccb99d 100644
--- a/scrape/manager.go
+++ b/scrape/manager.go
@@ -107,7 +107,7 @@ type Manager struct {
 	scrapeConfigs          map[string]*config.ScrapeConfig
 	scrapePools            map[string]*scrapePool
 	newScrapeFailureLogger func(string) (*logging.JSONFileLogger, error)
-	scrapeFailureLoggers   map[string]*logging.JSONFileLogger
+	scrapeFailureLoggers   map[string]FailureLogger
 	targetSets             map[string][]*targetgroup.Group
 	buffers                *pool.Pool
 
@@ -249,7 +249,7 @@ func (m *Manager) ApplyConfig(cfg *config.Config) error {
 	}
 
 	c := make(map[string]*config.ScrapeConfig)
-	scrapeFailureLoggers := map[string]*logging.JSONFileLogger{
+	scrapeFailureLoggers := map[string]FailureLogger{
 		"": nil, // Emptying the file name sets the scrape logger to nil.
 	}
 	for _, scfg := range scfgs {
@@ -257,7 +257,7 @@ func (m *Manager) ApplyConfig(cfg *config.Config) error {
 		if _, ok := scrapeFailureLoggers[scfg.ScrapeFailureLogFile]; !ok {
 			// We promise to reopen the file on each reload.
 			var (
-				logger *logging.JSONFileLogger
+				logger FailureLogger
 				err    error
 			)
 			if m.newScrapeFailureLogger != nil {
diff --git a/scrape/scrape.go b/scrape/scrape.go
index eeab208d658..f445e8d2dfe 100644
--- a/scrape/scrape.go
+++ b/scrape/scrape.go
@@ -61,6 +61,15 @@ var AlignScrapeTimestamps = true
 
 var errNameLabelMandatory = fmt.Errorf("missing metric name (%s label)", labels.MetricName)
 
+var _ FailureLogger = (*logging.JSONFileLogger)(nil)
+
+// FailureLogger is an interface that can be used to log all failed
+// scrapes.
+type FailureLogger interface {
+	slog.Handler
+	io.Closer
+}
+
 // scrapePool manages scrapes for sets of targets.
 type scrapePool struct {
 	appendable storage.Appendable
@@ -90,7 +99,7 @@ type scrapePool struct {
 
 	metrics *scrapeMetrics
 
-	scrapeFailureLogger    *logging.JSONFileLogger
+	scrapeFailureLogger    FailureLogger
 	scrapeFailureLoggerMtx sync.RWMutex
 }
 
@@ -223,11 +232,11 @@ func (sp *scrapePool) DroppedTargetsCount() int {
 	return sp.droppedTargetsCount
 }
 
-func (sp *scrapePool) SetScrapeFailureLogger(l *logging.JSONFileLogger) {
+func (sp *scrapePool) SetScrapeFailureLogger(l FailureLogger) {
 	sp.scrapeFailureLoggerMtx.Lock()
 	defer sp.scrapeFailureLoggerMtx.Unlock()
 	if l != nil {
-		l.With("job_name", sp.config.JobName)
+		l = slog.New(l).With("job_name", sp.config.JobName).Handler().(FailureLogger)
 	}
 	sp.scrapeFailureLogger = l
 
@@ -238,7 +247,7 @@ func (sp *scrapePool) SetScrapeFailureLogger(l *logging.JSONFileLogger) {
 	}
 }
 
-func (sp *scrapePool) getScrapeFailureLogger() *logging.JSONFileLogger {
+func (sp *scrapePool) getScrapeFailureLogger() FailureLogger {
 	sp.scrapeFailureLoggerMtx.RLock()
 	defer sp.scrapeFailureLoggerMtx.RUnlock()
 	return sp.scrapeFailureLogger
@@ -860,7 +869,7 @@ func (s *targetScraper) readResponse(ctx context.Context, resp *http.Response, w
 type loop interface {
 	run(errc chan<- error)
 	setForcedError(err error)
-	setScrapeFailureLogger(*logging.JSONFileLogger)
+	setScrapeFailureLogger(FailureLogger)
 	stop()
 	getCache() *scrapeCache
 	disableEndOfRunStalenessMarkers()
@@ -876,7 +885,7 @@ type cacheEntry struct {
 type scrapeLoop struct {
 	scraper                  scraper
 	l                        *slog.Logger
-	scrapeFailureLogger      *logging.JSONFileLogger
+	scrapeFailureLogger      FailureLogger
 	scrapeFailureLoggerMtx   sync.RWMutex
 	cache                    *scrapeCache
 	lastScrapeSize           int
@@ -1267,11 +1276,11 @@ func newScrapeLoop(ctx context.Context,
 	return sl
 }
 
-func (sl *scrapeLoop) setScrapeFailureLogger(l *logging.JSONFileLogger) {
+func (sl *scrapeLoop) setScrapeFailureLogger(l FailureLogger) {
 	sl.scrapeFailureLoggerMtx.Lock()
 	defer sl.scrapeFailureLoggerMtx.Unlock()
 	if ts, ok := sl.scraper.(fmt.Stringer); ok && l != nil {
-		l.With("target", ts.String())
+		l = slog.New(l).With("target", ts.String()).Handler().(FailureLogger)
 	}
 	sl.scrapeFailureLogger = l
 }
@@ -1421,7 +1430,7 @@ func (sl *scrapeLoop) scrapeAndReport(last, appendTime time.Time, errc chan<- er
 		sl.l.Debug("Scrape failed", "err", scrapeErr)
 		sl.scrapeFailureLoggerMtx.RLock()
 		if sl.scrapeFailureLogger != nil {
-			sl.scrapeFailureLogger.Log(context.Background(), slog.LevelError, scrapeErr.Error())
+			slog.New(sl.scrapeFailureLogger).Error(scrapeErr.Error())
 		}
 		sl.scrapeFailureLoggerMtx.RUnlock()
 		if errc != nil {
diff --git a/util/logging/file.go b/util/logging/file.go
index 9db7fb722be..27fdec27589 100644
--- a/util/logging/file.go
+++ b/util/logging/file.go
@@ -16,16 +16,22 @@ package logging
 import (
 	"context"
 	"fmt"
+	"io"
 	"log/slog"
 	"os"
 
 	"github.com/prometheus/common/promslog"
 )
 
-// JSONFileLogger represents a logger that writes JSON to a file. It implements the promql.QueryLogger interface.
+var _ slog.Handler = (*JSONFileLogger)(nil)
+
+var _ io.Closer = (*JSONFileLogger)(nil)
+
+// JSONFileLogger represents a logger that writes JSON to a file. It implements
+// the slog.Handler interface, as well as the io.Closer interface.
 type JSONFileLogger struct {
-	logger *slog.Logger
-	file   *os.File
+	handler slog.Handler
+	file    *os.File
 }
 
 // NewJSONFileLogger returns a new JSONFileLogger.
@@ -42,24 +48,47 @@ func NewJSONFileLogger(s string) (*JSONFileLogger, error) {
 	jsonFmt := &promslog.AllowedFormat{}
 	_ = jsonFmt.Set("json")
 	return &JSONFileLogger{
-		logger: promslog.New(&promslog.Config{Format: jsonFmt, Writer: f}),
-		file:   f,
+		handler: promslog.New(&promslog.Config{Format: jsonFmt, Writer: f}).Handler(),
+		file:    f,
 	}, nil
 }
 
-// Close closes the underlying file. It implements the promql.QueryLogger interface.
+// Close closes the underlying file. It implements the io.Closer interface.
 func (l *JSONFileLogger) Close() error {
 	return l.file.Close()
 }
 
-// With calls the `With()` method on the underlying `log/slog.Logger` with the
-// provided msg and args. It implements the promql.QueryLogger interface.
-func (l *JSONFileLogger) With(args ...any) {
-	l.logger = l.logger.With(args...)
+// Enabled returns true if and only if the internal slog.Handler is enabled. It
+// implements the slog.Handler interface.
+func (l *JSONFileLogger) Enabled(ctx context.Context, level slog.Level) bool {
+	return l.handler.Enabled(ctx, level)
+}
+
+// Handle takes record created by an slog.Logger and forwards it to the
+// internal slog.Handler for dispatching the log call to the backing file. It
+// implements the slog.Handler interface.
+func (l *JSONFileLogger) Handle(ctx context.Context, r slog.Record) error {
+	return l.handler.Handle(ctx, r.Clone())
+}
+
+// WithAttrs returns a new *JSONFileLogger with a new internal handler that has
+// the provided attrs attached as attributes on all further log calls. It
+// implements the slog.Handler interface.
+func (l *JSONFileLogger) WithAttrs(attrs []slog.Attr) slog.Handler {
+	if len(attrs) == 0 {
+		return l
+	}
+
+	return &JSONFileLogger{file: l.file, handler: l.handler.WithAttrs(attrs)}
 }
 
-// Log calls the `Log()` method on the underlying `log/slog.Logger` with the
-// provided msg and args. It implements the promql.QueryLogger interface.
-func (l *JSONFileLogger) Log(ctx context.Context, level slog.Level, msg string, args ...any) {
-	l.logger.Log(ctx, level, msg, args...)
+// WithGroup returns a new *JSONFileLogger with a new internal handler that has
+// the provided group name attached, to group all other attributes added to the
+// logger. It implements the slog.Handler interface.
+func (l *JSONFileLogger) WithGroup(name string) slog.Handler {
+	if name == "" {
+		return l
+	}
+
+	return &JSONFileLogger{file: l.file, handler: l.handler.WithGroup(name)}
 }


Test patch:
diff --git a/promql/engine_test.go b/promql/engine_test.go
index 8b8dc3909c8..f0a76ecc4d9 100644
--- a/promql/engine_test.go
+++ b/promql/engine_test.go
@@ -17,8 +17,9 @@ import (
 	"context"
 	"errors"
 	"fmt"
-	"log/slog"
 	"math"
+	"os"
+	"path/filepath"
 	"sort"
 	"strings"
 	"sync"
@@ -38,6 +39,7 @@ import (
 	"github.com/prometheus/prometheus/storage"
 	"github.com/prometheus/prometheus/tsdb/chunkenc"
 	"github.com/prometheus/prometheus/util/annotations"
+	"github.com/prometheus/prometheus/util/logging"
 	"github.com/prometheus/prometheus/util/stats"
 	"github.com/prometheus/prometheus/util/testutil"
 )
@@ -2147,40 +2149,17 @@ func TestSubquerySelector(t *testing.T) {
 	}
 }
 
-type FakeQueryLogger struct {
-	closed bool
-	logs   []interface{}
-	attrs  []any
-}
+func getLogLines(t *testing.T, name string) []string {
+	content, err := os.ReadFile(name)
+	require.NoError(t, err)
 
-func NewFakeQueryLogger() *FakeQueryLogger {
-	return &FakeQueryLogger{
-		closed: false,
-		logs:   make([]interface{}, 0),
-		attrs:  make([]any, 0),
+	lines := strings.Split(string(content), "\n")
+	for i := len(lines) - 1; i >= 0; i-- {
+		if lines[i] == "" {
+			lines = append(lines[:i], lines[i+1:]...)
+		}
 	}
-}
-
-// It implements the promql.QueryLogger interface.
-func (f *FakeQueryLogger) Close() error {
-	f.closed = true
-	return nil
-}
-
-// It implements the promql.QueryLogger interface.
-func (f *FakeQueryLogger) Log(ctx context.Context, level slog.Level, msg string, args ...any) {
-	// Test usage only really cares about existence of keyvals passed in
-	// via args, just append in the log message before handling the
-	// provided args and any embedded kvs added via `.With()` on f.attrs.
-	log := append([]any{msg}, args...)
-	log = append(log, f.attrs...)
-	f.attrs = f.attrs[:0]
-	f.logs = append(f.logs, log...)
-}
-
-// It implements the promql.QueryLogger interface.
-func (f *FakeQueryLogger) With(args ...any) {
-	f.attrs = append(f.attrs, args...)
+	return lines
 }
 
 func TestQueryLogger_basic(t *testing.T) {
@@ -2205,32 +2184,45 @@ func TestQueryLogger_basic(t *testing.T) {
 	// promql.Query works without query log initialized.
 	queryExec()
 
-	f1 := NewFakeQueryLogger()
+	tmpDir := t.TempDir()
+	ql1File := filepath.Join(tmpDir, "query1.log")
+	f1, err := logging.NewJSONFileLogger(ql1File)
+	require.NoError(t, err)
+
 	engine.SetQueryLogger(f1)
 	queryExec()
-	require.Contains(t, f1.logs, `params`)
-	require.Contains(t, f1.logs, map[string]interface{}{"query": "test statement"})
+	logLines := getLogLines(t, ql1File)
+	require.Contains(t, logLines[0], "params", map[string]interface{}{"query": "test statement"})
+	require.Len(t, logLines, 1)
 
-	l := len(f1.logs)
+	l := len(logLines)
 	queryExec()
-	require.Len(t, f1.logs, 2*l)
+	logLines = getLogLines(t, ql1File)
+	l2 := len(logLines)
+	require.Equal(t, l2, 2*l)
 
-	// Test that we close the query logger when unsetting it.
-	require.False(t, f1.closed, "expected f1 to be open, got closed")
+	// Test that we close the query logger when unsetting it. The following
+	// attempt to close the file should error.
 	engine.SetQueryLogger(nil)
-	require.True(t, f1.closed, "expected f1 to be closed, got open")
+	err = f1.Close()
+	require.ErrorContains(t, err, "file already closed", "expected f1 to be closed, got open")
 	queryExec()
 
 	// Test that we close the query logger when swapping.
-	f2 := NewFakeQueryLogger()
-	f3 := NewFakeQueryLogger()
+	ql2File := filepath.Join(tmpDir, "query2.log")
+	f2, err := logging.NewJSONFileLogger(ql2File)
+	require.NoError(t, err)
+	ql3File := filepath.Join(tmpDir, "query3.log")
+	f3, err := logging.NewJSONFileLogger(ql3File)
+	require.NoError(t, err)
 	engine.SetQueryLogger(f2)
-	require.False(t, f2.closed, "expected f2 to be open, got closed")
 	queryExec()
 	engine.SetQueryLogger(f3)
-	require.True(t, f2.closed, "expected f2 to be closed, got open")
-	require.False(t, f3.closed, "expected f3 to be open, got closed")
+	err = f2.Close()
+	require.ErrorContains(t, err, "file already closed", "expected f2 to be closed, got open")
 	queryExec()
+	err = f3.Close()
+	require.NoError(t, err)
 }
 
 func TestQueryLogger_fields(t *testing.T) {
@@ -2242,7 +2234,14 @@ func TestQueryLogger_fields(t *testing.T) {
 	}
 	engine := promqltest.NewTestEngineWithOpts(t, opts)
 
-	f1 := NewFakeQueryLogger()
+	tmpDir := t.TempDir()
+	ql1File := filepath.Join(tmpDir, "query1.log")
+	f1, err := logging.NewJSONFileLogger(ql1File)
+	require.NoError(t, err)
+	t.Cleanup(func() {
+		require.NoError(t, f1.Close())
+	})
+
 	engine.SetQueryLogger(f1)
 
 	ctx, cancelCtx := context.WithCancel(context.Background())
@@ -2255,8 +2254,8 @@ func TestQueryLogger_fields(t *testing.T) {
 	res := query.Exec(ctx)
 	require.NoError(t, res.Err)
 
-	require.Contains(t, f1.logs, `foo`)
-	require.Contains(t, f1.logs, `bar`)
+	logLines := getLogLines(t, ql1File)
+	require.Contains(t, logLines[0], "foo", "bar")
 }
 
 func TestQueryLogger_error(t *testing.T) {
@@ -2268,7 +2267,14 @@ func TestQueryLogger_error(t *testing.T) {
 	}
 	engine := promqltest.NewTestEngineWithOpts(t, opts)
 
-	f1 := NewFakeQueryLogger()
+	tmpDir := t.TempDir()
+	ql1File := filepath.Join(tmpDir, "query1.log")
+	f1, err := logging.NewJSONFileLogger(ql1File)
+	require.NoError(t, err)
+	t.Cleanup(func() {
+		require.NoError(t, f1.Close())
+	})
+
 	engine.SetQueryLogger(f1)
 
 	ctx, cancelCtx := context.WithCancel(context.Background())
@@ -2282,10 +2288,9 @@ func TestQueryLogger_error(t *testing.T) {
 	res := query.Exec(ctx)
 	require.Error(t, res.Err, "query should have failed")
 
-	require.Contains(t, f1.logs, `params`)
-	require.Contains(t, f1.logs, map[string]interface{}{"query": "test statement"})
-	require.Contains(t, f1.logs, `error`)
-	require.Contains(t, f1.logs, testErr)
+	logLines := getLogLines(t, ql1File)
+	require.Contains(t, logLines[0], "error", testErr)
+	require.Contains(t, logLines[0], "params", map[string]interface{}{"query": "test statement"})
 }
 
 func TestPreprocessAndWrapWithStepInvariantExpr(t *testing.T) {
diff --git a/scrape/scrape_test.go b/scrape/scrape_test.go
index ce1686fecab..571b27086cd 100644
--- a/scrape/scrape_test.go
+++ b/scrape/scrape_test.go
@@ -55,7 +55,6 @@ import (
 	"github.com/prometheus/prometheus/model/value"
 	"github.com/prometheus/prometheus/storage"
 	"github.com/prometheus/prometheus/tsdb/chunkenc"
-	"github.com/prometheus/prometheus/util/logging"
 	"github.com/prometheus/prometheus/util/pool"
 	"github.com/prometheus/prometheus/util/teststorage"
 	"github.com/prometheus/prometheus/util/testutil"
@@ -252,7 +251,7 @@ type testLoop struct {
 	timeout      time.Duration
 }
 
-func (l *testLoop) setScrapeFailureLogger(*logging.JSONFileLogger) {
+func (l *testLoop) setScrapeFailureLogger(FailureLogger) {
 }
 
 func (l *testLoop) run(errc chan<- error) {
diff --git a/util/logging/file_test.go b/util/logging/file_test.go
index c9f7240feeb..bd34bc2a3a2 100644
--- a/util/logging/file_test.go
+++ b/util/logging/file_test.go
@@ -14,7 +14,6 @@
 package logging
 
 import (
-	"context"
 	"log/slog"
 	"os"
 	"strings"
@@ -24,6 +23,19 @@ import (
 	"github.com/stretchr/testify/require"
 )
 
+func getLogLines(t *testing.T, name string) []string {
+	content, err := os.ReadFile(name)
+	require.NoError(t, err)
+
+	lines := strings.Split(string(content), "\n")
+	for i := len(lines) - 1; i >= 0; i-- {
+		if lines[i] == "" {
+			lines = append(lines[:i], lines[i+1:]...)
+		}
+	}
+	return lines
+}
+
 func TestJSONFileLogger_basic(t *testing.T) {
 	f, err := os.CreateTemp("", "logging")
 	require.NoError(t, err)
@@ -32,24 +44,23 @@ func TestJSONFileLogger_basic(t *testing.T) {
 		require.NoError(t, os.Remove(f.Name()))
 	}()
 
-	l, err := NewJSONFileLogger(f.Name())
+	logHandler, err := NewJSONFileLogger(f.Name())
 	require.NoError(t, err)
-	require.NotNil(t, l, "logger can't be nil")
+	require.NotNil(t, logHandler, "logger handler can't be nil")
 
-	l.Log(context.Background(), slog.LevelInfo, "test", "hello", "world")
-	require.NoError(t, err)
-	r := make([]byte, 1024)
-	_, err = f.Read(r)
-	require.NoError(t, err)
+	logger := slog.New(logHandler)
+	logger.Info("test", "hello", "world")
 
-	result, err := regexp.Match(`^{"time":"[^"]+","level":"INFO","source":"file.go:\d+","msg":"test","hello":"world"}\n`, r)
+	r := getLogLines(t, f.Name())
+	require.Len(t, r, 1, "expected 1 log line")
+	result, err := regexp.Match(`^{"time":"[^"]+","level":"INFO","source":"\w+.go:\d+","msg":"test","hello":"world"}`, []byte(r[0]))
 	require.NoError(t, err)
 	require.True(t, result, "unexpected content: %s", r)
 
-	err = l.Close()
+	err = logHandler.Close()
 	require.NoError(t, err)
 
-	err = l.file.Close()
+	err = logHandler.file.Close()
 	require.Error(t, err)
 	require.True(t, strings.HasSuffix(err.Error(), os.ErrClosed.Error()), "file not closed")
 }
@@ -62,31 +73,31 @@ func TestJSONFileLogger_parallel(t *testing.T) {
 		require.NoError(t, os.Remove(f.Name()))
 	}()
 
-	l, err := NewJSONFileLogger(f.Name())
+	logHandler, err := NewJSONFileLogger(f.Name())
 	require.NoError(t, err)
-	require.NotNil(t, l, "logger can't be nil")
+	require.NotNil(t, logHandler, "logger handler can't be nil")
 
-	l.Log(context.Background(), slog.LevelInfo, "test", "hello", "world")
-	require.NoError(t, err)
+	logger := slog.New(logHandler)
+	logger.Info("test", "hello", "world")
 
-	l2, err := NewJSONFileLogger(f.Name())
+	logHandler2, err := NewJSONFileLogger(f.Name())
 	require.NoError(t, err)
-	require.NotNil(t, l, "logger can't be nil")
+	require.NotNil(t, logHandler2, "logger handler can't be nil")
 
-	l2.Log(context.Background(), slog.LevelInfo, "test", "hello", "world")
-	require.NoError(t, err)
+	logger2 := slog.New(logHandler2)
+	logger2.Info("test", "hello", "world")
 
-	err = l.Close()
+	err = logHandler.Close()
 	require.NoError(t, err)
 
-	err = l.file.Close()
+	err = logHandler.file.Close()
 	require.Error(t, err)
 	require.True(t, strings.HasSuffix(err.Error(), os.ErrClosed.Error()), "file not closed")
 
-	err = l2.Close()
+	err = logHandler2.Close()
 	require.NoError(t, err)
 
-	err = l2.file.Close()
+	err = logHandler2.file.Close()
 	require.Error(t, err)
 	require.True(t, strings.HasSuffix(err.Error(), os.ErrClosed.Error()), "file not closed")
 }