Repository: prometheus/prometheus (language: go)

Issue:
Memory Leak in v2.27.1: tsdb.txRing consumes much heap memory
<!--

    Please do *NOT* ask support questions in Github issues.

    If your issue is not a feature request or bug report use our
    community support.

    https://prometheus.io/community/

    There is also commercial support available.

    https://prometheus.io/support-training/

-->

**What did you do?**
We ran into the same problem as: https://github.com/prometheus/prometheus/issues/7120

**What did you expect to see?**
The memory usage is as stable as other Prometheus instances

**What did you see instead? Under which circumstances?**
The available memory of the node which hosts the Prometheus 2.27.1 instance gets decreasing like below:
![image](https://user-images.githubusercontent.com/15689619/130766138-438e9774-cdba-46fe-aa8a-a9e172484cf9.png)
![image](https://user-images.githubusercontent.com/15689619/130766139-dd1642aa-5d05-4d87-a648-ee7e9125c4ec.png)
![image](https://user-images.githubusercontent.com/15689619/130766893-eb106313-faf6-4e6e-ae4c-bceea4aabd02.png)
![image](https://user-images.githubusercontent.com/15689619/130767965-ba32dfe8-a03a-492a-b8a2-115a0373c88a.png)


**Environment**

* System information:

	Linux 4.9.65-netease x86_64

* Prometheus version:
        {
        "status": "success",
        "data": {
        "version": "2.27.1",
        "revision": "db7f0bcec27bd8aeebad6b08ac849516efa9ae02",
        "branch": "HEAD",
        "buildUser": "root@fd804fbd4f25",
        "buildDate": "20210518-14:17:54",
        "goVersion": "go1.16.4"
        }
        }

* Alertmanager version:

	insert output of `alertmanager --version` here (if relevant to the issue)

* Prometheus configuration file:
```
{
  "status": "success",
  "data": {
    "alertmanager.notification-queue-capacity": "10000",
    "alertmanager.timeout": "",
    "config.file": "/etc/prometheus/prometheus.yml",
    "enable-feature": "",
    "log.format": "logfmt",
    "log.level": "info",
    "query.lookback-delta": "5m",
    "query.max-concurrency": "20",
    "query.max-samples": "50000000",
    "query.timeout": "2m",
    "rules.alert.for-grace-period": "10m",
    "rules.alert.for-outage-tolerance": "1h",
    "rules.alert.resend-delay": "1m",
    "scrape.adjust-timestamps": "true",
    "storage.exemplars.exemplars-limit": "0",
    "storage.remote.flush-deadline": "1m",
    "storage.remote.read-concurrent-limit": "10",
    "storage.remote.read-max-bytes-in-frame": "1048576",
    "storage.remote.read-sample-limit": "50000000",
    "storage.tsdb.allow-overlapping-blocks": "false",
    "storage.tsdb.max-block-chunk-segment-size": "0B",
    "storage.tsdb.max-block-duration": "2h",
    "storage.tsdb.min-block-duration": "2h",
    "storage.tsdb.no-lockfile": "false",
    "storage.tsdb.path": "/prometheus",
    "storage.tsdb.retention": "0s",
    "storage.tsdb.retention.size": "256GiB",
    "storage.tsdb.retention.time": "1w",
    "storage.tsdb.wal-compression": "true",
    "storage.tsdb.wal-segment-size": "0B",
    "web.config.file": "",
    "web.console.libraries": "/usr/share/prometheus/console_libraries",
    "web.console.templates": "/usr/share/prometheus/consoles",
    "web.cors.origin": ".*",
    "web.enable-admin-api": "false",
    "web.enable-lifecycle": "false",
    "web.external-url": "",
    "web.listen-address": ":9090",
    "web.max-connections": "512",
    "web.page-title": "Prometheus Time Series Collection and Processing Server",
    "web.read-timeout": "5m",
    "web.route-prefix": "/",
    "web.user-assets": ""
  }
}
```

* Alertmanager configuration file:
```
insert configuration here (if relevant to the issue)
```


* Logs:
```
insert Prometheus and Alertmanager logs relevant to the issue here
```



Discussion / hints:
@ilixiaocui could you graph the low watermark and see if it is changing at all or if it got stuck for good? If the latter, is there anything that happened at that moment with your Prometheus? (Like a long running or failed query or anything interesting logged.)

If you restart your Prometheus, does the problem promptly recur, or does it take a while?

_Originally posted by @beorn7 in https://github.com/prometheus/prometheus/issues/9252#issuecomment-905344344_
> @ilixiaocui could you graph the low watermark and see if it is changing at all or if it got stuck for good? If the latter, is there anything that happened at that moment with your Prometheus? (Like a long running or failed query or anything interesting logged.)
> 
> If you restart your Prometheus, does the problem promptly recur, or does it take a while?
> 
> _Originally posted by @beorn7 in [#9252 (comment)](https://github.com/prometheus/prometheus/issues/9252#issuecomment-905344344)_

1. It is changing.
2. In order to keep the scene，I have not rebooting. No query operation was performed when the memory suddenly increased 

![image](https://user-images.githubusercontent.com/15689619/130770737-8c135ab8-c632-4ff5-ae1d-c41127351b67.png)

3. log is here 
[promethues.log](https://github.com/prometheus/prometheus/files/7046613/promethues.log)

4. tsdb status
[tsdb-staus.txt](https://github.com/prometheus/prometheus/files/7046718/tsdb-staus.txt)

Weird, so your watermark is still progressing most of the time, and the delta between high and low watermark, while on the larger side, isn't ginormous. Still you see many GiB in txRing…

Do you have any particular high load? Like frequent scrapes and/or scrapes of very large size?

In any case, we need to reproduce this to further debug.
> Weird, so your watermark is still progressing most of the time, and the delta between high and low watermark, while on the larger side, isn't ginormous. Still you see many GiB in txRing…
> 
> Do you have any particular high load? Like frequent scrapes and/or scrapes of very large size?
> 
> In any case, we need to reproduce this to further debug.

particular high load？Do you mean frenquent query large time span or need collecting many metrics?
Both. Or anything that is unusually expensive, long-running, or large.

Frequent large scrapes could lead to many entries in txRing, especially if combined with frequent long-running queries.

However, if that memory profile is correct, txRing is absurdly large. Which seems something is wrong (but the watermark isn't stuck, which makes this really weird). Ideally, we can just create the same conditions systematically and see this problem again. Then others have a chance to reproduce and debug.
> However, if that memory profile is correct, txRing is absurdly large. Which seems something is wrong (but the watermark isn't stuck, which makes this really weird). Ideally, we can just create the same conditions systematically and see this problem again. Then others have a chance to reproduce and debug.

We do not do any large query and no query fail during the time. There is no query operation at the moment  the memory suddenly grows. There is no error in Prometheus.log.  Is there a way to check growth of the metrics collected? We used the 2.12 version before and there was no such problem. 
@ilixiaocui could you post the pprof file please?  There may be some details in there that help to understand.

> @ilixiaocui could you post the pprof file please? There may be some details in there that help to understand.

here is profile(I add .txt to the file, otherwise it cannot be uploaded):
[profile.txt](https://github.com/prometheus/prometheus/files/7074557/profile.txt)

That looks like a CPU profile; please upload the heap profile you were showing extracts from in this issue.
> That looks like a CPU profile; please upload the heap profile you were showing extracts from in this issue.

[heap1.txt](https://github.com/prometheus/prometheus/files/7087905/heap1.txt)
[heap2.txt](https://github.com/prometheus/prometheus/files/7087906/heap2.txt)

Thanks.  

`heap1.txt` shows 73.15GB used by `txRing.add()` in 2,396,902 objects; that is exactly 32,768 bytes each.
Since each element is a `uint64` at 8 bytes, that gives 4,096 entries in the ring.

Other object counts are consistent with there being ~2.4 million series active.

I can't see the bug, though.
I notice from your profile that it seems to be calling `scrapeLoop.append()` way more than the Prometheus where I work.
What scrape frequency are you using?

> What scrape frequency are you using?

we use promethues to monitor a block storage system. We statistic every IO.

we use [bvar](https://github.com/apache/incubator-brpc/blob/master/docs/en/bvar.md) to do statistic.

here is our metric. [client_metric.h](https://github.com/opencurve/curve/blob/master/src/client/client_metric.h
)     [chunkserver_metric.h](https://github.com/opencurve/curve/blob/master/src/chunkserver/chunkserver_metrics.h
)

Eevry io we will count:

```
void IOMetric::OnResponse(size_t size, int64_t latUs, bool hasError) {

if (!hasError) {

        ioNum_ << 1;

        sizeRecorder_ << size;

        ioBytes_ << size;

        latencyRecorder_ << latUs;

    } else {

        errorNum_ << 1;

    }
}

```

Perhaps I confused things: what I wanted is defined by `scrape_interval` in your [config file](https://prometheus.io/docs/prometheus/latest/configuration/configuration). 
> Perhaps I confused things: what I wanted is defined by `scrape_interval` in your [config file](https://prometheus.io/docs/prometheus/latest/configuration/configuration).

```
# my global config
global:
  scrape_interval:     3s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'curve_metrics'

    file_sd_configs:
    - files: ['*.json']
```


Thanks for that.  2.4 million series scraped every 3 seconds gives 1.25 microseconds per series; I wonder if Prometheus just can't keep up at times, so the append transactions pile up.

Could you plot `prometheus_tsdb_head_active_appenders` over time, especially in the periods when you see rapid growth in memory?

same problem in our promehteus instance，when prometheus instance runing with args “--storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h” 。through test we find when    min-block-duration/max-block-duration  values within “60m”。prometheus instance   will runing well。
@marquis-wang if you have strong evidence that it's the same, please post that evidence, and all the additional detail requested in this issue.

Otherwise, please open a different issue and supply full detail there.
@marquis-wang what @bboreham said. It would be extremely surprising if the block duration settings change anything in terms of the tsdb.txRing size. (But if it actually did, it would be some kind of smoking gun that something is extremely wrong.)
@ilixiaocui as @bboreham pointed out, your ingest load seems to be around 1 million samples per second. That's a bit on the high side but should be no problem in general.

Just to clarify: Are you seeing these RAM increases regularly or was it a one-off? Can you easily reproduce the effect after restarting the Prometheus server? How long does it take for it to show up? Is that time predictable or does it strike at random moments? Does the Prometheus server recover eventually or does it keep the large tsdb.txRing forever?
Note there is no code to shrink `tsdb.txRing`; once it goes up for a series it will stay that size.
Right. I guess the thought was if it goes up that far once, we'll need it again in that size, so there is no point in shrinking and then regrowing it…
Yes, that was exactly the thought. Once you have one query that runs for 2m, you're bound to have more in future.
Given a txring of size 4096, does that suggest >2048 scrapes passed during one query?  Which would be over 100 minutes at 3s per scrape. 
@bboreham our prometheus instance has more than 2000 node_exporter , storage.tsdb.retention.time is set by "60d"。query requests is less than 50 times in one minute 。when prometheus starting ，the memory of RES is about 20GB。then the memory of RES Growing faster and faster。using pprof tools find total memory of prometheus instance is about 64GB，memory of "tsdb (*txRing) add " is about 37860MB(60%)。when the memory is growing we find the checkpoint dir in WAL not created or updated ，files in WAL dir is more and more。when we first find this guess it May be related to the environment and WAL write。In order to reproduce as soon as possible， we change min-block-duration/max-block-duration to “10m” then everything is ok。when change min-block-duration/max-block-duration to “2h”，the memory is Leak。now，min-block-duration/max-block-duration is set by “60m”。it running well about 14 days。 
![image](https://user-images.githubusercontent.com/22267584/134467826-d47c7741-84eb-447d-bec4-5305b2930e3c.png)
the format of heap_0903.txt is “svg” svg is not allow to upload  
[heap_0903.txt](https://github.com/prometheus/prometheus/files/7215615/heap_0903.txt)

Personally I find those graphs very difficult to read; I can work with the pprof file.
There is a service at https://share.polarsignals.com/ which allows upload and viewing, however you can also upload the file renamed here.

@marquis-wang please answer same questions as for the other instance:
* could you graph the low watermark
* Could you plot prometheus_tsdb_head_active_appenders over time, especially in the periods when you see rapid growth in memory?
* What is your scrape interval?
* Are you seeing these RAM increases regularly or was it a one-off? 
* Can you easily reproduce the effect after restarting the Prometheus server? 
* How long does it take for it to show up? 
* Is that time predictable or does it strike at random moments? 

Today we have somewhat similar symptoms in one of Grafana Labs' production clusters. 
Scrape interval is 30s; process is OOMing around 200GB, with 18GB of `txRing` shown in heap profiles.
~20 million series.
`prometheus_tsdb_isolation_low_watermark` is flat, gets to 4 or 6 then never increases.
`prometheus_tsdb_isolation_high_watermark` goes up linearly.

Right now it is taking about 1 hour to load the WAL then 1 hour to crash, rinse and repeat.
Is it prometheus or agent or cortex?
Prometheus, v2.30.1.
That is interesting. I guess a goroutime dump would be useful to see if a querier is blocked somewhere.
You prefer `debug=1` or `debug=2` ?  The latter is 17MB.
Only 1MB compressed:
[prometheus-goroutines.gz](https://github.com/prometheus/prometheus/files/7513028/prometheus-goroutines.gz)

> Today we have somewhat similar symptoms in one of Grafana Labs' production clusters. Scrape interval is 30s; process is OOMing around 200GB, with 18GB of `txRing` shown in heap profiles. ~20 million series. `prometheus_tsdb_isolation_low_watermark` is flat, gets to 4 or 6 then never increases. `prometheus_tsdb_isolation_high_watermark` goes up linearly.
> 
> Right now it is taking about 1 hour to load the WAL then 1 hour to crash, rinse and repeat.

same on `v2.32.1` but with both - `prometheus_tsdb_isolation_high_watermark` and `prometheus_tsdb_isolation_low_watermark` in my case

`scrape_interval` was forced to 30s for all jobs; it consumes ~54GB before OOM; ~8,8M series

![Screen Shot 2022-01-21 at 01 08 44](https://user-images.githubusercontent.com/28726429/150430143-c6f95d6e-0c4a-4e99-96d3-727b6214007e.png)
> same on v2.32.1 but with both - prometheus_tsdb_isolation_high_watermark and prometheus_tsdb_isolation_low_watermark in my case

Can you clarify what you think is the same?  What links your symptoms to those discussed in this issue?
54GB for 8.8M series is relatively much less than anyone else has reported here.
We had two Prometheus OOM at 400GB of RAM yesterday, so I spent about five hours studying metrics and profiles, and I think I understand what is happening.

First, head compaction holds an `isolationState`, acquired here:
https://github.com/prometheus/prometheus/blob/ec6c1f17d1be530d77482a9d9becfe0012239319/tsdb/head.go#L1058

called from `LeveledCompactor).populateBlock`; example stack trace: 
```
#	0x231eb17	github.com/prometheus/prometheus/tsdb/chunks.(*Writer).WriteChunks+0x137		/app/tsdb/chunks/chunks.go:399
#	0x238eb6e	github.com/prometheus/prometheus/tsdb.(*instrumentedChunkWriter).WriteChunks+0x4e	/app/tsdb/compact.go:545
#	0x23915aa	github.com/prometheus/prometheus/tsdb.(*LeveledCompactor).populateBlock+0x120a		/app/tsdb/compact.go:784
#	0x238f2e8	github.com/prometheus/prometheus/tsdb.(*LeveledCompactor).write+0x648			/app/tsdb/compact.go:597
#	0x238e584	github.com/prometheus/prometheus/tsdb.(*LeveledCompactor).Write+0x224			/app/tsdb/compact.go:504
#	0x23969cc	github.com/prometheus/prometheus/tsdb.(*DB).compactHead+0x6c				/app/tsdb/db.go:970
#	0x2395e92	github.com/prometheus/prometheus/tsdb.(*DB).Compact+0x2b2				/app/tsdb/db.go:928
#	0x239564e	github.com/prometheus/prometheus/tsdb.(*DB).run+0x3ee					/app/tsdb/db.go:828
```

This `isolationState` holds the 'low watermark' flat until compaction is finished.
Chart showing `prometheus_tsdb_isolation_high_watermark - prometheus_tsdb_isolation_low_watermark` for the two Prometheus:

<img width="699" alt="image" src="https://user-images.githubusercontent.com/8125524/189862716-4b375f19-3d0a-46b8-b831-1af3e3ee36e2.png">

Each scrape creates an `Appender`. Any appenders created while this `isolationState` exists, i.e. while compaction is running, are added to `txRing` of series they touch.  So the memory for `txRing` grows bigger and bigger.

Head compaction can take multiple hours. I found it quite difficult to pin down when exactly it is happening, but an increase in `prometheus_tsdb_compaction_duration_seconds_sum` followed by an increase in `prometheus_tsdb_head_gc_duration_seconds_sum` marks the end:

<img width="698" alt="image" src="https://user-images.githubusercontent.com/8125524/189863599-d3daac2f-bf10-4069-bd98-fb2b72c65e06.png">

Chunks older than the most recent one have been written out to memory-mapped files, and must be paged back in. As the rest of Prometheus gets bigger, available RAM to page in memory-mapped files gets tighter so the page must come from disk and compaction slows down dramatically. And compaction takes place on a single goroutine.
Here is a chart of hard page-faults per second (`rate(container_memory_failures_total{failure_type="pgmajfault"}[5m])`:

<img width="700" alt="image" src="https://user-images.githubusercontent.com/8125524/189863871-4c17659b-51d5-4dd0-8014-059c4c144e14.png">

When Prometheus OOMs before completing the head compaction, it will restart and re-read everything from WAL, then start the head compaction again from the same point as before. Now the data it wants to read is even older and likely to need paging in from disk, so most likely this compaction will take even longer.

Awesome investigation. Now we only need to find a fix. :-o

Fix patch:
diff --git a/tsdb/db.go b/tsdb/db.go
index 00c1bceedf8..cf917ebbb63 100644
--- a/tsdb/db.go
+++ b/tsdb/db.go
@@ -925,7 +925,15 @@ func (db *DB) Compact() (returnErr error) {
 		// so in order to make sure that overlaps are evaluated
 		// consistently, we explicitly remove the last value
 		// from the block interval here.
-		if err := db.compactHead(NewRangeHead(db.head, mint, maxt-1)); err != nil {
+		rh := NewRangeHeadWithIsolationDisabled(db.head, mint, maxt-1)
+
+		// Compaction runs with isolation disabled, because head.compactable()
+		// ensures that maxt is more than chunkRange/2 back from now, and
+		// head.appendableMinValidTime() ensures that no new appends can start within the compaction range.
+		// We do need to wait for any overlapping appenders that started previously to finish.
+		db.head.WaitForAppendersOverlapping(rh.MaxTime())
+
+		if err := db.compactHead(rh); err != nil {
 			return errors.Wrap(err, "compact head")
 		}
 		// Consider only successful compactions for WAL truncation.
diff --git a/tsdb/head.go b/tsdb/head.go
index 32e85c59930..f2c61a4443a 100644
--- a/tsdb/head.go
+++ b/tsdb/head.go
@@ -886,6 +886,13 @@ func (h *Head) WaitForPendingReadersInTimeRange(mint, maxt int64) {
 	}
 }
 
+// WaitForAppendersOverlapping waits for appends overlapping maxt to finish.
+func (h *Head) WaitForAppendersOverlapping(maxt int64) {
+	for maxt >= h.iso.lowestAppendTime() {
+		time.Sleep(500 * time.Millisecond)
+	}
+}
+
 // IsQuerierCollidingWithTruncation returns if the current querier needs to be closed and if a new querier
 // has to be created. In the latter case, the method also returns the new mint to be used for creating the
 // new range head and the new querier. This methods helps preventing races with the truncation of in-memory data.
@@ -1038,6 +1045,8 @@ func (h *Head) Stats(statsByLabelName string) *Stats {
 type RangeHead struct {
 	head       *Head
 	mint, maxt int64
+
+	isolationOff bool
 }
 
 // NewRangeHead returns a *RangeHead.
@@ -1050,12 +1059,23 @@ func NewRangeHead(head *Head, mint, maxt int64) *RangeHead {
 	}
 }
 
+// NewRangeHeadWithIsolationDisabled returns a *RangeHead that does not create an isolationState.
+func NewRangeHeadWithIsolationDisabled(head *Head, mint, maxt int64) *RangeHead {
+	rh := NewRangeHead(head, mint, maxt)
+	rh.isolationOff = true
+	return rh
+}
+
 func (h *RangeHead) Index() (IndexReader, error) {
 	return h.head.indexRange(h.mint, h.maxt), nil
 }
 
 func (h *RangeHead) Chunks() (ChunkReader, error) {
-	return h.head.chunksRange(h.mint, h.maxt, h.head.iso.State(h.mint, h.maxt))
+	var isoState *isolationState
+	if !h.isolationOff {
+		isoState = h.head.iso.State(h.mint, h.maxt)
+	}
+	return h.head.chunksRange(h.mint, h.maxt, isoState)
 }
 
 func (h *RangeHead) Tombstones() (tombstones.Reader, error) {
diff --git a/tsdb/head_append.go b/tsdb/head_append.go
index 1331fbe2526..f3e9016a6ae 100644
--- a/tsdb/head_append.go
+++ b/tsdb/head_append.go
@@ -124,7 +124,8 @@ func (h *Head) Appender(_ context.Context) storage.Appender {
 }
 
 func (h *Head) appender() *headAppender {
-	appendID, cleanupAppendIDsBelow := h.iso.newAppendID() // Every appender gets an ID that is cleared upon commit/rollback.
+	minValidTime := h.appendableMinValidTime()
+	appendID, cleanupAppendIDsBelow := h.iso.newAppendID(minValidTime) // Every appender gets an ID that is cleared upon commit/rollback.
 
 	// Allocate the exemplars buffer only if exemplars are enabled.
 	var exemplarsBuf []exemplarWithSeriesRef
@@ -134,7 +135,7 @@ func (h *Head) appender() *headAppender {
 
 	return &headAppender{
 		head:                  h,
-		minValidTime:          h.appendableMinValidTime(),
+		minValidTime:          minValidTime,
 		mint:                  math.MaxInt64,
 		maxt:                  math.MinInt64,
 		samples:               h.getAppendBuffer(),
diff --git a/tsdb/head_read.go b/tsdb/head_read.go
index ca34b9bbddd..7192fb7c94b 100644
--- a/tsdb/head_read.go
+++ b/tsdb/head_read.go
@@ -253,7 +253,9 @@ type headChunkReader struct {
 }
 
 func (h *headChunkReader) Close() error {
-	h.isoState.Close()
+	if h.isoState != nil {
+		h.isoState.Close()
+	}
 	return nil
 }
 
diff --git a/tsdb/isolation.go b/tsdb/isolation.go
index 46d13edf815..74d63c6af04 100644
--- a/tsdb/isolation.go
+++ b/tsdb/isolation.go
@@ -14,6 +14,7 @@
 package tsdb
 
 import (
+	"math"
 	"sync"
 )
 
@@ -45,6 +46,7 @@ func (i *isolationState) IsolationDisabled() bool {
 
 type isolationAppender struct {
 	appendID uint64
+	minTime  int64
 	prev     *isolationAppender
 	next     *isolationAppender
 }
@@ -116,6 +118,21 @@ func (i *isolation) lowWatermarkLocked() uint64 {
 	return i.appendsOpenList.next.appendID
 }
 
+// lowestAppendTime returns the lowest minTime for any open appender,
+// or math.MaxInt64 if no open appenders.
+func (i *isolation) lowestAppendTime() int64 {
+	var lowest int64 = math.MaxInt64
+	i.appendMtx.RLock()
+	defer i.appendMtx.RUnlock()
+
+	for a := i.appendsOpenList.next; a != i.appendsOpenList; a = a.next {
+		if lowest > a.minTime {
+			lowest = a.minTime
+		}
+	}
+	return lowest
+}
+
 // State returns an object used to control isolation
 // between a query and appends. Must be closed when complete.
 func (i *isolation) State(mint, maxt int64) *isolationState {
@@ -164,7 +181,7 @@ func (i *isolation) TraverseOpenReads(f func(s *isolationState) bool) {
 // newAppendID increments the transaction counter and returns a new transaction
 // ID. The first ID returned is 1.
 // Also returns the low watermark, to keep lock/unlock operations down.
-func (i *isolation) newAppendID() (uint64, uint64) {
+func (i *isolation) newAppendID(minTime int64) (uint64, uint64) {
 	if i.disabled {
 		return 0, 0
 	}
@@ -177,6 +194,7 @@ func (i *isolation) newAppendID() (uint64, uint64) {
 
 	app := i.appendersPool.Get().(*isolationAppender)
 	app.appendID = i.appendsOpenList.appendID
+	app.minTime = minTime
 	app.prev = i.appendsOpenList.prev
 	app.next = i.appendsOpenList
 


Test patch:
diff --git a/tsdb/isolation_test.go b/tsdb/isolation_test.go
index 6e27e06fc34..36083a1029e 100644
--- a/tsdb/isolation_test.go
+++ b/tsdb/isolation_test.go
@@ -18,8 +18,67 @@ import (
 	"strconv"
 	"sync"
 	"testing"
+
+	"github.com/stretchr/testify/require"
 )
 
+func TestIsolation(t *testing.T) {
+	type result struct {
+		id           uint64
+		lowWatermark uint64
+	}
+	var appendA, appendB result
+	iso := newIsolation(false)
+
+	// Low watermark starts at 1.
+	require.Equal(t, uint64(0), iso.lowWatermark())
+	require.Equal(t, int64(math.MaxInt64), iso.lowestAppendTime())
+
+	// Pretend we are starting to append.
+	appendA.id, appendA.lowWatermark = iso.newAppendID(10)
+	require.Equal(t, result{1, 1}, appendA)
+	require.Equal(t, uint64(1), iso.lowWatermark())
+
+	require.Equal(t, 0, countOpenReads(iso))
+	require.Equal(t, int64(10), iso.lowestAppendTime())
+
+	// Now we start a read.
+	stateA := iso.State(10, 20)
+	require.Equal(t, 1, countOpenReads(iso))
+
+	// Second appender.
+	appendB.id, appendB.lowWatermark = iso.newAppendID(20)
+	require.Equal(t, result{2, 1}, appendB)
+	require.Equal(t, uint64(1), iso.lowWatermark())
+	require.Equal(t, int64(10), iso.lowestAppendTime())
+
+	iso.closeAppend(appendA.id)
+	// Low watermark remains at 1 because stateA is still open
+	require.Equal(t, uint64(1), iso.lowWatermark())
+
+	require.Equal(t, 1, countOpenReads(iso))
+	require.Equal(t, int64(20), iso.lowestAppendTime())
+
+	// Finish the read and low watermark should rise.
+	stateA.Close()
+	require.Equal(t, uint64(2), iso.lowWatermark())
+
+	require.Equal(t, 0, countOpenReads(iso))
+
+	iso.closeAppend(appendB.id)
+	require.Equal(t, uint64(2), iso.lowWatermark())
+	require.Equal(t, int64(math.MaxInt64), iso.lowestAppendTime())
+}
+
+func countOpenReads(iso *isolation) int {
+	count := 0
+	iso.TraverseOpenReads(func(s *isolationState) bool {
+		count++
+		return true
+	})
+	return count
+}
+
 func BenchmarkIsolation(b *testing.B) {
 	for _, goroutines := range []int{10, 100, 1000, 10000} {
 		b.Run(strconv.Itoa(goroutines), func(b *testing.B) {
@@ -36,7 +95,7 @@ func BenchmarkIsolation(b *testing.B) {
 					<-start
 
 					for i := 0; i < b.N; i++ {
-						appendID, _ := iso.newAppendID()
+						appendID, _ := iso.newAppendID(0)
 
 						iso.closeAppend(appendID)
 					}
@@ -66,7 +125,7 @@ func BenchmarkIsolationWithState(b *testing.B) {
 					<-start
 
 					for i := 0; i < b.N; i++ {
-						appendID, _ := iso.newAppendID()
+						appendID, _ := iso.newAppendID(0)
 
 						iso.closeAppend(appendID)
 					}