Repository: prometheus/prometheus (language: go)

Issue:
Rule evaluation & scrapes are frequently blocked or delayed with Prometheus >= v2.33.0 
**What did you do?**

Upgraded Prometheus from v2.32.1 to v2.33.1

**What did you expect to see?**

Similar performance and smooth graphs.

**What did you see instead? Under which circumstances?**

It appears that starting with v2.33.0 Prometheus hits some scalability limits for us.
Something takes longer than it used to OR it started to block when it didn't use to.

We've started noticing that ever since we've upgrade Prometheus from v2.32.1 to v2.33.1 (same issue with v2.33.4) we're seeing an issue where on our biggest instances every 30 minutes we have:
- some counter updates are delayed, which looks like either scrapes are getting delayed - I only mention counters here because it's more visible on counters vs gauge, so either the actual HTTP scrape is delayed OR sample insertion to tsdb is delayed (or wherever sample timestamp is set on scrape)
- we see a massive spike in rule evaluation duration
- we see rule evaluation duration spike
- we see rule evaluation timeouts `query timed out in expression evaluation`

Still trying to debug it, so far it doesn't seem to be related to:
- queries - we don't see any elevated spike in queries
- go-routines - they stay flat, so it's not like Go gets so many goroutines that some gets left behind
- cpu or memory - don't see any elevated resource usage when this happens
- chunk write queue - since this is new code added in 2.33 with a default queue size of 1000 and our metrics show that the rate of elements added to the queue spikes to around 250k/s I tested Prometheus with a bigger queue size (up to 50M) with no effect on this issue
- query concurrency limit - since that's 20 by default and we seem to usually issue more queries per second I suspected we might be queuing queries too much, but bumping this value up to 96 (on a server with 128 cores) didn't seem to change anything

Since this is happening every 30 minutes and only seem to affect our biggest instances with ~15M time series I've checked what else is happening every 30 minutes. We do have `--storage.tsdb.min-block-duration=30m` & `--storage.tsdb.max-block-duration=30m`, mostly to reduce memory usage as we do have a fair amount of metrics churn and so more frequent HEAD compaction helps keep memory usage lower than it would be without it.
What I've also noticed is that tsdb HEAD active appenders are spiking around the time of this issue. Likely because they spike when there's HEAD/block compaction, so not sure if that's the effect or the cause. Looking at historical metrics I see that it was always spiking around that time, but with 2.33 spikes are bigger. See metrics below:

19dm12 - v2.32.1
19dm13 - v2.33.4

![screencapture-metrics-cfdata-org-graph-2022-03-01-10_40_04](https://user-images.githubusercontent.com/114288/156164902-0cab7b94-6536-48fc-941a-c432f66545d2.png)

I didn't find any useful logs that would point me in any other direction so far and not sure what other metrics might be relevant here. Any tips on further debugging would be very helpful.

**Environment**

* System information:

	`Linux 5.15.19 x86_64`

* Prometheus version:

	insert output of `prometheus --version` here

* Alertmanager version:

	insert output of `alertmanager --version` here (if relevant to the issue)

* Prometheus configuration file:
```
insert configuration here
```

* Alertmanager configuration file:
```
insert configuration here (if relevant to the issue)
```


* Logs:
```
insert Prometheus and Alertmanager logs relevant to the issue here
```



Discussion / hints:
CPU profiles via pprof didn't show me much but after setting `DEBUG=1` I was able to capture 
mutex profile which seems a bit more useful:

![profile001](https://user-images.githubusercontent.com/114288/156210233-78b350fa-f08d-49d5-91bc-e5e055fb8a69.svg)

It does seem to point to chunk write queue mutex if I read that right.
https://github.com/prometheus/prometheus/blob/main/tsdb/chunks/chunk_write_queue.go#L140-L142

Metrics do show a spike in chunk write queue operations around the time of the issue. I've noticed that before but suspected channel queue length, which I verified isn't the issue here. What I see on metrics is add/complete operations spiking to ~200k per second, while get stays at <100 per second. Given that get uses read lock is it possible that get read locks are starved by write locks?

I will try to replace `sync.RWMutex` there with a simple `sync.Mutex`, hoping it won't starve reads, and see if that improves things for me.
Based on some quick tests it does seem that simply swapping RWMutex for a Mutex solves this for me, raised #10379.
But I'll need to give it a bit more time to confirm.
^ 
I don’t think mutex is enough to fix read starvation.
After applying this patch there was initial improvement but it was likely due to fewer writes, after a while it went back to being slow again.
Another pprof capture:

![svgviewer-output](https://user-images.githubusercontent.com/114288/156341243-e6866a06-b472-4112-acca-f840b4049be4.svg)

Given that scrape results go through the same chunk write queue it could explain the delay in some metrics.
@colega / @codesome I believe this issue is a direct result of #10051 and I'm not sure what can be done here to resolve this.
cc @replay 
The first idea which comes to my mind to alleviate the lock contention on `chunkWriteQueue.chunkRefMapMtx` would be to shard the `chunkWriteQueue.chunkRefMap`. All access to it (except for `chunkWriteQueue.queueSize()` which is only used in tests) happens via keys that are of the type `ChunkDiskMapperRef` which is a numeric type so a `% <shard count>` should be fast, we could shard it by a fix number like for example `1024`, then we could also split the lock `chunkWriteQueue.chunkRefMapMtx` into `1024` locks. WDYT?

Edit: We would have to first check how the distribution of the `ChunkDiskMapperRef` values would look like if we `% 1024` them, I'm not sure about that right now. If the distribution is very uneven we might have to use some hash which would be more expensive.
@prymitive do you possibly have any profiles and goroutine dumps when the queries are taking too long? Or can you get one?

@replay sounds like a plan, and would be good if we can see the difference in a benchmark. I would suggest sharding on the series ref for a more uniform spread.
I don't have goroutine dumps but I can try to capture some, although metrics are showing around 350k goroutines so that might be hard to read.
I can capture more pprof profiles, is it the cpu profile you're after?
In general I have a single instance I've upgraded to v2.33.4 that I'm using for testing, I can usually capture all sort of debugging data there, just need to know what to run.
Of those 350K, most will be sleeping waiting to scrape something esle.  There are [tools](https://github.com/maruel/panicparse) to crunch down all similar goroutines.
If you GET `/debug/pprof/goroutine?debug=1` it will pre-crunch them down, though you lose a bit of detail.
`/debug/pprof/goroutine?debug=2` has full detail.
here's `go tool pprof ... /debug/pprof/goroutine?debug=1`, passing `debug=2` gives me `unrecognized profile format` for some reason.

![profile003](https://user-images.githubusercontent.com/114288/156394445-6425bf79-d50a-4059-843a-8f149a08b680.svg)

[pprof.prometheus.goroutine.027.pb.gz](https://github.com/prometheus/prometheus/files/8170801/pprof.prometheus.goroutine.027.pb.gz)



The `debug` ones are just text; just `curl` it and pipe to a file.
[debug2-2022-03-02T16:32:36+00:00.txt.gz](https://github.com/prometheus/prometheus/files/8171275/debug2-2022-03-02T16.32.36%2B00.00.txt.gz)

Forgot to mention that this was from a slightly patched version with some extra metrics I've added to try and debug this.
I'll revert to vanilla v2.33.4 for future captures
Provided some more debug details on #10386

I've also collected same profiles from an instance running v2.32.1 and scraping same metrics.
Here's how it compares:

# Block profile

v2.32.1
```
lukasz@19dm12:~$ go tool pprof -top http://metrics-interface:3000/debug/pprof/block | head -n 20
Fetching profile over HTTP from http://metrics-interface:3000/debug/pprof/block
Saved profile in /home/lukasz/pprof/pprof.prometheus.contentions.delay.035.pb.gz
File: prometheus
Type: delay
Time: Mar 3, 2022 at 12:31pm (UTC)
Showing nodes accounting for 91182.88hrs, 100% of 91211.55hrs total
Dropped 543 nodes (cum <= 456.06hrs)
      flat  flat%   sum%        cum   cum%
91182.88hrs   100%   100% 91182.88hrs   100%  runtime.selectgo
         0     0%   100% 51691.40hrs 56.67%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).run
         0     0%   100% 39372.41hrs 43.17%  net/http.(*persistConn).writeLoop
```

v2.33.4 + patch from #10386
```
lukasz@19dm13:~$ go tool pprof -top http://metrics-interface:3000/debug/pprof/block | head -n 20
Fetching profile over HTTP from http://metrics-interface:3000/debug/pprof/block
Saved profile in /home/lukasz/pprof/pprof.prometheus.contentions.delay.146.pb.gz
File: prometheus
Type: delay
Time: Mar 3, 2022 at 12:31pm (UTC)
Showing nodes accounting for 34228.43hrs, 99.92% of 34254.25hrs total
Dropped 533 nodes (cum <= 171.27hrs)
      flat  flat%   sum%        cum   cum%
33771.09hrs 98.59% 98.59% 33771.09hrs 98.59%  runtime.selectgo
 457.34hrs  1.34% 99.92%  457.34hrs  1.34%  sync.(*Mutex).Lock (inline)
         0     0% 99.92% 19655.63hrs 57.38%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).run
         0     0% 99.92%  475.13hrs  1.39%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport
         0     0% 99.92%  443.07hrs  1.29%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport.func1
         0     0% 99.92%  443.37hrs  1.29%  github.com/prometheus/prometheus/storage.(*fanoutAppender).Commit
         0     0% 99.92%  443.37hrs  1.29%  github.com/prometheus/prometheus/tsdb.(*headAppender).Commit
         0     0% 99.92%  441.58hrs  1.29%  github.com/prometheus/prometheus/tsdb.(*memSeries).append
         0     0% 99.92%  441.58hrs  1.29%  github.com/prometheus/prometheus/tsdb.(*memSeries).cutNewHeadChunk
         0     0% 99.92%  441.58hrs  1.29%  github.com/prometheus/prometheus/tsdb.(*memSeries).mmapCurrentHeadChunk
         0     0% 99.92%  443.37hrs  1.29%  github.com/prometheus/prometheus/tsdb.dbAppender.Commit
         0     0% 99.92%  441.58hrs  1.29%  github.com/prometheus/prometheus/tsdb/chunks.(*ChunkDiskMapper).WriteChunk
         0     0% 99.92% 14515.88hrs 42.38%  net/http.(*persistConn).writeLoop
```

# Mutex profile

v2.32.1
```
lukasz@19dm12:~$ go tool pprof -top http://metrics-interface:3000/debug/pprof/mutex | head -n 20
Fetching profile over HTTP from http://metrics-interface:3000/debug/pprof/mutex
Saved profile in /home/lukasz/pprof/pprof.prometheus.contentions.delay.034.pb.gz
File: prometheus
Type: delay
Time: Mar 3, 2022 at 12:31pm (UTC)
Showing nodes accounting for 172.62s, 100% of 172.62s total
Dropped 151 nodes (cum <= 0.86s)
      flat  flat%   sum%        cum   cum%
   135.83s 78.68% 78.68%    135.83s 78.68%  sync.(*Mutex).Unlock (partial-inline)
    34.93s 20.23% 98.92%     86.42s 50.06%  sync.(*RWMutex).Unlock
     1.87s  1.08%   100%      1.87s  1.08%  sync.(*RWMutex).RUnlock
         0     0%   100%      2.16s  1.25%  github.com/prometheus/client_golang/prometheus.(*summary).Observe
         0     0%   100%      1.27s  0.74%  github.com/prometheus/prometheus/promql.(*Engine).exec
         0     0%   100%      1.27s  0.74%  github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt
         0     0%   100%      1.27s  0.74%  github.com/prometheus/prometheus/promql.(*query).Exec
         0     0%   100%      1.70s  0.99%  github.com/prometheus/prometheus/rules.(*Group).Eval
         0     0%   100%      1.70s  0.99%  github.com/prometheus/prometheus/rules.(*Group).Eval.func1
         0     0%   100%      1.71s  0.99%  github.com/prometheus/prometheus/rules.(*Group).run
         0     0%   100%      1.70s  0.99%  github.com/prometheus/prometheus/rules.(*Group).run.func1
         0     0%   100%      1.71s  0.99%  github.com/prometheus/prometheus/rules.(*Manager).Update.func1
         0     0%   100%      1.22s  0.71%  github.com/prometheus/prometheus/rules.EngineQueryFunc.func1
         0     0%   100%     11.80s  6.83%  github.com/prometheus/prometheus/scrape.(*Manager).TargetsActive.func1
```

v2.33.4 + patch from #10386
```
lukasz@19dm13:~$ go tool pprof -top http://metrics-interface:3000/debug/pprof/mutex | head -n 20
Fetching profile over HTTP from http://metrics-interface:3000/debug/pprof/mutex
Saved profile in /home/lukasz/pprof/pprof.prometheus.contentions.delay.145.pb.gz
File: prometheus
Type: delay
Time: Mar 3, 2022 at 12:31pm (UTC)
Showing nodes accounting for 2666.91s, 99.68% of 2675.61s total
Dropped 185 nodes (cum <= 13.38s)
      flat  flat%   sum%        cum   cum%
  2639.59s 98.65% 98.65%   2639.59s 98.65%  sync.(*Mutex).Unlock (partial-inline)
    27.32s  1.02% 99.68%   2419.18s 90.42%  sync.(*RWMutex).Unlock
         0     0% 99.68%    102.38s  3.83%  github.com/prometheus/prometheus/rules.(*Group).Eval
         0     0% 99.68%    102.38s  3.83%  github.com/prometheus/prometheus/rules.(*Group).Eval.func1
         0     0% 99.68%     99.77s  3.73%  github.com/prometheus/prometheus/rules.(*Group).Eval.func1.2
         0     0% 99.68%    102.38s  3.83%  github.com/prometheus/prometheus/rules.(*Group).run
         0     0% 99.68%    102.38s  3.83%  github.com/prometheus/prometheus/rules.(*Group).run.func1
         0     0% 99.68%    102.38s  3.83%  github.com/prometheus/prometheus/rules.(*Manager).Update.func1
         0     0% 99.68%     27.20s  1.02%  github.com/prometheus/prometheus/scrape.(*scrapeCache).setHelp
         0     0% 99.68%     39.67s  1.48%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).append
         0     0% 99.68%   2478.36s 92.63%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).run
         0     0% 99.68%   2478.36s 92.63%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport
         0     0% 99.68%   2418.23s 90.38%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport.func1
         0     0% 99.68%     17.68s  0.66%  github.com/prometheus/prometheus/scrape.newScrapePool.func2.3
```

Raw profile captures & goroutine dumps:

![19dm12 block](https://user-images.githubusercontent.com/114288/156567296-87e2edc1-0db3-4c5f-affe-5a3162c5b31c.svg)
![19dm12 mutex](https://user-images.githubusercontent.com/114288/156567298-9bbcbb41-64e4-4258-a91d-d4915a8910a0.svg)
[19dm12.txt.gz](https://github.com/prometheus/prometheus/files/8177915/19dm12.txt.gz)
![19dm13 block](https://user-images.githubusercontent.com/114288/156567317-020531d5-1607-43bd-807a-802ca14281cc.svg)
![19dm13 mutex](https://user-images.githubusercontent.com/114288/156567319-d5dc42b7-45e1-4a06-8e20-baf81de3e2de.svg)
[19dm13.txt.gz](https://github.com/prometheus/prometheus/files/8177917/19dm13.txt.gz)


Greetings - we are also experiencing this behavior on v2.33.0 and v2.33.5, where scrapes, normally with 15s interval, can stall up to 80s before the next sample is scraped/written to disk.  We can see this by examining timestamps of raw data in table view, in this case `node_netstat_Tcp_InSegs[2m]`:

```
2345711942670 @1646783980.222
2345718048732 @1646783995.22
2345724484235 @1646784010.22
2345758596459 @1646784088.538
```

Graphing `time() - timestamp(node_netstat_Tcp_InSegs)` allows us a better overview of these stalls per target: 
![Screen Shot 2022-03-08 at 4 13 25 PM](https://user-images.githubusercontent.com/49920412/157347575-3d777bbf-f0c7-4cda-93d7-d0ee128ce033.png)

As this stall behavior is not uniform among targets, it leads to surprising dips in `sum(node_netstat_Tcp_InSegs)` recording rules and site wide surveys of things like network activity. We see other job/instance pairs have different chances of experiencing a scrape stall, where the largest jobs per `scrape_samples_scraped` seem to have the worst stall behavior. 

We are also seeing large spikes in `prometheus_rule_group_last_duration_seconds`
![Screen Shot 2022-03-08 at 4 20 22 PM](https://user-images.githubusercontent.com/49920412/157348673-3b0fd568-d310-4197-9343-36771bb2487a.png)

And `prometheus_rule_group_iterations_missed_total`
![Screen Shot 2022-03-08 at 4 26 06 PM](https://user-images.githubusercontent.com/49920412/157348809-54207d94-1030-4976-a298-163bc902513e.png)

We have seen these stalls on Prometheus nodes with between 4M and 2.5M `prometheus_tsdb_head_series` and >130k/s prometheus_tsdb_head_samples_appended_total. We are configured with `--storage.tsdb.max-block-duration=2h --storage.tsdb.min-block-duration=2h` so we are seeing the stalls every 2h. 

We have been able to downgrade to v2.32.1 and have some hosts on latest released to continue debugging.
I see that [v2.34.0-rc.1](https://github.com/prometheus/prometheus/releases/tag/v2.34.0-rc.1) is out so final v2.34 is around the corner. At the same time the only potential fix - #10386 - didn't get any progress and I don't have high hopes that adding more locks will resolve this.

Is it possible to revert chunk write queue until all locking issues around it are addressed so that there is still a working upgrade path from v2.32.x?
I think we can start with adding an option to disable the queue (not by default) while we try to improve the performance. But it will go in v2.35.

We are suspecting it is something to do with expensive rules. So we are investigating in that line.
@replay did some investigations and turns out the rule evaluation duration metric includes the time taken to write the result of the rule evaluation back to the TSDB. And given scrapes and rules are getting blocked, it is most likely the writes that are getting bottle-necked.

@replay agreed to work on introducing an option to disable the queue.
Thanks, looking forward to testing more patches.

> I think we can start with adding an option to disable the queue (not by default) while we try to improve the performance. But it will go in v2.35.

Doesn't this essentially mean that everyone needs to opt-in to get the same level of performance as with 2.32?
If new code is a source of a major performance degradation then shouldn't the new code path be opt-in, rather than opt-out?
While working on this I think I have found a bug, which might very well be the reason for the issues you're seeing. Would you mind giving this branch a try? https://github.com/prometheus/prometheus/pull/10426
Will do, thanks!
I've deployed 2.33.5 + patch from #10426 to my test instance and it doesn't look any better so far, it might be even worse, I see more query timeouts in logs and rule evaluation metric shows even bigger and longer spike.
That being said I'll have more metrics if I let it run overnight.


![12](https://user-images.githubusercontent.com/114288/157509260-e469630b-868c-40b1-9ba6-7fd27e957d3a.png)



That's unfortunate, would you mind providing another block profile when you get a chance? I'd be curious to know if it still blocks at the same place now that the queue has a size `>0`.

Edit:
It would likely also help to try a larger queue size, like for example `100000` or `1000000`, basically enough to hold a whole ingestion spike in the queue. In our internal test we have set it to `6M`, so maybe the default of `1k` is just way too low.
In this PR I'm making it possible to completely disable the chunk write queue. I'm also raising the default size to `1M` because I have come to believe that `1k` is way too low:
https://github.com/prometheus/prometheus/pull/10425
> That's unfortunate, would you mind providing another block profile when you get a chance? I'd be curious to know if it still blocks at the same place now that the queue has a size `>0`.

![profile001](https://user-images.githubusercontent.com/114288/157644383-9962e92c-78e5-459f-b780-5da5a527a7f8.svg)
![profile002](https://user-images.githubusercontent.com/114288/157644388-9142d547-3a3f-4d23-a7a9-4a6dbdbbfc51.svg)


I did deploy v2.33.5 + patch from #10425, just applying the patch didn't change anything, same problem as before.
Then I restarted that instance with `--storage.tsdb.no-head-chunks-write-queue` flag and the issue is gone.

![profile001](https://user-images.githubusercontent.com/114288/157674398-46720354-161e-470f-b9f3-00062103669e.svg)
![profile002](https://user-images.githubusercontent.com/114288/157674401-83fccc3c-c5fa-48a3-a94f-ed5f18d955ec.svg)


To remove any pain for users, we will disable it by default while still being able to enable if required. And in the meantime we will run it enabled in our clusters and try to reduce the performance issues. We can remove it entirely if we are not able to make it better.
FYI I updated this PR accordingly: https://github.com/prometheus/prometheus/pull/10425/
Thanks @prymitive for running all the PRs and giving us numbers. Hopefully the updated https://github.com/prometheus/prometheus/pull/10425 now does not cause any issues.
Thanks!
Due to this issue I've been having a closer look our rule evaluation and one of the alerts I've added pointed me at a single instance running v2.32.1 that shows similar symptoms. I do see rule evaluation timeouts every 30 minutes on that instance, but it's all limited to a single rule group that runs something expensive. Normally those queries take about 50s (with 1m interval) but when compaction happens it spikes up to over 2m, causing timeouts. Some other queries also show a spike in duration, but because these are faster they still finish under 2m.
Looking at historical metrics I see that this was faster before, while still on v2.32.1, so we probably added some expensive queries to that rule group around the same time we've upgraded to v2.33.x, and it's likely that the combination of:
a) having expensive queries that finish just under rule interval
b) adding chunk write queue
that made this whole issue more visible and impacting for us by slowing things down during compaction by some extra % that finally caused too many queries to timeout during compaction and creating big noticeable blast radius.

I'll get mutex/block pprofs later today but looking at a goroutine dumps I see that the limiting path seems to be isolation code in tsdb.

```
[...]
2202: semacquire [Created by scrape.(*scrapePool).sync @ scrape.go:584]
    sync          sema.go:71                  runtime_SemacquireMutex(*, *, *)
    sync          mutex.go:138                (*Mutex).lockSlow(#29182)
    sync          mutex.go:81                 (*Mutex).Lock(...)
    sync          rwmutex.go:111              (*RWMutex).Lock(*)
    tsdb          isolation.go:171            (*isolation).newAppendID(#29182)
    tsdb          head_append.go:117          (*Head).appender(#29139)
    tsdb          head_append.go:113          (*Head).Appender(#29139, {*, *})
    tsdb          db.go:833                   (*DB).Appender(#29181, {#8672, *})
    main          main.go:1343                (*readyStorage).Appender(*, {#8672, *})
    storage       fanout.go:118               (*fanout).Appender(#29176, {#8672, *})
    scrape        scrape.go:308               newScrapePool.func2.3({#8672, *})
    scrape        scrape.go:1244              (*scrapeLoop).scrapeAndReport(*, {*, #6, #6}, {*, #6, #6}, 0)
    scrape        scrape.go:1203              (*scrapeLoop).run(*, *)
7: runnable [Created by rules.(*Manager).Update @ manager.go:989]
    sync          sema.go:71                  runtime_SemacquireMutex(*, *, *)
    sync          rwmutex.go:63               (*RWMutex).RLock(...)
    tsdb          isolation.go:122            (*isolation).State(#29182, *, *)
    tsdb          head.go:1026                (*RangeHead).Chunks(*)
    tsdb          querier.go:62               newBlockBaseQuerier({#8687, *}, *, *)
    tsdb          querier.go:116              NewBlockQuerier({#8687, *}, *, *)
    tsdb          db.go:1518                  (*DB).Querier(#29181, {*, #3722}, *, *)
    main          main.go:1313                (*readyStorage).Querier(0, {#5, *}, *, *)
    storage       fanout.go:74                (*fanout).Querier(#29176, {#5, *}, *, *)
    promql        engine.go:568               (*Engine).execEvalStmt(#29203, {#5, *}, *, *)
    promql        engine.go:548               (*Engine).exec(#29203, {#5, *}, *)
    promql        engine.go:186               (*query).Exec(*, {#5, *})
    rules         manager.go:191              EngineQueryFunc.func1({#5, *}, {*, *}, {*, *, 0})
    rules         alerting.go:308             (*AlertingRule).Eval(*, {#5, *}, {#8647, #8490, 0}, #29202, #29147, 0)
    rules         manager.go:600              (*Group).Eval.func1({#5, *}, *, {*, *, 0}, *, 0, {#8691, *})
    rules         manager.go:680              (*Group).Eval(*, {#5, *}, {#8647, #6, 0})
    rules         manager.go:350              (*Group).run.func1()
    rules         manager.go:425              (*Group).run(*, {#8673, #29131})
    rules         manager.go:999              (*Manager).Update.func1(*)
7: runnable [Created by rules.(*Manager).Update @ manager.go:989]
    sync          sema.go:71                  runtime_SemacquireMutex(*, *, *)
    sync          rwmutex.go:63               (*RWMutex).RLock(...)
    tsdb          isolation.go:122            (*isolation).State(#29182, *, *)
    tsdb          head.go:1026                (*RangeHead).Chunks(*)
    tsdb          querier.go:62               newBlockBaseQuerier({#8687, *}, *, *)
    tsdb          querier.go:116              NewBlockQuerier({#8687, *}, *, *)
    tsdb          db.go:1518                  (*DB).Querier(#29181, {*, #3722}, *, *)
    main          main.go:1313                (*readyStorage).Querier(0, {#5, *}, *, *)
    storage       fanout.go:74                (*fanout).Querier(#29176, {#5, *}, *, *)
    promql        engine.go:568               (*Engine).execEvalStmt(#29203, {#5, *}, *, *)
    promql        engine.go:548               (*Engine).exec(#29203, {#5, *}, *)
    promql        engine.go:186               (*query).Exec(*, {#5, *})
    rules         manager.go:191              EngineQueryFunc.func1({#5, *}, {*, *}, {*, *, 0})
    rules         recording.go:77             (*RecordingRule).Eval(*, {#5, *}, {#8647, #8490, 0}, #29202, 0, 0)
    rules         manager.go:600              (*Group).Eval.func1({#5, *}, *, {*, *, 0}, *, 0, {#8692, *})
    rules         manager.go:680              (*Group).Eval(*, {#5, *}, {#8647, #6, 0})
    rules         manager.go:350              (*Group).run.func1()
    rules         manager.go:425              (*Group).run(*, {#8673, #29131})
    rules         manager.go:999              (*Manager).Update.func1(*)
1: semacquire [Created by scrape.(*scrapePool).sync @ scrape.go:584]
    sync          sema.go:71                  runtime_SemacquireMutex(*uint32(#29182), true, 1)
    sync          rwmutex.go:116              (*RWMutex).Lock(*RWMutex(#125052))
    tsdb          isolation.go:171            (*isolation).newAppendID(#29182)
    tsdb          head_append.go:117          (*Head).appender(#29139)
    tsdb          head_append.go:113          (*Head).Appender(#29139, {#157719, 0xf52f25f0eb3fda95})
    tsdb          db.go:833                   (*DB).Appender(#29181, {#8672, #55035})
    main          main.go:1343                (*readyStorage).Appender(#187656, {#8672, #55035})
    storage       fanout.go:118               (*fanout).Appender(#29176, {#8672, #55035})
    scrape        scrape.go:308               newScrapePool.func2.3({#8672, #55035})
    scrape        scrape.go:1244              (*scrapeLoop).scrapeAndReport(#90907, {#29111, #6, #6}, {#29098, #6, #6}, 0)
    scrape        scrape.go:1203              (*scrapeLoop).run(#90907, 0)
1: runnable [Created by rules.(*Manager).Update @ manager.go:989]
    tsdb          isolation.go:134            (*isolation).State(#29182, #174408, #185914)
    tsdb          head.go:1026                (*RangeHead).Chunks(#164141)
    tsdb          querier.go:62               newBlockBaseQuerier({#8687, #164141}, #174408, #185914)
    tsdb          querier.go:116              NewBlockQuerier({#8687, #164141}, #163010, #163011)
    tsdb          db.go:1518                  (*DB).Querier(#29181, {#163012, #3722}, #174408, #185914)
    main          main.go:1313                (*readyStorage).Querier(0, {#5, #155099}, #8665, #52289)
    storage       fanout.go:74                (*fanout).Querier(#29176, {#5, #155099}, #11114, #163301)
    promql        engine.go:568               (*Engine).execEvalStmt(#29203, {#5, #155098}, #149612, #127812)
    promql        engine.go:548               (*Engine).exec(#29203, {#5, #155098}, #149612)
    promql        engine.go:186               (*query).Exec(#149612, {#5, #155097})
    rules         manager.go:191              EngineQueryFunc.func1({#5, #155097}, {#157980, 0}, {0, 0, 0})
    rules         alerting.go:308             (*AlertingRule).Eval(#49741, {#5, #155097}, {#8647, #8490, 0}, #29202, #29147, 0)
    rules         manager.go:600              (*Group).Eval.func1({#5, #42598}, #49742, {#163300, #163300, 0}, #163013, 0, {#8691, #49741})
    rules         manager.go:680              (*Group).Eval(#49742, {#5, #42598}, {#8647, #6, 0})
    rules         manager.go:350              (*Group).run.func1()
    rules         manager.go:425              (*Group).run(#49742, {#8673, #29131})
    rules         manager.go:999              (*Manager).Update.func1(#48437)
[...]
```


![profile001](https://user-images.githubusercontent.com/114288/158180168-965275c8-e286-465c-ad18-a2923e1860f5.svg)

I did spend some time looking at this and the trigger seems to be `--storage.tsdb.max-block-duration=30m` we have set.
This causes Prometheus to cut new chunks every 30 minutes (synchronized to wall clock, so they all "expire" at the same time) and when that happens we see (according to metrics) up to ~200k chunks created per second. This seems to slow down sample commits from rule evaluations (taking well into 30-50 seconds) because it causes a call to `WriteChunk()` which seem to be the bottleneck here, possibly due to lock contention, but need to look more into that path to see what's slowing it down.

Because cutting new chunks due to reaching `--storage.tsdb.max-block-duration=30m` seems to be bound to wall clock I'm not sure what can be done about it because I don't fully understand what would happen if a chunk remains open past that point (will it break anything if we cut a new one a few minutes later?) and so not sure if rate limiting new chunk cuts would be sufficient & safe there.
Another option might be to reduce strip size which is currently 16K to try to bring down the overall number of chunks down. I will try to experiment a bit more next week, any pointers appreciated.
I did try to rate limit chunk creation to verify if that would solve the issues I'm seeing with this simple patch:

```diff
diff --git a/tsdb/head_append.go b/tsdb/head_append.go
index 13d56de7b..b956030db 100644
--- a/tsdb/head_append.go
+++ b/tsdb/head_append.go
@@ -20,6 +20,7 @@ import (
 
        "github.com/go-kit/log/level"
        "github.com/pkg/errors"
+       "golang.org/x/time/rate"
 
        "github.com/prometheus/prometheus/model/exemplar"
        "github.com/prometheus/prometheus/model/labels"
@@ -29,6 +30,10 @@ import (
        "github.com/prometheus/prometheus/tsdb/record"
 )
 
+var (
+       newChunkRateLimiter = rate.NewLimiter(rate.Limit(1000), 2000)
+)
+
 // initAppender is a helper to initialize the time bounds of the head
 // upon the first sample it receives.
 type initAppender struct {
@@ -524,7 +529,7 @@ func (s *memSeries) append(t int64, v float64, appendID uint64, chunkDiskMapper
        // Since we assume that the rate is higher, we're being conservative and cutting at 2*samplesPerChunk
        // as we expect more chunks to come.
        // Note that next chunk will have its nextAt recalculated for the new rate.
-       if t >= s.nextAt || numSamples >= samplesPerChunk*2 {
+       if (t >= s.nextAt || numSamples >= samplesPerChunk*2) && newChunkRateLimiter.Allow() {
                c = s.cutNewHeadChunk(t, chunkDiskMapper)
                chunkCreated = true
        }
```

and it looks like it does, no longer seeing rule evaluation getting too slow. And I do see a a long low rate new chunks cuts after max-block-duration is reached (instead of a short big burst). That being said my previous comment still stands - no idea if this is *safe*.
I did have some time to look a bit more into it and using trace endpoint seem to give me better information as I can see a better goroutine profile there with more in-depth information on what are all those goroutines spending time on.
Once again locking comes on top. There's a lot of rules and scrapes waiting on locks and they spend a lot of time there.

Tracing the code back I see that the problem is (as noted above) triggered by the fact that in-memory chunks get written out and mmapped all at the same time - which was already identified as a bottleneck in #9869.

Main problem seems to be the fact that writing a new chunk is slow since it's effectively serialized, I *think* to ensure that they're written in the correct order - https://github.com/prometheus/prometheus/blob/78f264578722493dcd8bf25c43bde7865a18792f/tsdb/chunks/head_chunks.go#L380-L383
Which means that when, in my test setup, we have 21+M time series, we will need to write out 21M chunks, one by one. And it's easy for appends to get stuck trying to get a lock for long enough to timeout a scrape or rule evaluation.
Reads are also at risk of getting stuck waiting on a lock that way, but it looks like it's the cutting of new chunks that's the biggest problem here.

I know that chunk write queue was suppose to address this limitation, but it somehow made it worse. You can see that on screenshots below that clearly show a much bigger and much longer drop in appended sample rate when chunk_write_queue was enabled with 10M value.

<details>
  <summary>Chunk Write Queue disabled</summary>
 </details>
![Screenshot from 2022-05-09 13-39-06](https://user-images.githubusercontent.com/114288/167459628-5843a3c1-83a0-412d-b0ba-e009dd1a0907.png)
![scrape-block-profile](https://user-images.githubusercontent.com/114288/167459630-f40c3963-b0d3-41e0-a528-b79686c50ef8.svg)
![update-block-profile](https://user-images.githubusercontent.com/114288/167459632-49855084-65fa-44e9-a9bf-942648c91f43.svg)
![screencapture-127-0-0-1-40375-goroutine-2022-05-09-13_35_00](https://user-images.githubusercontent.com/114288/167459634-ebc763fb-4f3d-4a06-bf13-6ad6ee31523d.png)

<details>

  <summary>Chunk Write Queue enabled</summary>
  
![queue-scrape-block-profile](https://user-images.githubusercontent.com/114288/167459667-92cfe2f1-29ea-45c6-b7c7-3092565cf5f2.svg)
![Screenshot from 2022-05-09 15-41-29](https://user-images.githubusercontent.com/114288/167459671-920f70af-d736-416e-9f3a-f58b4cb42011.png)
![queue-update-block-profile](https://user-images.githubusercontent.com/114288/167459678-c0b6d3b4-dff6-42b5-9fb6-ba6a5e3c82e4.svg)
![screencapture-127-0-0-1-39431-goroutine-2022-05-09-15_40_31](https://user-images.githubusercontent.com/114288/167459679-1095c122-34ec-4444-a910-e75298258c8d.png)

</details>

I experimented a bit with turning memSeries.headChunk into a slice, rather then just a single chunk.
https://github.com/prometheus/prometheus/blob/78f264578722493dcd8bf25c43bde7865a18792f/tsdb/head.go#L1524

Hoping that I could just append a new chunk to that slice when we need to cut a new one, and fire off a background goroutine to write out all but last chunk on that slice. But the first problem I've run into was with chunk references in https://github.com/prometheus/prometheus/blob/main/tsdb/head_read.go#L149 since it doesn't seem I can just keep adding one to each still in-memory chunk. So then I tried to merge samples from all in-memory chunks on read, which kinda works, but fails tests (seems like I'm not doing it properly, https://github.com/prymitive/prometheus/commit/26bd3a8c4058918b4b037497648f6d097122e9a2) and so I cannot test it to verify if it improves anything.

Happy to help moving this forward but I lack context and familiarity with the code base here to offer any help, unless I get some pointers for a potential solution from @codesome . Whether that's tweaking chunk_write_queue or some other changes.
I spent some time on a patch that delays mmapping of chunks - see #10709.
At the moment we have a single in-memory chunk and everything other than that most recent chunk is mmapped.
With my patch head keeps a list of in-memory chunks, so when we need to cut a new chunk we just append it there. Then all but last chunk will be written out and mmapped on a schedule.
The goal is to remove mmapping from the hot path.

Now I've added a background goroutine to head to deal with mmapping and it runs on a timer, but that's an arbitrary choice and so others might have better ideas.

Early testing shows that the issue where rules and scrapes are timing out when all head chunks reach `--storage.tsdb.max-block-duration` is mostly gone for me.
I've been running 2 instances patched with #10709 for the past week and in both cases this issue is solved for me when using that patch.
It's hard to write a benchmark that reproduces it but I think the key ingredients (I think) are:
- enough time series (and so chunks) so that mmapping of it all takes long enough (21-25M series in my case)
- plenty of rules producing a lot of samples that depend on each other - when a rule that produces lots of samples commit results and that commit get slow because chunks reach max time at the same time, and so there's a need to mmap a lot of chunks, which is then effectively serialised here: https://github.com/prometheus/prometheus/blob/d3c9c4f57488d924cf9a6ba3b1a7e98f6cf35464/tsdb/chunks/head_chunks.go#L381-L384, then next rule evaluation of a rule that wants to read samples from the chunk that's still being mmapped will get blocked on it. An example would be having a rule that calculates rate() for all target, then a rule that aggregates rate across all targets, if that aggregation uses more than one metric (ideally one that's also using another rule calculating rate) then any blocking will become more visible
@prymitive Out of curiosity, did you also notice that WAL replay was also much slower with Prometheus 2.33+? We run some similar-sized instances to what you describe (~12M series), and I've seen them take **much** longer to restart than they used to. It's not 100% repeatable either - but when it happens, there is hardly any CPU or IO activity for several minutes (when it would normally peg every core at 100%) - leading me to believe that something is waiting for a lock.

Two identical instances, with identical config (i.e. an HA pair) have wildly different WAL replay times - here with Prometheus v2.33.5:
```
Jun 19 11:14:34 prom2 prometheus[4004465]: ts=2022-06-19T11:14:34.572Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655625600001 maxt=1655632800000 ulid=01G5XSX5WDSWJCMTDGHFZ6SRFT
Jun 19 11:15:12 prom2 prometheus[4004465]: ts=2022-06-19T11:15:12.328Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
Jun 19 11:15:12 prom2 prometheus[4004465]: ts=2022-06-19T11:15:12.660Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=331.753036ms
Jun 19 11:15:12 prom2 prometheus[4004465]: ts=2022-06-19T11:15:12.660Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while"
Jun 19 11:23:00 prom2 prometheus[4004465]: ts=2022-06-19T11:23:00.246Z caller=head.go:569 level=info component=tsdb msg="WAL checkpoint loaded"
Jun 19 11:23:03 prom2 prometheus[4004465]: ts=2022-06-19T11:23:03.312Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=316598 maxSegment=316642
...
Jun 19 11:24:38 prom2 prometheus[4004465]: ts=2022-06-19T11:24:38.151Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=316642 maxSegment=316642
Jun 19 11:24:38 prom2 prometheus[4004465]: ts=2022-06-19T11:24:38.151Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=7m47.585735244s wal_replay_duration=1m37.905557061s total_replay_duration=9m25.823117397s
```
```
Jun 19 11:32:19 prom1 prometheus[4185870]: ts=2022-06-19T11:32:19.315Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655625600004 maxt=1655632800000 ulid=01G5XSX5W8E25M5E362XE342QH
Jun 19 11:32:52 prom1 prometheus[4185870]: ts=2022-06-19T11:32:52.522Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
Jun 19 11:32:52 prom1 prometheus[4185870]: ts=2022-06-19T11:32:52.963Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=441.278266ms
Jun 19 11:32:52 prom1 prometheus[4185870]: ts=2022-06-19T11:32:52.963Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while"
Jun 19 11:48:32 prom1 prometheus[4185870]: ts=2022-06-19T11:48:32.969Z caller=head.go:569 level=info component=tsdb msg="WAL checkpoint loaded"
Jun 19 11:48:36 prom1 prometheus[4185870]: ts=2022-06-19T11:48:36.494Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=65912 maxSegment=65968
...
Jun 19 11:50:36 prom1 prometheus[4185870]: ts=2022-06-19T11:50:36.181Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=65968 maxSegment=65968
Jun 19 11:50:36 prom1 prometheus[4185870]: ts=2022-06-19T11:50:36.181Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=15m40.006639977s wal_replay_duration=2m3.211154975s total_replay_duration=17m43.659164629s
```
Specifically, the huge difference is in the `checkpoint_replay_duration` - it's almost twice as long on one of the instances.
I didn't look at replay times much.
I think that the issue reported here (which #10709 fixed for me on all instances) would only affect WAL replay times if your WAL had enough samples for each time series to fill a chunk and force new chunk creation. Which should happen after 120 samples since that's the max number of samples a chunk should have.
I have scrape interval of 53s on most jobs and `--storage.tsdb.max-block-duration=30m` flag set to keep memory usage lower we don't usually manage to fill chunks before writing blocks, since 1800s / 53s = ~34, way below that 120 max. 
Just to keep this updated.
We've been running patch from #10709 for a few months now and that solved the problem for us.
Since then we've dropped `--storage.tsdb.(min|max)-block-duration` flags from all instances except those uploading to Thanos, since Thanos requires it.
I've just tried 2.37.0 build without that patch on an instance without any `--storage.tsdb.(min|max)-block-duration` flags , to verify we still hit this, and the issue is very much present. I'm hoping a some fix will eventually make it into Prometheus.

Graphs still show a spike in rule evaluation and drop in sample ingestion when new chunks are cut for all series:

![image](https://user-images.githubusercontent.com/114288/185095315-7c4127c4-edbe-40ec-848f-f3c4ebc56152.png)

![image](https://user-images.githubusercontent.com/114288/185095383-39d26a2f-de72-41b1-bbe9-40a6c8248c51.png)


@prymitive there were some improvements in the async queue for mmapping chunks recently. Can you try setting `--storage.tsdb.head-chunks-write-queue-size` to some higher value in 2.37.0 and see how it performs?
I tried running same v2.37.0 instance with `--storage.tsdb.head-chunks-write-queue-size=100000`.
I got same results as before - enabling chunk write queue makes it way way worse for me, there's a bigger drop in samples ingested and it lasts for longer, same for rule evaluation duration spike.

![image](https://user-images.githubusercontent.com/114288/185190430-475b3e4b-6c2e-47a7-bbbc-b4087ff7dcdc.png)
![image](https://user-images.githubusercontent.com/114288/185190477-7ffa1e7e-275c-4dbe-9a34-3155dcad108d.png)
![screencapture-15dm13-prometheus-access-cfdata-org-graph-2022-08-17-17_15_35](https://user-images.githubusercontent.com/114288/185190614-5a4a0caa-e898-4f12-a57d-2281ea72edc6.png)


For comparison: this is an instance with patch from #10709 scraping same set of targets in the same location.

![image](https://user-images.githubusercontent.com/114288/185190826-073bef95-577b-4671-aa26-94f9b133ce16.png)

![image](https://user-images.githubusercontent.com/114288/185190764-4c476df4-661e-4157-a882-4a644fd3d232.png)


Fix patch:
diff --git a/tsdb/chunks/chunks.go b/tsdb/chunks/chunks.go
index 6d04998e803..9817fe47a1a 100644
--- a/tsdb/chunks/chunks.go
+++ b/tsdb/chunks/chunks.go
@@ -85,13 +85,21 @@ func (p HeadChunkRef) Unpack() (HeadSeriesRef, HeadChunkID) {
 //   - less than the above, but >= memSeries.firstID, then it's
 //     memSeries.mmappedChunks[i] where i = HeadChunkID - memSeries.firstID.
 //
+// If memSeries.headChunks is non-nil it points to a *memChunk that holds the current
+// "open" (accepting appends) instance. *memChunk is a linked list and memChunk.next pointer
+// might link to the older *memChunk instance.
+// If there are multiple *memChunk instances linked to each other from memSeries.headChunks
+// they will be m-mapped as soon as possible leaving only "open" *memChunk instance.
+//
 // Example:
 // assume a memSeries.firstChunkID=7 and memSeries.mmappedChunks=[p5,p6,p7,p8,p9].
 // | HeadChunkID value | refers to ...                                                                          |
 // |-------------------|----------------------------------------------------------------------------------------|
 // |               0-6 | chunks that have been compacted to blocks, these won't return data for queries in Head |
 // |              7-11 | memSeries.mmappedChunks[i] where i is 0 to 4.                                          |
-// |                12 | memSeries.headChunk                                                                    |
+// |                12 |                                                         *memChunk{next: nil}
+// |                13 |                                         *memChunk{next: ^}
+// |                14 | memSeries.headChunks -> *memChunk{next: ^}
 type HeadChunkID uint64
 
 // BlockChunkRef refers to a chunk within a persisted block.
diff --git a/tsdb/db.go b/tsdb/db.go
index 2ca6034a035..0c69ae6e698 100644
--- a/tsdb/db.go
+++ b/tsdb/db.go
@@ -973,6 +973,8 @@ func (db *DB) run() {
 			case db.compactc <- struct{}{}:
 			default:
 			}
+			// We attempt mmapping of head chunks regularly.
+			db.head.mmapHeadChunks()
 		case <-db.compactc:
 			db.metrics.compactionsTriggered.Inc()
 
diff --git a/tsdb/head.go b/tsdb/head.go
index 499be067ada..e18bd55a5cc 100644
--- a/tsdb/head.go
+++ b/tsdb/head.go
@@ -344,6 +344,7 @@ type headMetrics struct {
 	mmapChunkCorruptionTotal  prometheus.Counter
 	snapshotReplayErrorTotal  prometheus.Counter // Will be either 0 or 1.
 	oooHistogram              prometheus.Histogram
+	mmapChunksTotal           prometheus.Counter
 }
 
 const (
@@ -468,6 +469,10 @@ func newHeadMetrics(h *Head, r prometheus.Registerer) *headMetrics {
 				60 * 60 * 12, // 12h
 			},
 		}),
+		mmapChunksTotal: prometheus.NewCounter(prometheus.CounterOpts{
+			Name: "prometheus_tsdb_mmap_chunks_total",
+			Help: "Total number of chunks that were memory-mapped.",
+		}),
 	}
 
 	if r != nil {
@@ -495,6 +500,7 @@ func newHeadMetrics(h *Head, r prometheus.Registerer) *headMetrics {
 			m.checkpointDeleteTotal,
 			m.checkpointCreationFail,
 			m.checkpointCreationTotal,
+			m.mmapChunksTotal,
 			m.mmapChunkCorruptionTotal,
 			m.snapshotReplayErrorTotal,
 			// Metrics bound to functions and not needed in tests
@@ -880,11 +886,11 @@ func (h *Head) loadMmappedChunks(refSeries map[chunks.HeadSeriesRef]*memSeries)
 			numSamples: numSamples,
 		})
 		h.updateMinMaxTime(mint, maxt)
-		if ms.headChunk != nil && maxt >= ms.headChunk.minTime {
+		if ms.headChunks != nil && maxt >= ms.headChunks.minTime {
 			// The head chunk was completed and was m-mapped after taking the snapshot.
 			// Hence remove this chunk.
 			ms.nextAt = 0
-			ms.headChunk = nil
+			ms.headChunks = nil
 			ms.app = nil
 		}
 		return nil
@@ -1574,6 +1580,10 @@ func (h *Head) Close() error {
 	defer h.closedMtx.Unlock()
 	h.closed = true
 
+	// mmap all but last chunk in case we're performing snapshot since that only
+	// takes samples from most recent head chunk.
+	h.mmapHeadChunks()
+
 	errs := tsdb_errors.NewMulti(h.chunkDiskMapper.Close())
 	if errs.Err() == nil && h.opts.EnableMemorySnapshotOnShutdown {
 		errs.Add(h.performChunkSnapshot())
@@ -1630,6 +1640,37 @@ func (h *Head) getOrCreateWithID(id chunks.HeadSeriesRef, hash uint64, lset labe
 	return s, true, nil
 }
 
+// mmapHeadChunks will iterate all memSeries stored on Head and call mmapHeadChunks() on each of them.
+//
+// There are two types of chunks that store samples for each memSeries:
+// A) Head chunk - stored on Go heap, when new samples are appended they go there.
+// B) M-mapped chunks - memory mapped chunks, kernel manages the memory for us on-demand, these chunks
+//
+//	are read-only.
+//
+// Calling mmapHeadChunks() will iterate all memSeries and m-mmap all chunks that should be m-mapped.
+// The m-mapping operation is needs to be serialised and so it goes via central lock.
+// If there are multiple concurrent memSeries that need to m-map some chunk then they can block each-other.
+//
+// To minimise the effect of locking on TSDB operations m-mapping is serialised and done away from
+// sample append path, since waiting on a lock inside an append would lock the entire memSeries for
+// (potentially) a long time, since that could eventually delay next scrape and/or cause query timeouts.
+func (h *Head) mmapHeadChunks() {
+	var count int
+	for i := 0; i < h.series.size; i++ {
+		h.series.locks[i].RLock()
+		for _, all := range h.series.hashes[i] {
+			for _, series := range all {
+				series.Lock()
+				count += series.mmapChunks(h.chunkDiskMapper)
+				series.Unlock()
+			}
+		}
+		h.series.locks[i].RUnlock()
+	}
+	h.metrics.mmapChunksTotal.Add(float64(count))
+}
+
 // seriesHashmap is a simple hashmap for memSeries by their label set. It is built
 // on top of a regular hashmap and holds a slice of series to resolve hash collisions.
 // Its methods require the hash to be submitted with it to avoid re-computations throughout
@@ -1760,7 +1801,7 @@ func (s *stripeSeries) gc(mint int64, minOOOMmapRef chunks.ChunkDiskMapperRef) (
 						minOOOTime = series.ooo.oooHeadChunk.minTime
 					}
 				}
-				if len(series.mmappedChunks) > 0 || series.headChunk != nil || series.pendingCommit ||
+				if len(series.mmappedChunks) > 0 || series.headChunks != nil || series.pendingCommit ||
 					(series.ooo != nil && (len(series.ooo.oooMmappedChunks) > 0 || series.ooo.oooHeadChunk != nil)) {
 					seriesMint := series.minTime()
 					if seriesMint < actualMint {
@@ -1915,8 +1956,11 @@ type memSeries struct {
 	//
 	// pN is the pointer to the mmappedChunk referered to by HeadChunkID=N
 	mmappedChunks []*mmappedChunk
-	headChunk     *memChunk          // Most recent chunk in memory that's still being built.
-	firstChunkID  chunks.HeadChunkID // HeadChunkID for mmappedChunks[0]
+	// Most recent chunks in memory that are still being built or waiting to be mmapped.
+	// This is a linked list, headChunks points to the most recent chunk, headChunks.next points
+	// to older chunk and so on.
+	headChunks   *memChunk
+	firstChunkID chunks.HeadChunkID // HeadChunkID for mmappedChunks[0]
 
 	ooo *memSeriesOOOFields
 
@@ -1932,7 +1976,7 @@ type memSeries struct {
 	lastFloatHistogramValue *histogram.FloatHistogram
 
 	// Current appender for the head chunk. Set when a new head chunk is cut.
-	// It is nil only if headChunk is nil. E.g. if there was an appender that created a new series, but rolled back the commit
+	// It is nil only if headChunks is nil. E.g. if there was an appender that created a new series, but rolled back the commit
 	// (the first sample would create a headChunk, hence appender, but rollback skipped it while the Append() call would create a series).
 	app chunkenc.Appender
 
@@ -1966,17 +2010,16 @@ func (s *memSeries) minTime() int64 {
 	if len(s.mmappedChunks) > 0 {
 		return s.mmappedChunks[0].minTime
 	}
-	if s.headChunk != nil {
-		return s.headChunk.minTime
+	if s.headChunks != nil {
+		return s.headChunks.oldest().minTime
 	}
 	return math.MinInt64
 }
 
 func (s *memSeries) maxTime() int64 {
 	// The highest timestamps will always be in the regular (non-OOO) chunks, even if OOO is enabled.
-	c := s.head()
-	if c != nil {
-		return c.maxTime
+	if s.headChunks != nil {
+		return s.headChunks.maxTime
 	}
 	if len(s.mmappedChunks) > 0 {
 		return s.mmappedChunks[len(s.mmappedChunks)-1].maxTime
@@ -1989,12 +2032,29 @@ func (s *memSeries) maxTime() int64 {
 // Chunk IDs remain unchanged.
 func (s *memSeries) truncateChunksBefore(mint int64, minOOOMmapRef chunks.ChunkDiskMapperRef) int {
 	var removedInOrder int
-	if s.headChunk != nil && s.headChunk.maxTime < mint {
-		// If head chunk is truncated, we can truncate all mmapped chunks.
-		removedInOrder = 1 + len(s.mmappedChunks)
-		s.firstChunkID += chunks.HeadChunkID(removedInOrder)
-		s.headChunk = nil
-		s.mmappedChunks = nil
+	if s.headChunks != nil {
+		var i int
+		var nextChk *memChunk
+		chk := s.headChunks
+		for chk != nil {
+			if chk.maxTime < mint {
+				// If any head chunk is truncated, we can truncate all mmapped chunks.
+				removedInOrder = chk.len() + len(s.mmappedChunks)
+				s.firstChunkID += chunks.HeadChunkID(removedInOrder)
+				if i == 0 {
+					// This is the first chunk on the list so we need to remove the entire list.
+					s.headChunks = nil
+				} else {
+					// This is NOT the first chunk, unlink it from parent.
+					nextChk.prev = nil
+				}
+				s.mmappedChunks = nil
+				break
+			}
+			nextChk = chk
+			chk = chk.prev
+			i++
+		}
 	}
 	if len(s.mmappedChunks) > 0 {
 		for i, c := range s.mmappedChunks {
@@ -2034,13 +2094,52 @@ func (s *memSeries) cleanupAppendIDsBelow(bound uint64) {
 	}
 }
 
-func (s *memSeries) head() *memChunk {
-	return s.headChunk
-}
-
 type memChunk struct {
 	chunk            chunkenc.Chunk
 	minTime, maxTime int64
+	prev             *memChunk // Link to the previous element on the list.
+}
+
+// len returns the length of memChunk list, including the element it was called on.
+func (mc *memChunk) len() (count int) {
+	elem := mc
+	for elem != nil {
+		count++
+		elem = elem.prev
+	}
+	return count
+}
+
+// oldest returns the oldest element on the list.
+// For single element list this will be the same memChunk oldest() was called on.
+func (mc *memChunk) oldest() (elem *memChunk) {
+	elem = mc
+	for elem.prev != nil {
+		elem = elem.prev
+	}
+	return elem
+}
+
+// atOffset returns a memChunk that's Nth element on the linked list.
+func (mc *memChunk) atOffset(offset int) (elem *memChunk) {
+	if offset == 0 {
+		return mc
+	}
+	if offset < 0 {
+		return nil
+	}
+
+	var i int
+	elem = mc
+	for i < offset {
+		i++
+		elem = elem.prev
+		if elem == nil {
+			break
+		}
+	}
+
+	return elem
 }
 
 type oooHeadChunk struct {
diff --git a/tsdb/head_append.go b/tsdb/head_append.go
index cbbb60f031e..b9818316483 100644
--- a/tsdb/head_append.go
+++ b/tsdb/head_append.go
@@ -395,7 +395,7 @@ func (a *headAppender) Append(ref storage.SeriesRef, lset labels.Labels, t int64
 func (s *memSeries) appendable(t int64, v float64, headMaxt, minValidTime, oooTimeWindow int64) (isOOO bool, oooDelta int64, err error) {
 	// Check if we can append in the in-order chunk.
 	if t >= minValidTime {
-		if s.head() == nil {
+		if s.headChunks == nil {
 			// The series has no sample and was freshly created.
 			return false, 0, nil
 		}
@@ -433,15 +433,14 @@ func (s *memSeries) appendable(t int64, v float64, headMaxt, minValidTime, oooTi
 
 // appendableHistogram checks whether the given histogram is valid for appending to the series.
 func (s *memSeries) appendableHistogram(t int64, h *histogram.Histogram) error {
-	c := s.head()
-	if c == nil {
+	if s.headChunks == nil {
 		return nil
 	}
 
-	if t > c.maxTime {
+	if t > s.headChunks.maxTime {
 		return nil
 	}
-	if t < c.maxTime {
+	if t < s.headChunks.maxTime {
 		return storage.ErrOutOfOrderSample
 	}
 
@@ -455,15 +454,14 @@ func (s *memSeries) appendableHistogram(t int64, h *histogram.Histogram) error {
 
 // appendableFloatHistogram checks whether the given float histogram is valid for appending to the series.
 func (s *memSeries) appendableFloatHistogram(t int64, fh *histogram.FloatHistogram) error {
-	c := s.head()
-	if c == nil {
+	if s.headChunks == nil {
 		return nil
 	}
 
-	if t > c.maxTime {
+	if t > s.headChunks.maxTime {
 		return nil
 	}
-	if t < c.maxTime {
+	if t < s.headChunks.maxTime {
 		return storage.ErrOutOfOrderSample
 	}
 
@@ -1200,12 +1198,11 @@ func (s *memSeries) appendHistogram(t int64, h *histogram.Histogram, appendID ui
 		return true, false
 	}
 
-	// This is a brand new chunk, switch out the head chunk (based on cutNewHeadChunk).
-	s.mmapCurrentHeadChunk(o.chunkDiskMapper)
-	s.headChunk = &memChunk{
+	s.headChunks = &memChunk{
 		chunk:   newChunk,
 		minTime: t,
 		maxTime: t,
+		prev:    s.headChunks,
 	}
 	s.nextAt = rangeForTimestamp(t, o.chunkRange)
 	return true, true
@@ -1258,12 +1255,11 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram,
 		return true, false
 	}
 
-	// This is a brand new chunk, switch out the head chunk (based on cutNewHeadChunk).
-	s.mmapCurrentHeadChunk(o.chunkDiskMapper)
-	s.headChunk = &memChunk{
+	s.headChunks = &memChunk{
 		chunk:   newChunk,
 		minTime: t,
 		maxTime: t,
+		prev:    s.headChunks,
 	}
 	s.nextAt = rangeForTimestamp(t, o.chunkRange)
 	return true, true
@@ -1273,7 +1269,7 @@ func (s *memSeries) appendFloatHistogram(t int64, fh *histogram.FloatHistogram,
 // It is unsafe to call this concurrently with s.iterator(...) without holding the series lock.
 // This should be called only when appending data.
 func (s *memSeries) appendPreprocessor(t int64, e chunkenc.Encoding, o chunkOpts) (c *memChunk, sampleInOrder, chunkCreated bool) {
-	c = s.head()
+	c = s.headChunks
 
 	if c == nil {
 		if len(s.mmappedChunks) > 0 && s.mmappedChunks[len(s.mmappedChunks)-1].maxTime >= t {
@@ -1281,7 +1277,7 @@ func (s *memSeries) appendPreprocessor(t int64, e chunkenc.Encoding, o chunkOpts
 			return c, false, false
 		}
 		// There is no head chunk in this series yet, create the first chunk for the sample.
-		c = s.cutNewHeadChunk(t, e, o.chunkDiskMapper, o.chunkRange)
+		c = s.cutNewHeadChunk(t, e, o.chunkRange)
 		chunkCreated = true
 	}
 
@@ -1293,8 +1289,9 @@ func (s *memSeries) appendPreprocessor(t int64, e chunkenc.Encoding, o chunkOpts
 	if c.chunk.Encoding() != e {
 		// The chunk encoding expected by this append is different than the head chunk's
 		// encoding. So we cut a new chunk with the expected encoding.
-		c = s.cutNewHeadChunk(t, e, o.chunkDiskMapper, o.chunkRange)
+		c = s.cutNewHeadChunk(t, e, o.chunkRange)
 		chunkCreated = true
+
 	}
 
 	numSamples := c.chunk.NumSamples()
@@ -1318,7 +1315,7 @@ func (s *memSeries) appendPreprocessor(t int64, e chunkenc.Encoding, o chunkOpts
 	// as we expect more chunks to come.
 	// Note that next chunk will have its nextAt recalculated for the new rate.
 	if t >= s.nextAt || numSamples >= o.samplesPerChunk*2 {
-		c = s.cutNewHeadChunk(t, e, o.chunkDiskMapper, o.chunkRange)
+		c = s.cutNewHeadChunk(t, e, o.chunkRange)
 		chunkCreated = true
 	}
 
@@ -1338,36 +1335,37 @@ func computeChunkEndTime(start, cur, max int64) int64 {
 	return start + (max-start)/n
 }
 
-func (s *memSeries) cutNewHeadChunk(
-	mint int64, e chunkenc.Encoding, chunkDiskMapper *chunks.ChunkDiskMapper, chunkRange int64,
-) *memChunk {
-	s.mmapCurrentHeadChunk(chunkDiskMapper)
-
-	s.headChunk = &memChunk{
+func (s *memSeries) cutNewHeadChunk(mint int64, e chunkenc.Encoding, chunkRange int64) *memChunk {
+	// When cutting a new head chunk we create a new memChunk instance with .prev
+	// pointing at the current .headChunks, so it forms a linked list.
+	// All but first headChunks list elements will be m-mapped as soon as possible
+	// so this is a single element list most of the time.
+	s.headChunks = &memChunk{
 		minTime: mint,
 		maxTime: math.MinInt64,
+		prev:    s.headChunks,
 	}
 
 	if chunkenc.IsValidEncoding(e) {
 		var err error
-		s.headChunk.chunk, err = chunkenc.NewEmptyChunk(e)
+		s.headChunks.chunk, err = chunkenc.NewEmptyChunk(e)
 		if err != nil {
 			panic(err) // This should never happen.
 		}
 	} else {
-		s.headChunk.chunk = chunkenc.NewXORChunk()
+		s.headChunks.chunk = chunkenc.NewXORChunk()
 	}
 
 	// Set upper bound on when the next chunk must be started. An earlier timestamp
 	// may be chosen dynamically at a later point.
 	s.nextAt = rangeForTimestamp(mint, chunkRange)
 
-	app, err := s.headChunk.chunk.Appender()
+	app, err := s.headChunks.chunk.Appender()
 	if err != nil {
 		panic(err)
 	}
 	s.app = app
-	return s.headChunk
+	return s.headChunks
 }
 
 // cutNewOOOHeadChunk cuts a new OOO chunk and m-maps the old chunk.
@@ -1401,19 +1399,32 @@ func (s *memSeries) mmapCurrentOOOHeadChunk(chunkDiskMapper *chunks.ChunkDiskMap
 	return chunkRef
 }
 
-func (s *memSeries) mmapCurrentHeadChunk(chunkDiskMapper *chunks.ChunkDiskMapper) {
-	if s.headChunk == nil || s.headChunk.chunk.NumSamples() == 0 {
-		// There is no head chunk, so nothing to m-map here.
+// mmapChunks will m-map all but first chunk on s.headChunks list.
+func (s *memSeries) mmapChunks(chunkDiskMapper *chunks.ChunkDiskMapper) (count int) {
+	if s.headChunks == nil || s.headChunks.prev == nil {
+		// There is none or only one head chunk, so nothing to m-map here.
 		return
 	}
 
-	chunkRef := chunkDiskMapper.WriteChunk(s.ref, s.headChunk.minTime, s.headChunk.maxTime, s.headChunk.chunk, false, handleChunkWriteError)
-	s.mmappedChunks = append(s.mmappedChunks, &mmappedChunk{
-		ref:        chunkRef,
-		numSamples: uint16(s.headChunk.chunk.NumSamples()),
-		minTime:    s.headChunk.minTime,
-		maxTime:    s.headChunk.maxTime,
-	})
+	// Write chunks starting from the oldest one and stop before we get to current s.headChunk.
+	// If we have this chain: s.headChunk{t4} -> t3 -> t2 -> t1 -> t0
+	// then we need to write chunks t0 to t3, but skip s.headChunks.
+	for i := s.headChunks.len() - 1; i > 0; i-- {
+		chk := s.headChunks.atOffset(i)
+		chunkRef := chunkDiskMapper.WriteChunk(s.ref, chk.minTime, chk.maxTime, chk.chunk, false, handleChunkWriteError)
+		s.mmappedChunks = append(s.mmappedChunks, &mmappedChunk{
+			ref:        chunkRef,
+			numSamples: uint16(chk.chunk.NumSamples()),
+			minTime:    chk.minTime,
+			maxTime:    chk.maxTime,
+		})
+		count++
+	}
+
+	// Once we've written out all chunks except s.headChunks we need to unlink these from s.headChunk.
+	s.headChunks.prev = nil
+
+	return count
 }
 
 func handleChunkWriteError(err error) {
diff --git a/tsdb/head_read.go b/tsdb/head_read.go
index b2af74ace92..f27d4ef762c 100644
--- a/tsdb/head_read.go
+++ b/tsdb/head_read.go
@@ -174,12 +174,27 @@ func (h *headIndexReader) Series(ref storage.SeriesRef, builder *labels.ScratchB
 			Ref:     chunks.ChunkRef(chunks.NewHeadChunkRef(s.ref, s.headChunkID(i))),
 		})
 	}
-	if s.headChunk != nil && s.headChunk.OverlapsClosedInterval(h.mint, h.maxt) {
-		*chks = append(*chks, chunks.Meta{
-			MinTime: s.headChunk.minTime,
-			MaxTime: math.MaxInt64, // Set the head chunks as open (being appended to).
-			Ref:     chunks.ChunkRef(chunks.NewHeadChunkRef(s.ref, s.headChunkID(len(s.mmappedChunks)))),
-		})
+
+	if s.headChunks != nil {
+		var maxTime int64
+		var i, j int
+		for i = s.headChunks.len() - 1; i >= 0; i-- {
+			chk := s.headChunks.atOffset(i)
+			if i == 0 {
+				// Set the head chunk as open (being appended to) for the first headChunk.
+				maxTime = math.MaxInt64
+			} else {
+				maxTime = chk.maxTime
+			}
+			if chk.OverlapsClosedInterval(h.mint, h.maxt) {
+				*chks = append(*chks, chunks.Meta{
+					MinTime: chk.minTime,
+					MaxTime: maxTime,
+					Ref:     chunks.ChunkRef(chunks.NewHeadChunkRef(s.ref, s.headChunkID(len(s.mmappedChunks)+j))),
+				})
+			}
+			j++
+		}
 	}
 
 	return nil
@@ -187,7 +202,7 @@ func (h *headIndexReader) Series(ref storage.SeriesRef, builder *labels.ScratchB
 
 // headChunkID returns the HeadChunkID referred to by the given position.
 // * 0 <= pos < len(s.mmappedChunks) refer to s.mmappedChunks[pos]
-// * pos == len(s.mmappedChunks) refers to s.headChunk
+// * pos >= len(s.mmappedChunks) refers to s.headChunks linked list
 func (s *memSeries) headChunkID(pos int) chunks.HeadChunkID {
 	return chunks.HeadChunkID(pos) + s.firstChunkID
 }
@@ -296,7 +311,7 @@ func (h *headChunkReader) chunk(meta chunks.Meta, copyLastChunk bool) (chunkenc.
 	}
 
 	s.Lock()
-	c, headChunk, err := s.chunk(cid, h.head.chunkDiskMapper, &h.head.memChunkPool)
+	c, headChunk, isOpen, err := s.chunk(cid, h.head.chunkDiskMapper, &h.head.memChunkPool)
 	if err != nil {
 		s.Unlock()
 		return nil, 0, err
@@ -305,6 +320,7 @@ func (h *headChunkReader) chunk(meta chunks.Meta, copyLastChunk bool) (chunkenc.
 		if !headChunk {
 			// Set this to nil so that Go GC can collect it after it has been used.
 			c.chunk = nil
+			c.prev = nil
 			h.head.memChunkPool.Put(c)
 		}
 	}()
@@ -316,14 +332,14 @@ func (h *headChunkReader) chunk(meta chunks.Meta, copyLastChunk bool) (chunkenc.
 	}
 
 	chk, maxTime := c.chunk, c.maxTime
-	if headChunk && copyLastChunk {
+	if headChunk && isOpen && copyLastChunk {
 		// The caller may ask to copy the head chunk in order to take the
 		// bytes of the chunk without causing the race between read and append.
-		b := s.headChunk.chunk.Bytes()
+		b := s.headChunks.chunk.Bytes()
 		newB := make([]byte, len(b))
 		copy(newB, b) // TODO(codesome): Use bytes.Clone() when we upgrade to Go 1.20.
 		// TODO(codesome): Put back in the pool (non-trivial).
-		chk, err = h.head.opts.ChunkPool.Get(s.headChunk.chunk.Encoding(), newB)
+		chk, err = h.head.opts.ChunkPool.Get(s.headChunks.chunk.Encoding(), newB)
 		if err != nil {
 			return nil, 0, err
 		}
@@ -341,34 +357,60 @@ func (h *headChunkReader) chunk(meta chunks.Meta, copyLastChunk bool) (chunkenc.
 // chunk returns the chunk for the HeadChunkID from memory or by m-mapping it from the disk.
 // If headChunk is false, it means that the returned *memChunk
 // (and not the chunkenc.Chunk inside it) can be garbage collected after its usage.
-func (s *memSeries) chunk(id chunks.HeadChunkID, chunkDiskMapper *chunks.ChunkDiskMapper, memChunkPool *sync.Pool) (chunk *memChunk, headChunk bool, err error) {
+// if isOpen is true, it means that the returned *memChunk is used for appends.
+func (s *memSeries) chunk(id chunks.HeadChunkID, chunkDiskMapper *chunks.ChunkDiskMapper, memChunkPool *sync.Pool) (chunk *memChunk, headChunk, isOpen bool, err error) {
 	// ix represents the index of chunk in the s.mmappedChunks slice. The chunk id's are
 	// incremented by 1 when new chunk is created, hence (id - firstChunkID) gives the slice index.
 	// The max index for the s.mmappedChunks slice can be len(s.mmappedChunks)-1, hence if the ix
-	// is len(s.mmappedChunks), it represents the next chunk, which is the head chunk.
+	// is >= len(s.mmappedChunks), it represents one of the chunks on s.headChunks linked list.
+	// The order of elemens is different for slice and linked list.
+	// For s.mmappedChunks slice newer chunks are appended to it.
+	// For s.headChunks list newer chunks are prepended to it.
+	//
+	// memSeries {
+	//   mmappedChunks: [t0, t1, t2]
+	//   headChunk:     {t5}->{t4}->{t3}
+	// }
 	ix := int(id) - int(s.firstChunkID)
-	if ix < 0 || ix > len(s.mmappedChunks) {
-		return nil, false, storage.ErrNotFound
+
+	var headChunksLen int
+	if s.headChunks != nil {
+		headChunksLen = s.headChunks.len()
 	}
 
-	if ix == len(s.mmappedChunks) {
-		if s.headChunk == nil {
-			return nil, false, errors.New("invalid head chunk")
-		}
-		return s.headChunk, true, nil
+	if ix < 0 || ix > len(s.mmappedChunks)+headChunksLen-1 {
+		return nil, false, false, storage.ErrNotFound
 	}
-	chk, err := chunkDiskMapper.Chunk(s.mmappedChunks[ix].ref)
-	if err != nil {
-		if _, ok := err.(*chunks.CorruptionErr); ok {
-			panic(err)
+
+	if ix < len(s.mmappedChunks) {
+		chk, err := chunkDiskMapper.Chunk(s.mmappedChunks[ix].ref)
+		if err != nil {
+			if _, ok := err.(*chunks.CorruptionErr); ok {
+				panic(err)
+			}
+			return nil, false, false, err
 		}
-		return nil, false, err
+		mc := memChunkPool.Get().(*memChunk)
+		mc.chunk = chk
+		mc.minTime = s.mmappedChunks[ix].minTime
+		mc.maxTime = s.mmappedChunks[ix].maxTime
+		return mc, false, false, nil
 	}
-	mc := memChunkPool.Get().(*memChunk)
-	mc.chunk = chk
-	mc.minTime = s.mmappedChunks[ix].minTime
-	mc.maxTime = s.mmappedChunks[ix].maxTime
-	return mc, false, nil
+
+	ix -= len(s.mmappedChunks)
+
+	offset := headChunksLen - ix - 1
+	// headChunks is a linked list where first element is the most recent one and the last one is the oldest.
+	// This order is reversed when compared with mmappedChunks, since mmappedChunks[0] is the oldest chunk,
+	// while headChunk.atOffset(0) would give us the most recent chunk.
+	// So when calling headChunk.atOffset() we need to reverse the value of ix.
+	elem := s.headChunks.atOffset(offset)
+	if elem == nil {
+		// This should never really happen and would mean that headChunksLen value is NOT equal
+		// to the length of the headChunks list.
+		return nil, false, false, storage.ErrNotFound
+	}
+	return elem, true, offset == 0, nil
 }
 
 // oooMergedChunk returns the requested chunk based on the given chunks.Meta
@@ -660,8 +702,21 @@ func (s *memSeries) iterator(id chunks.HeadChunkID, c chunkenc.Chunk, isoState *
 			}
 		}
 
-		if s.headChunk != nil {
-			totalSamples += s.headChunk.chunk.NumSamples()
+		ix -= len(s.mmappedChunks)
+		if s.headChunks != nil {
+			// Iterate all head chunks from the oldest to the newest.
+			headChunksLen := s.headChunks.len()
+			for j := headChunksLen - 1; j >= 0; j-- {
+				chk := s.headChunks.atOffset(j)
+				chkSamples := chk.chunk.NumSamples()
+				totalSamples += chkSamples
+				// Chunk ID is len(s.mmappedChunks) + $(headChunks list position).
+				// Where $(headChunks list position) is zero for the oldest chunk and $(s.headChunks.len() - 1)
+				// for the newest (open) chunk.
+				if headChunksLen-1-j < ix {
+					previousSamples += chkSamples
+				}
+			}
 		}
 
 		// Removing the extra transactionIDs that are relevant for samples that
diff --git a/tsdb/head_wal.go b/tsdb/head_wal.go
index 2397a9ec971..3ed95887ec4 100644
--- a/tsdb/head_wal.go
+++ b/tsdb/head_wal.go
@@ -503,7 +503,7 @@ func (h *Head) resetSeriesWithMMappedChunks(mSeries *memSeries, mmc, oooMmc []*m
 
 	// Any samples replayed till now would already be compacted. Resetting the head chunk.
 	mSeries.nextAt = 0
-	mSeries.headChunk = nil
+	mSeries.headChunks = nil
 	mSeries.app = nil
 	return
 }
@@ -595,6 +595,7 @@ func (wp *walSubsetProcessor) processWALSamples(h *Head, mmappedChunks, oooMmapp
 			if _, chunkCreated := ms.append(s.T, s.V, 0, appendChunkOpts); chunkCreated {
 				h.metrics.chunksCreated.Inc()
 				h.metrics.chunks.Inc()
+				_ = ms.mmapChunks(h.chunkDiskMapper)
 			}
 			if s.T > maxt {
 				maxt = s.T
@@ -960,15 +961,15 @@ func (s *memSeries) encodeToSnapshotRecord(b []byte) []byte {
 	buf.PutBE64int64(0) // Backwards-compatibility; was chunkRange but now unused.
 
 	s.Lock()
-	if s.headChunk == nil {
+	if s.headChunks == nil {
 		buf.PutUvarint(0)
 	} else {
-		enc := s.headChunk.chunk.Encoding()
+		enc := s.headChunks.chunk.Encoding()
 		buf.PutUvarint(1)
-		buf.PutBE64int64(s.headChunk.minTime)
-		buf.PutBE64int64(s.headChunk.maxTime)
+		buf.PutBE64int64(s.headChunks.minTime)
+		buf.PutBE64int64(s.headChunks.maxTime)
 		buf.PutByte(byte(enc))
-		buf.PutUvarintBytes(s.headChunk.chunk.Bytes())
+		buf.PutUvarintBytes(s.headChunks.chunk.Bytes())
 
 		switch enc {
 		case chunkenc.EncXOR:
@@ -1414,12 +1415,12 @@ func (h *Head) loadChunkSnapshot() (int, int, map[chunks.HeadSeriesRef]*memSerie
 					continue
 				}
 				series.nextAt = csr.mc.maxTime // This will create a new chunk on append.
-				series.headChunk = csr.mc
+				series.headChunks = csr.mc
 				series.lastValue = csr.lastValue
 				series.lastHistogramValue = csr.lastHistogramValue
 				series.lastFloatHistogramValue = csr.lastFloatHistogramValue
 
-				app, err := series.headChunk.chunk.Appender()
+				app, err := series.headChunks.chunk.Appender()
 				if err != nil {
 					errChan <- err
 					return


Test patch:
diff --git a/tsdb/db_test.go b/tsdb/db_test.go
index 772fcf9d125..0eb361db7e6 100644
--- a/tsdb/db_test.go
+++ b/tsdb/db_test.go
@@ -1535,6 +1535,7 @@ func TestSizeRetention(t *testing.T) {
 		}
 	}
 	require.NoError(t, headApp.Commit())
+	db.Head().mmapHeadChunks()
 
 	require.Eventually(t, func() bool {
 		return db.Head().chunkDiskMapper.IsQueueEmpty()
@@ -6049,12 +6050,14 @@ func TestDiskFillingUpAfterDisablingOOO(t *testing.T) {
 
 	// Check that m-map files gets deleted properly after compactions.
 
+	db.head.mmapHeadChunks()
 	checkMmapFileContents([]string{"000001", "000002"}, nil)
 	require.NoError(t, db.Compact())
 	checkMmapFileContents([]string{"000002"}, []string{"000001"})
 	require.Nil(t, ms.ooo, "OOO mmap chunk was not compacted")
 
 	addSamples(501, 650)
+	db.head.mmapHeadChunks()
 	checkMmapFileContents([]string{"000002", "000003"}, []string{"000001"})
 	require.NoError(t, db.Compact())
 	checkMmapFileContents(nil, []string{"000001", "000002", "000003"})
diff --git a/tsdb/head_read_test.go b/tsdb/head_read_test.go
index 2712bcd1a15..ad0a59d34e8 100644
--- a/tsdb/head_read_test.go
+++ b/tsdb/head_read_test.go
@@ -15,11 +15,14 @@ package tsdb
 
 import (
 	"fmt"
+	"sync"
 	"testing"
 
 	"github.com/stretchr/testify/require"
 
+	"github.com/prometheus/prometheus/model/labels"
 	"github.com/prometheus/prometheus/tsdb/chunkenc"
+	"github.com/prometheus/prometheus/tsdb/chunks"
 )
 
 func TestBoundedChunk(t *testing.T) {
@@ -176,3 +179,387 @@ func newTestChunk(numSamples int) chunkenc.Chunk {
 	}
 	return xor
 }
+
+// TestMemSeries_chunk runs a series of tests on memSeries.chunk() calls.
+// It will simulate various conditions to ensure all code paths in that function are covered.
+func TestMemSeries_chunk(t *testing.T) {
+	const chunkRange int64 = 100
+	const chunkStep int64 = 5
+
+	appendSamples := func(t *testing.T, s *memSeries, start, end int64, cdm *chunks.ChunkDiskMapper) {
+		for i := start; i < end; i += chunkStep {
+			ok, _ := s.append(i, float64(i), 0, chunkOpts{
+				chunkDiskMapper: cdm,
+				chunkRange:      chunkRange,
+				samplesPerChunk: DefaultSamplesPerChunk,
+			})
+			require.True(t, ok, "sample append failed")
+		}
+	}
+
+	type setupFn func(*testing.T, *memSeries, *chunks.ChunkDiskMapper)
+
+	type callOutput uint8
+	const (
+		outOpenHeadChunk   callOutput = iota // memSeries.chunk() call returned memSeries.headChunks with headChunk=true & isOpen=true
+		outClosedHeadChunk                   // memSeries.chunk() call returned memSeries.headChunks with headChunk=true & isOpen=false
+		outMmappedChunk                      // memSeries.chunk() call returned a chunk from memSeries.mmappedChunks with headChunk=false
+		outErr                               // memSeries.chunk() call returned an error
+	)
+
+	tests := []struct {
+		name     string
+		setup    setupFn            // optional function called just before the test memSeries.chunk() call
+		inputID  chunks.HeadChunkID // requested chunk id for memSeries.chunk() call
+		expected callOutput
+	}{
+		{
+			name:     "call ix=0 on empty memSeries",
+			inputID:  0,
+			expected: outErr,
+		},
+		{
+			name:     "call ix=1 on empty memSeries",
+			inputID:  1,
+			expected: outErr,
+		},
+		{
+			name: "firstChunkID > ix",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange, cdm)
+				require.Len(t, s.mmappedChunks, 0, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, int64(0), s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, chunkRange-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+				s.firstChunkID = 5
+			},
+			inputID:  1,
+			expected: outErr,
+		},
+		{
+			name: "call ix=0 on memSeries with no mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange, cdm)
+				require.Len(t, s.mmappedChunks, 0, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, int64(0), s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, chunkRange-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  0,
+			expected: outOpenHeadChunk,
+		},
+		{
+			name: "call ix=1 on memSeries with no mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange, cdm)
+				require.Len(t, s.mmappedChunks, 0, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, int64(0), s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, chunkRange-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  1,
+			expected: outErr,
+		},
+		{
+			name: "call ix=10 on memSeries with no mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange, cdm)
+				require.Len(t, s.mmappedChunks, 0, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, int64(0), s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, chunkRange-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  10,
+			expected: outErr,
+		},
+		{
+			name: "call ix=0 on memSeries with 3 mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*4)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  0,
+			expected: outMmappedChunk,
+		},
+		{
+			name: "call ix=1 on memSeries with 3 mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*4)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  1,
+			expected: outMmappedChunk,
+		},
+		{
+			name: "call ix=3 on memSeries with 3 mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*4)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  3,
+			expected: outOpenHeadChunk,
+		},
+		{
+			name: "call ix=0 on memSeries with 3 mmapped chunks and no headChunk",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*4)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+				s.headChunks = nil
+			},
+			inputID:  0,
+			expected: outMmappedChunk,
+		},
+		{
+			name: "call ix=2 on memSeries with 3 mmapped chunks and no headChunk",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*4)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+				s.headChunks = nil
+			},
+			inputID:  2,
+			expected: outMmappedChunk,
+		},
+		{
+			name: "call ix=3 on memSeries with 3 mmapped chunks and no headChunk",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*4)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+				s.headChunks = nil
+			},
+			inputID:  3,
+			expected: outErr,
+		},
+		{
+			name: "call ix=1 on memSeries with 3 mmapped chunks and closed ChunkDiskMapper",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*4)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+				cdm.Close()
+			},
+			inputID:  1,
+			expected: outErr,
+		},
+		{
+			name: "call ix=3 on memSeries with 3 mmapped chunks and closed ChunkDiskMapper",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*4)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+				cdm.Close()
+			},
+			inputID:  3,
+			expected: outOpenHeadChunk,
+		},
+		{
+			name: "call ix=0 on memSeries with 3 head chunks and no mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*3, cdm)
+				require.Len(t, s.mmappedChunks, 0, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 3, "wrong number of headChunks")
+				require.Equal(t, int64(0), s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*3)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  0,
+			expected: outClosedHeadChunk,
+		},
+		{
+			name: "call ix=1 on memSeries with 3 head chunks and no mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*3, cdm)
+				require.Len(t, s.mmappedChunks, 0, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 3, "wrong number of headChunks")
+				require.Equal(t, int64(0), s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*3)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  1,
+			expected: outClosedHeadChunk,
+		},
+		{
+			name: "call ix=10 on memSeries with 3 head chunks and no mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*3, cdm)
+				require.Len(t, s.mmappedChunks, 0, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 3, "wrong number of headChunks")
+				require.Equal(t, int64(0), s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*3)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  10,
+			expected: outErr,
+		},
+		{
+			name: "call ix=0 on memSeries with 3 head chunks and 3 mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+
+				appendSamples(t, s, chunkRange*4, chunkRange*6, cdm)
+				require.Equal(t, s.headChunks.len(), 3, "wrong number of headChunks")
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*6)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  0,
+			expected: outMmappedChunk,
+		},
+		{
+			name: "call ix=2 on memSeries with 3 head chunks and 3 mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+
+				appendSamples(t, s, chunkRange*4, chunkRange*6, cdm)
+				require.Equal(t, s.headChunks.len(), 3, "wrong number of headChunks")
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*6)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  2,
+			expected: outMmappedChunk,
+		},
+		{
+			name: "call ix=3 on memSeries with 3 head chunks and 3 mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+
+				appendSamples(t, s, chunkRange*4, chunkRange*6, cdm)
+				require.Equal(t, s.headChunks.len(), 3, "wrong number of headChunks")
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*6)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  3,
+			expected: outClosedHeadChunk,
+		},
+		{
+			name: "call ix=5 on memSeries with 3 head chunks and 3 mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+
+				appendSamples(t, s, chunkRange*4, chunkRange*6, cdm)
+				require.Equal(t, s.headChunks.len(), 3, "wrong number of headChunks")
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*6)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  5,
+			expected: outOpenHeadChunk,
+		},
+		{
+			name: "call ix=6 on memSeries with 3 head chunks and 3 mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+
+				appendSamples(t, s, chunkRange*4, chunkRange*6, cdm)
+				require.Equal(t, s.headChunks.len(), 3, "wrong number of headChunks")
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*6)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  6,
+			expected: outErr,
+		},
+
+		{
+			name: "call ix=10 on memSeries with 3 head chunks and 3 mmapped chunks",
+			setup: func(t *testing.T, s *memSeries, cdm *chunks.ChunkDiskMapper) {
+				appendSamples(t, s, 0, chunkRange*4, cdm)
+				s.mmapChunks(cdm)
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, s.headChunks.len(), 1, "wrong number of headChunks")
+
+				appendSamples(t, s, chunkRange*4, chunkRange*6, cdm)
+				require.Equal(t, s.headChunks.len(), 3, "wrong number of headChunks")
+				require.Len(t, s.mmappedChunks, 3, "wrong number of mmappedChunks")
+				require.Equal(t, chunkRange*3, s.headChunks.oldest().minTime, "wrong minTime on last headChunks element")
+				require.Equal(t, (chunkRange*6)-chunkStep, s.headChunks.maxTime, "wrong maxTime on first headChunks element")
+			},
+			inputID:  10,
+			expected: outErr,
+		},
+	}
+
+	memChunkPool := &sync.Pool{
+		New: func() interface{} {
+			return &memChunk{}
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			dir := t.TempDir()
+			chunkDiskMapper, err := chunks.NewChunkDiskMapper(nil, dir, chunkenc.NewPool(), chunks.DefaultWriteBufferSize, chunks.DefaultWriteQueueSize)
+			require.NoError(t, err)
+			defer func() {
+				require.NoError(t, chunkDiskMapper.Close())
+			}()
+
+			series := newMemSeries(labels.EmptyLabels(), 1, true)
+
+			if tc.setup != nil {
+				tc.setup(t, series, chunkDiskMapper)
+			}
+
+			chk, headChunk, isOpen, err := series.chunk(tc.inputID, chunkDiskMapper, memChunkPool)
+			switch tc.expected {
+			case outOpenHeadChunk:
+				require.NoError(t, err, "unexpected error")
+				require.True(t, headChunk, "expected a chunk with headChunk=true but got headChunk=%v", headChunk)
+				require.True(t, isOpen, "expected a chunk with isOpen=true but got isOpen=%v", isOpen)
+			case outClosedHeadChunk:
+				require.NoError(t, err, "unexpected error")
+				require.True(t, headChunk, "expected a chunk with headChunk=true but got headChunk=%v", headChunk)
+				require.False(t, isOpen, "expected a chunk with isOpen=false but got isOpen=%v", isOpen)
+			case outMmappedChunk:
+				require.NoError(t, err, "unexpected error")
+				require.False(t, headChunk, "expected a chunk with headChunk=false but got gc=%v", headChunk)
+			case outErr:
+				require.Nil(t, chk, "got a non-nil chunk reference returned with an error")
+				require.Error(t, err)
+			}
+		})
+	}
+}
diff --git a/tsdb/head_test.go b/tsdb/head_test.go
index 0a6eef66cab..9b49aca034f 100644
--- a/tsdb/head_test.go
+++ b/tsdb/head_test.go
@@ -292,7 +292,7 @@ func BenchmarkLoadWAL(b *testing.B) {
 							// Create one mmapped chunk per series, with one sample at the given time.
 							s := newMemSeries(labels.Labels{}, chunks.HeadSeriesRef(k)*101, defaultIsolationDisabled)
 							s.append(c.mmappedChunkT, 42, 0, cOpts)
-							s.mmapCurrentHeadChunk(chunkDiskMapper)
+							s.mmapChunks(chunkDiskMapper)
 						}
 						require.NoError(b, chunkDiskMapper.Close())
 					}
@@ -587,15 +587,15 @@ func TestHead_ReadWAL(t *testing.T) {
 				return x
 			}
 
-			c, _, err := s10.chunk(0, head.chunkDiskMapper, &head.memChunkPool)
+			c, _, _, err := s10.chunk(0, head.chunkDiskMapper, &head.memChunkPool)
 			require.NoError(t, err)
 			require.Equal(t, []sample{{100, 2, nil, nil}, {101, 5, nil, nil}}, expandChunk(c.chunk.Iterator(nil)))
-			c, _, err = s50.chunk(0, head.chunkDiskMapper, &head.memChunkPool)
+			c, _, _, err = s50.chunk(0, head.chunkDiskMapper, &head.memChunkPool)
 			require.NoError(t, err)
 			require.Equal(t, []sample{{101, 6, nil, nil}}, expandChunk(c.chunk.Iterator(nil)))
 			// The samples before the new series record should be discarded since a duplicate record
 			// is only possible when old samples were compacted.
-			c, _, err = s100.chunk(0, head.chunkDiskMapper, &head.memChunkPool)
+			c, _, _, err = s100.chunk(0, head.chunkDiskMapper, &head.memChunkPool)
 			require.NoError(t, err)
 			require.Equal(t, []sample{{101, 7, nil, nil}}, expandChunk(c.chunk.Iterator(nil)))
 
@@ -822,30 +822,200 @@ func TestMemSeries_truncateChunks(t *testing.T) {
 		ok, _ := s.append(int64(i), float64(i), 0, cOpts)
 		require.True(t, ok, "sample append failed")
 	}
+	s.mmapChunks(chunkDiskMapper)
 
 	// Check that truncate removes half of the chunks and afterwards
 	// that the ID of the last chunk still gives us the same chunk afterwards.
 	countBefore := len(s.mmappedChunks) + 1 // +1 for the head chunk.
 	lastID := s.headChunkID(countBefore - 1)
-	lastChunk, _, err := s.chunk(lastID, chunkDiskMapper, &memChunkPool)
+	lastChunk, _, _, err := s.chunk(lastID, chunkDiskMapper, &memChunkPool)
 	require.NoError(t, err)
 	require.NotNil(t, lastChunk)
 
-	chk, _, err := s.chunk(0, chunkDiskMapper, &memChunkPool)
+	chk, _, _, err := s.chunk(0, chunkDiskMapper, &memChunkPool)
 	require.NotNil(t, chk)
 	require.NoError(t, err)
 
 	s.truncateChunksBefore(2000, 0)
 
 	require.Equal(t, int64(2000), s.mmappedChunks[0].minTime)
-	_, _, err = s.chunk(0, chunkDiskMapper, &memChunkPool)
+	_, _, _, err = s.chunk(0, chunkDiskMapper, &memChunkPool)
 	require.Equal(t, storage.ErrNotFound, err, "first chunks not gone")
 	require.Equal(t, countBefore/2, len(s.mmappedChunks)+1) // +1 for the head chunk.
-	chk, _, err = s.chunk(lastID, chunkDiskMapper, &memChunkPool)
+	chk, _, _, err = s.chunk(lastID, chunkDiskMapper, &memChunkPool)
 	require.NoError(t, err)
 	require.Equal(t, lastChunk, chk)
 }
 
+func TestMemSeries_truncateChunks_scenarios(t *testing.T) {
+	const chunkRange = 100
+	const chunkStep = 5
+
+	tests := []struct {
+		name                 string
+		headChunks           int                // the number of head chubks to create on memSeries by appending enough samples
+		mmappedChunks        int                // the number of mmapped chunks to create on memSeries by appending enough samples
+		truncateBefore       int64              // the mint to pass to truncateChunksBefore()
+		expectedTruncated    int                // the number of chunks that we're expecting be truncated and returned by truncateChunksBefore()
+		expectedHead         int                // the expected number of head chunks after truncation
+		expectedMmap         int                // the expected number of mmapped chunks after truncation
+		expectedFirstChunkID chunks.HeadChunkID // the expected series.firstChunkID after truncation
+	}{
+		{
+			name:           "empty memSeries",
+			truncateBefore: chunkRange * 10,
+		},
+		{
+			name:         "single head chunk, not truncated",
+			headChunks:   1,
+			expectedHead: 1,
+		},
+		{
+			name:                 "single head chunk, truncated",
+			headChunks:           1,
+			truncateBefore:       chunkRange,
+			expectedTruncated:    1,
+			expectedHead:         0,
+			expectedFirstChunkID: 1,
+		},
+		{
+			name:         "2 head chunks, not truncated",
+			headChunks:   2,
+			expectedHead: 2,
+		},
+		{
+			name:                 "2 head chunks, first truncated",
+			headChunks:           2,
+			truncateBefore:       chunkRange,
+			expectedTruncated:    1,
+			expectedHead:         1,
+			expectedFirstChunkID: 1,
+		},
+		{
+			name:                 "2 head chunks, everything truncated",
+			headChunks:           2,
+			truncateBefore:       chunkRange * 2,
+			expectedTruncated:    2,
+			expectedHead:         0,
+			expectedFirstChunkID: 2,
+		},
+		{
+			name:                 "no head chunks, 3 mmap chunks, second mmap truncated",
+			headChunks:           0,
+			mmappedChunks:        3,
+			truncateBefore:       chunkRange * 2,
+			expectedTruncated:    2,
+			expectedHead:         0,
+			expectedMmap:         1,
+			expectedFirstChunkID: 2,
+		},
+		{
+			name:          "single head chunk, single mmap chunk, not truncated",
+			headChunks:    1,
+			mmappedChunks: 1,
+			expectedHead:  1,
+			expectedMmap:  1,
+		},
+		{
+			name:                 "single head chunk, single mmap chunk, mmap truncated",
+			headChunks:           1,
+			mmappedChunks:        1,
+			truncateBefore:       chunkRange,
+			expectedTruncated:    1,
+			expectedHead:         1,
+			expectedMmap:         0,
+			expectedFirstChunkID: 1,
+		},
+		{
+			name:                 "5 head chunk, 5 mmap chunk, third head truncated",
+			headChunks:           5,
+			mmappedChunks:        5,
+			truncateBefore:       chunkRange * 7,
+			expectedTruncated:    7,
+			expectedHead:         3,
+			expectedMmap:         0,
+			expectedFirstChunkID: 7,
+		},
+		{
+			name:                 "2 head chunks, 3 mmap chunks, second mmap truncated",
+			headChunks:           2,
+			mmappedChunks:        3,
+			truncateBefore:       chunkRange * 2,
+			expectedTruncated:    2,
+			expectedHead:         2,
+			expectedMmap:         1,
+			expectedFirstChunkID: 2,
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			dir := t.TempDir()
+			chunkDiskMapper, err := chunks.NewChunkDiskMapper(nil, dir, chunkenc.NewPool(), chunks.DefaultWriteBufferSize, chunks.DefaultWriteQueueSize)
+			require.NoError(t, err)
+			defer func() {
+				require.NoError(t, chunkDiskMapper.Close())
+			}()
+
+			series := newMemSeries(labels.EmptyLabels(), 1, true)
+
+			cOpts := chunkOpts{
+				chunkDiskMapper: chunkDiskMapper,
+				chunkRange:      chunkRange,
+				samplesPerChunk: DefaultSamplesPerChunk,
+			}
+
+			var headStart int
+			if tc.mmappedChunks > 0 {
+				headStart = (tc.mmappedChunks + 1) * chunkRange
+				for i := 0; i < (tc.mmappedChunks+1)*chunkRange; i += chunkStep {
+					ok, _ := series.append(int64(i), float64(i), 0, cOpts)
+					require.True(t, ok, "sample append failed")
+				}
+				series.mmapChunks(chunkDiskMapper)
+			}
+
+			if tc.headChunks == 0 {
+				series.headChunks = nil
+			} else {
+				for i := headStart; i < chunkRange*(tc.mmappedChunks+tc.headChunks); i += chunkStep {
+					ok, _ := series.append(int64(i), float64(i), 0, cOpts)
+					require.True(t, ok, "sample append failed: %d", i)
+				}
+			}
+
+			if tc.headChunks > 0 {
+				require.NotNil(t, series.headChunks, "head chunk is missing")
+				require.Equal(t, tc.headChunks, series.headChunks.len(), "wrong number of head chunks")
+			} else {
+				require.Nil(t, series.headChunks, "head chunk is present")
+			}
+			require.Equal(t, tc.mmappedChunks, len(series.mmappedChunks), "wrong number of mmapped chunks")
+
+			truncated := series.truncateChunksBefore(tc.truncateBefore, 0)
+			require.Equal(t, tc.expectedTruncated, truncated, "wrong number of truncated chunks returned")
+
+			require.Equal(t, tc.expectedMmap, len(series.mmappedChunks), "wrong number of mmappedChunks after truncation")
+
+			if tc.expectedHead > 0 {
+				require.NotNil(t, series.headChunks, "headChunks should is nil after truncation")
+				require.Equal(t, tc.expectedHead, series.headChunks.len(), "wrong number of head chunks after truncation")
+				require.Nil(t, series.headChunks.oldest().prev, "last head chunk cannot have any next chunk set")
+			} else {
+				require.Nil(t, series.headChunks, "headChunks should is non-nil after truncation")
+			}
+
+			if series.headChunks != nil || len(series.mmappedChunks) > 0 {
+				require.GreaterOrEqual(t, series.maxTime(), tc.truncateBefore, "wrong value of series.maxTime() after truncation")
+			} else {
+				require.Equal(t, int64(math.MinInt64), series.maxTime(), "wrong value of series.maxTime() after truncation")
+			}
+
+			require.Equal(t, tc.expectedFirstChunkID, series.firstChunkID, "wrong firstChunkID after truncation")
+		})
+	}
+}
+
 func TestHeadDeleteSeriesWithoutSamples(t *testing.T) {
 	for _, compress := range []wlog.CompressionType{wlog.CompressionNone, wlog.CompressionSnappy, wlog.CompressionZstd} {
 		t.Run(fmt.Sprintf("compress=%s", compress), func(t *testing.T) {
@@ -1363,6 +1533,7 @@ func TestMemSeries_append(t *testing.T) {
 	ok, chunkCreated = s.append(999, 2, 0, cOpts)
 	require.True(t, ok, "append failed")
 	require.False(t, chunkCreated, "second sample should use same chunk")
+	s.mmapChunks(chunkDiskMapper)
 
 	ok, chunkCreated = s.append(1000, 3, 0, cOpts)
 	require.True(t, ok, "append failed")
@@ -1372,11 +1543,12 @@ func TestMemSeries_append(t *testing.T) {
 	require.True(t, ok, "append failed")
 	require.False(t, chunkCreated, "second sample should use same chunk")
 
+	s.mmapChunks(chunkDiskMapper)
 	require.Equal(t, 1, len(s.mmappedChunks), "there should be only 1 mmapped chunk")
 	require.Equal(t, int64(998), s.mmappedChunks[0].minTime, "wrong chunk range")
 	require.Equal(t, int64(999), s.mmappedChunks[0].maxTime, "wrong chunk range")
-	require.Equal(t, int64(1000), s.headChunk.minTime, "wrong chunk range")
-	require.Equal(t, int64(1001), s.headChunk.maxTime, "wrong chunk range")
+	require.Equal(t, int64(1000), s.headChunks.minTime, "wrong chunk range")
+	require.Equal(t, int64(1001), s.headChunks.maxTime, "wrong chunk range")
 
 	// Fill the range [1000,2000) with many samples. Intermediate chunks should be cut
 	// at approximately 120 samples per chunk.
@@ -1384,6 +1556,7 @@ func TestMemSeries_append(t *testing.T) {
 		ok, _ := s.append(1001+int64(i), float64(i), 0, cOpts)
 		require.True(t, ok, "append failed")
 	}
+	s.mmapChunks(chunkDiskMapper)
 
 	require.Greater(t, len(s.mmappedChunks)+1, 7, "expected intermediate chunks")
 
@@ -1437,21 +1610,23 @@ func TestMemSeries_appendHistogram(t *testing.T) {
 	require.True(t, ok, "append failed")
 	require.False(t, chunkCreated, "second sample should use same chunk")
 
+	s.mmapChunks(chunkDiskMapper)
 	require.Equal(t, 1, len(s.mmappedChunks), "there should be only 1 mmapped chunk")
 	require.Equal(t, int64(998), s.mmappedChunks[0].minTime, "wrong chunk range")
 	require.Equal(t, int64(999), s.mmappedChunks[0].maxTime, "wrong chunk range")
-	require.Equal(t, int64(1000), s.headChunk.minTime, "wrong chunk range")
-	require.Equal(t, int64(1001), s.headChunk.maxTime, "wrong chunk range")
+	require.Equal(t, int64(1000), s.headChunks.minTime, "wrong chunk range")
+	require.Equal(t, int64(1001), s.headChunks.maxTime, "wrong chunk range")
 
 	ok, chunkCreated = s.appendHistogram(1002, histogramWithOneMoreBucket, 0, cOpts)
 	require.True(t, ok, "append failed")
 	require.False(t, chunkCreated, "third sample should trigger a re-encoded chunk")
 
+	s.mmapChunks(chunkDiskMapper)
 	require.Equal(t, 1, len(s.mmappedChunks), "there should be only 1 mmapped chunk")
 	require.Equal(t, int64(998), s.mmappedChunks[0].minTime, "wrong chunk range")
 	require.Equal(t, int64(999), s.mmappedChunks[0].maxTime, "wrong chunk range")
-	require.Equal(t, int64(1000), s.headChunk.minTime, "wrong chunk range")
-	require.Equal(t, int64(1002), s.headChunk.maxTime, "wrong chunk range")
+	require.Equal(t, int64(1000), s.headChunks.minTime, "wrong chunk range")
+	require.Equal(t, int64(1002), s.headChunks.maxTime, "wrong chunk range")
 }
 
 func TestMemSeries_append_atVariableRate(t *testing.T) {
@@ -1495,6 +1670,7 @@ func TestMemSeries_append_atVariableRate(t *testing.T) {
 	require.True(t, ok, "new chunk sample was not appended")
 	require.True(t, chunkCreated, "sample at block duration timestamp should create a new chunk")
 
+	s.mmapChunks(chunkDiskMapper)
 	var totalSamplesInChunks int
 	for i, c := range s.mmappedChunks {
 		totalSamplesInChunks += int(c.numSamples)
@@ -1841,6 +2017,7 @@ func TestHeadReadWriterRepair(t *testing.T) {
 			require.True(t, ok, "series append failed")
 			require.False(t, chunkCreated, "chunk was created")
 			h.chunkDiskMapper.CutNewFile()
+			s.mmapChunks(h.chunkDiskMapper)
 		}
 		require.NoError(t, h.Close())
 
@@ -1985,6 +2162,7 @@ func TestMemSeriesIsolation(t *testing.T) {
 			_, err := app.Append(0, labels.FromStrings("foo", "bar"), int64(i), float64(i))
 			require.NoError(t, err)
 			require.NoError(t, app.Commit())
+			h.mmapHeadChunks()
 		}
 		return i
 	}
@@ -2666,7 +2844,7 @@ func TestIteratorSeekIntoBuffer(t *testing.T) {
 		require.True(t, ok, "sample append failed")
 	}
 
-	c, _, err := s.chunk(0, chunkDiskMapper, &sync.Pool{
+	c, _, _, err := s.chunk(0, chunkDiskMapper, &sync.Pool{
 		New: func() interface{} {
 			return &memChunk{}
 		},
@@ -3092,6 +3270,7 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) {
 			}
 		}
 		require.NoError(t, app.Commit())
+		head.mmapHeadChunks()
 	}
 
 	// There should be 11 mmap chunks in s1.
@@ -3103,7 +3282,7 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) {
 		cpy := *mmap
 		expMmapChunks = append(expMmapChunks, &cpy)
 	}
-	expHeadChunkSamples := ms.headChunk.chunk.NumSamples()
+	expHeadChunkSamples := ms.headChunks.chunk.NumSamples()
 	require.Greater(t, expHeadChunkSamples, 0)
 
 	// Series with mix of histograms and float.
@@ -3199,7 +3378,7 @@ func TestHistogramInWALAndMmapChunk(t *testing.T) {
 	// Checking contents of s1.
 	ms = head.series.getByHash(s1.Hash(), s1)
 	require.Equal(t, expMmapChunks, ms.mmappedChunks)
-	require.Equal(t, expHeadChunkSamples, ms.headChunk.chunk.NumSamples())
+	require.Equal(t, expHeadChunkSamples, ms.headChunks.chunk.NumSamples())
 
 	testQuery := func() {
 		q, err := NewBlockQuerier(head, head.MinTime(), head.MaxTime())
@@ -3738,6 +3917,8 @@ func testHistogramStaleSampleHelper(t *testing.T, floatHistogram bool) {
 	// Only 1 chunk in the memory, no m-mapped chunk.
 	s := head.series.getByHash(l.Hash(), l)
 	require.NotNil(t, s)
+	require.NotNil(t, s.headChunks)
+	require.Equal(t, s.headChunks.len(), 1)
 	require.Equal(t, 0, len(s.mmappedChunks))
 	testQuery(1)
 
@@ -3766,10 +3947,13 @@ func testHistogramStaleSampleHelper(t *testing.T, floatHistogram bool) {
 		expHistograms = append(expHistograms, timedHistogram{t: 100*int64(len(expHistograms)) + 1, h: &histogram.Histogram{Sum: math.Float64frombits(value.StaleNaN)}})
 	}
 	require.NoError(t, app.Commit())
+	head.mmapHeadChunks()
 
 	// Total 2 chunks, 1 m-mapped.
 	s = head.series.getByHash(l.Hash(), l)
 	require.NotNil(t, s)
+	require.NotNil(t, s.headChunks)
+	require.Equal(t, s.headChunks.len(), 1)
 	require.Equal(t, 1, len(s.mmappedChunks))
 	testQuery(2)
 }
@@ -3804,6 +3988,7 @@ func TestHistogramCounterResetHeader(t *testing.T) {
 
 				ms, _, err := head.getOrCreate(l.Hash(), l)
 				require.NoError(t, err)
+				ms.mmapChunks(head.chunkDiskMapper)
 				require.Len(t, ms.mmappedChunks, len(expHeaders)-1) // One is the head chunk.
 
 				for i, mmapChunk := range ms.mmappedChunks {
@@ -3816,9 +4001,9 @@ func TestHistogramCounterResetHeader(t *testing.T) {
 					}
 				}
 				if floatHisto {
-					require.Equal(t, expHeaders[len(expHeaders)-1], ms.headChunk.chunk.(*chunkenc.FloatHistogramChunk).GetCounterResetHeader())
+					require.Equal(t, expHeaders[len(expHeaders)-1], ms.headChunks.chunk.(*chunkenc.FloatHistogramChunk).GetCounterResetHeader())
 				} else {
-					require.Equal(t, expHeaders[len(expHeaders)-1], ms.headChunk.chunk.(*chunkenc.HistogramChunk).GetCounterResetHeader())
+					require.Equal(t, expHeaders[len(expHeaders)-1], ms.headChunks.chunk.(*chunkenc.HistogramChunk).GetCounterResetHeader())
 				}
 			}
 
@@ -3909,7 +4094,7 @@ func TestAppendingDifferentEncodingToSameSeries(t *testing.T) {
 		require.NoError(t, err)
 		require.False(t, created)
 		require.NotNil(t, ms)
-		require.Len(t, ms.mmappedChunks, count-1) // One will be the head chunk.
+		require.Equal(t, count, ms.headChunks.len())
 	}
 
 	appends := []struct {
@@ -4350,6 +4535,7 @@ func TestHeadInit_DiscardChunksWithUnsupportedEncoding(t *testing.T) {
 	require.False(t, created, "should already exist")
 	require.NotNil(t, series, "should return the series we created above")
 
+	series.mmapChunks(h.chunkDiskMapper)
 	expChunks := make([]*mmappedChunk, len(series.mmappedChunks))
 	copy(expChunks, series.mmappedChunks)
 
@@ -4507,6 +4693,7 @@ func TestReplayAfterMmapReplayError(t *testing.T) {
 	require.NoError(t, f.Close())
 
 	openHead()
+	h.mmapHeadChunks()
 
 	// There should be less m-map files due to corruption.
 	files, err = os.ReadDir(filepath.Join(dir, "chunks_head"))
@@ -4697,7 +4884,7 @@ func TestOOOAppendWithNoSeries(t *testing.T) {
 		require.False(t, created)
 		require.NotNil(t, ms)
 
-		require.Nil(t, ms.headChunk)
+		require.Nil(t, ms.headChunks)
 		require.NotNil(t, ms.ooo.oooHeadChunk)
 		require.Equal(t, expSamples, ms.ooo.oooHeadChunk.chunk.NumSamples())
 	}
@@ -4709,8 +4896,8 @@ func TestOOOAppendWithNoSeries(t *testing.T) {
 		require.NotNil(t, ms)
 
 		require.Nil(t, ms.ooo)
-		require.NotNil(t, ms.headChunk)
-		require.Equal(t, expSamples, ms.headChunk.chunk.NumSamples())
+		require.NotNil(t, ms.headChunks)
+		require.Equal(t, expSamples, ms.headChunks.chunk.NumSamples())
 	}
 
 	newLabels := func(idx int) labels.Labels { return labels.FromStrings("foo", fmt.Sprintf("%d", idx)) }
@@ -4821,6 +5008,7 @@ func TestGaugeHistogramWALAndChunkHeader(t *testing.T) {
 	appendHistogram(hists[4])
 
 	checkHeaders := func() {
+		head.mmapHeadChunks()
 		ms, _, err := head.getOrCreate(l.Hash(), l)
 		require.NoError(t, err)
 		require.Len(t, ms.mmappedChunks, 3)
@@ -4835,7 +5023,7 @@ func TestGaugeHistogramWALAndChunkHeader(t *testing.T) {
 			require.NoError(t, err)
 			require.Equal(t, expHeaders[i], chk.(*chunkenc.HistogramChunk).GetCounterResetHeader())
 		}
-		require.Equal(t, expHeaders[len(expHeaders)-1], ms.headChunk.chunk.(*chunkenc.HistogramChunk).GetCounterResetHeader())
+		require.Equal(t, expHeaders[len(expHeaders)-1], ms.headChunks.chunk.(*chunkenc.HistogramChunk).GetCounterResetHeader())
 	}
 	checkHeaders()
 
@@ -4898,6 +5086,7 @@ func TestGaugeFloatHistogramWALAndChunkHeader(t *testing.T) {
 	checkHeaders := func() {
 		ms, _, err := head.getOrCreate(l.Hash(), l)
 		require.NoError(t, err)
+		head.mmapHeadChunks()
 		require.Len(t, ms.mmappedChunks, 3)
 		expHeaders := []chunkenc.CounterResetHeader{
 			chunkenc.UnknownCounterReset,
@@ -4910,7 +5099,7 @@ func TestGaugeFloatHistogramWALAndChunkHeader(t *testing.T) {
 			require.NoError(t, err)
 			require.Equal(t, expHeaders[i], chk.(*chunkenc.FloatHistogramChunk).GetCounterResetHeader())
 		}
-		require.Equal(t, expHeaders[len(expHeaders)-1], ms.headChunk.chunk.(*chunkenc.FloatHistogramChunk).GetCounterResetHeader())
+		require.Equal(t, expHeaders[len(expHeaders)-1], ms.headChunks.chunk.(*chunkenc.FloatHistogramChunk).GetCounterResetHeader())
 	}
 	checkHeaders()