Repository: prometheus/tsdb
Pull request #450 (2018-11-12 14:12:38)

Add WALSegmentSize as an option of tsdb creation
## Problem to solve :

Prometheus server is running on a raspberry pi : 
In order to preserve the SD Card lifetime, I mounted the WAL directory on the tmpfs (memory) partition.

The tmpfs partition is quickly full.

## Solution :

Provides the wal semgent file size as an option via `wal.Options` struct


### Results

Then, I will be able to limit the wal segment file sizes ( I set 1048576 for example ). Smaller files are written, and the oldest ones are truncated : the wal directory keep an acceptable size on tmpfs partition.

I prepared an [associated patch on prometheus server project](https://github.com/prometheus/prometheus/compare/v2.4.3...glutamatt:v2.4.3)

#### It [works perfectly](https://github.com/prometheus/tsdb/pull/450#issuecomment-437943996) on my raspberry now ✨


<details>
  <summary>output of the prometheus server on the raspberry when it crashed</summary>
  

```
level=warn ts=2018-11-10T03:17:09.556470872Z caller=scrape.go:804 component="scrape manager" scrape_pool=teleinfo target=http://localhost:2112/metrics msg="append failed" err="write to WAL: log samples: write /var/prometheus/data/wal/00000000: no space left on device"
panic: runtime error: slice bounds out of range

goroutine 180 [running]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.(*WAL).log(0x33eaa20, 0x3d40000, 0x39, 0x400, 0x1, 0xfb9f8d30, 0x166)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal/wal.go:480 +0x3ac
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.(*WAL).Log(0x33eaa20, 0x39ebd54, 0x1, 0x1, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal/wal.go:450 +0xc8
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).log(0x344f9c0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:678 +0x158
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Commit(0x344f9c0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:689 +0xac
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.dbAppender.Commit(0x14f6228, 0x344f9c0, 0x30a2f50, 0x40080000, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:349 +0x24
github.com/prometheus/prometheus/storage/tsdb.appender.Commit(0x14f6468, 0x3c9db20, 0x0, 0xfb9f8d30)
        /go/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:258 +0x24
github.com/prometheus/prometheus/storage.(*fanoutAppender).Commit(0x3bdfc20, 0x3c9db30, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/fanout.go:161 +0x28
github.com/prometheus/prometheus/scrape.(*scrapeLoop).report(0x3f82640, 0x60f163df, 0xbef1b14d, 0x68485c3d, 0x5d7c, 0x233d9b0, 0x4006ce, 0x0, 0x3, 0x3, ...)
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:1127 +0x3b8
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x3f82640, 0x77359400, 0x0, 0x77359400, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:818 +0x650
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:329 +0x384
```
</details>

krasi-georgiev: That is strange the `defaultSegmentSize` should not affect the overall WAL folder size, but only when a new segment is started.
The size of the WAL folder is determined by the amount of metrics you are scraping and the block time time range.

so in your case changing the  `defaultSegmentSize` to a smaller value should result in smaller files sizes, but more more overall segments.

For example if before the change you had 10x128mb = 1280mb after changing the default size to 12mb should end up with something like: 107x12mb = 1280mb
glutamatt: > The size of the WAL folder is determined by the amount of metrics you are scraping and the block time time range.

before my patch, there was only one file in the wal folder `/var/prometheus/data/wal/00000000`
the file was growing (i guess until the default size but it crashed at 20M, the tmpfs partition size)

After the patch (and the config) : the new folders (NOT the wal) with chunks are written as expected each 2 hours , with just one file `000001` in the `chunks` subdirectory. Perfectly fine. I guess the tsdb block duration settings do their jobs.

The patch+config only affect the segments size of the WAL, and how many are at the same time.
So in the wal directory, the smaller the segments are, the more they are numerous and rotating (thanks to the db.reaload -> head.Truncat -> Checkpoints calls)

krasi-georgiev: yeah that is true, the checkpointing will be triggered more often which would clear the obsolete samples more often.

@gouthamve 	, @fabxc 	can you think of any negative side effects of making this configurable?


@glutamatt 	btw I don't see any reason why you should expose the defaultSegmentSize constant.
glutamatt: > why you should expose the defaultSegmentSize constant

https://github.com/prometheus/tsdb/pull/450/files#diff-7a117cbfd5485ed35efcbb2995c37aa0R48



> Also don't forget that using the tmpfs as WAL means a restart of the RPI would cause all data from the last block to be lost.

Sure, but i prefer to lose some data on a reboot rather than burning my sd card (a changing/buying an other one will lead to more "unpersisted" data 🤡)
glutamatt: > I don't see any reason why you should expose the defaultSegmentSize constant

I preferred to be fully backward compatible :  In `db.go` file, we're in the `tsdb` package. So I expose the constant to set the default option or use it if the option is let to zero.

_maybe if someone is interested of [the project of my use case](https://github.com/glutamatt/go-teleinfo-prom-exporter)_
krasi-georgiev: Making the wal segment configurable will be very useful for tests.
glutamatt: Hi @krasi-georgiev , 
is there something i could help to bring this update to be merged ? 
thank you in advance
krasi-georgiev: @glutamatt 	sorry for the delay. We started maintaining a change file so do you mind adding an entry for the API change and the new exposed `WALSegmentSize` option.

https://github.com/prometheus/tsdb/blob/master/CHANGELOG.md
glutamatt: > adding an entry for the API change and the new exposed WALSegmentSize option

Done ! 👌

If you think about a better wording or syntax, I would be fine to update the changelog 
glutamatt: suggestions are commited and rebased with sign off :white_check_mark: 
glutamatt: > sorry to be a pain but I just remembered that we need to add a test for this new behaviour.

:+1: 

> Something like:
> Open a new db with the smallest WALSegmentSize that makes sense(to keep the total test time short).
> Start writing to the wal and ensure that wal files get split at the WALSegmentSize.
> 
> If you get stuck with the test ping me and I will try to help.

here we are : 
https://github.com/prometheus/tsdb/pull/450/files#diff-65348100fd3ccd8cb8fb4f6bad6bc2d9R698

glutamatt: > minor nits

minor nits done, you're welcome
krasi-georgiev: yep, all good, thanks for the patience. 

I just a did the  0.3.0 release and will wait for this to get merged in Prometheus and after that will merge your PR as well.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1f2abce..8e3c01f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,11 +1,10 @@
 ## master / unreleased
-
+ - [CHANGE] New `WALSegmentSize` option to override the `DefaultOptions.WALSegmentSize`. Added to allow using smaller wal files. For example using tmpfs on a RPI to minimise the SD card wear out from the constant WAL writes. As part of this change the `DefaultOptions.WALSegmentSize` constant was also exposed.
 
 ## 0.3.1
 - [BUGFIX] Fixed most windows test and some actual bugs for unclosed file readers.
 
 ## 0.3.0
-
  - [CHANGE] `LastCheckpoint()` used to return just the segment name and now it returns the full relative path.
  - [CHANGE] `NewSegmentsRangeReader()` can now read over miltiple wal ranges by using the new `SegmentRange{}` struct.
  - [CHANGE] `CorruptionErr{}` now also exposes the Segment `Dir` which is added when displaying any errors.
diff --git a/db.go b/db.go
index 3a47f0b..e5a057c 100644
--- a/db.go
+++ b/db.go
@@ -45,6 +45,7 @@ import (
 // millisecond precision timestamps.
 var DefaultOptions = &Options{
 	WALFlushInterval:  5 * time.Second,
+	WALSegmentSize:    wal.DefaultSegmentSize,
 	RetentionDuration: 15 * 24 * 60 * 60 * 1000, // 15 days in milliseconds
 	BlockRanges:       ExponentialBlockRanges(int64(2*time.Hour)/1e6, 3, 5),
 	NoLockfile:        false,
@@ -55,6 +56,9 @@ type Options struct {
 	// The interval at which the write ahead log is flushed to disk.
 	WALFlushInterval time.Duration
 
+	// Segments (wal files) max size
+	WALSegmentSize int
+
 	// Duration of persisted data to keep.
 	RetentionDuration uint64
 
@@ -263,7 +267,11 @@ func Open(dir string, l log.Logger, r prometheus.Registerer, opts *Options) (db
 		return nil, errors.Wrap(err, "create leveled compactor")
 	}
 
-	wlog, err := wal.New(l, r, filepath.Join(dir, "wal"))
+	segmentSize := wal.DefaultSegmentSize
+	if opts.WALSegmentSize > 0 {
+		segmentSize = opts.WALSegmentSize
+	}
+	wlog, err := wal.NewSize(l, r, filepath.Join(dir, "wal"), segmentSize)
 	if err != nil {
 		return nil, err
 	}
diff --git a/wal/wal.go b/wal/wal.go
index 5134850..92374e3 100644
--- a/wal/wal.go
+++ b/wal/wal.go
@@ -35,7 +35,7 @@ import (
 )
 
 const (
-	defaultSegmentSize = 128 * 1024 * 1024 // 128 MB
+	DefaultSegmentSize = 128 * 1024 * 1024 // 128 MB
 	pageSize           = 32 * 1024         // 32KB
 	recordHeaderSize   = 7
 )
@@ -174,7 +174,7 @@ type WAL struct {
 
 // New returns a new WAL over the given directory.
 func New(logger log.Logger, reg prometheus.Registerer, dir string) (*WAL, error) {
-	return NewSize(logger, reg, dir, defaultSegmentSize)
+	return NewSize(logger, reg, dir, DefaultSegmentSize)
 }
 
 // NewSize returns a new WAL over the given directory.


Test patch:
diff --git a/db_test.go b/db_test.go
index 6217849..0ed4356 100644
--- a/db_test.go
+++ b/db_test.go
@@ -695,6 +695,30 @@ func TestWALFlushedOnDBClose(t *testing.T) {
 	testutil.Equals(t, []string{"labelvalue"}, values)
 }
 
+func TestWALSegmentSizeOption(t *testing.T) {
+	options := *DefaultOptions
+	options.WALSegmentSize = 2 * 32 * 1024
+	db, close := openTestDB(t, &options)
+	defer close()
+	app := db.Appender()
+	for i := int64(0); i < 155; i++ {
+		_, err := app.Add(labels.Labels{labels.Label{Name: "wal", Value: "size"}}, i, rand.Float64())
+		testutil.Ok(t, err)
+		testutil.Ok(t, app.Commit())
+	}
+
+	files, err := ioutil.ReadDir(filepath.Join(db.Dir(), "wal"))
+	testutil.Assert(t, len(files) > 1, "current WALSegmentSize should result in more than a single WAL file.")
+	testutil.Ok(t, err)
+	for i, f := range files {
+		if len(files)-1 != i {
+			testutil.Equals(t, int64(options.WALSegmentSize), f.Size(), "WAL file size doesn't match WALSegmentSize option, filename: %v", f.Name())
+			continue
+		}
+		testutil.Assert(t, int64(options.WALSegmentSize) > f.Size(), "last WAL file size is not smaller than the WALSegmentSize option, filename: %v", f.Name())
+	}
+}
+
 func TestTombstoneClean(t *testing.T) {
 	numSamples := int64(10)