Repository: onedr0p/exportarr (go)

Issue:
Prowlarr Backfill: Add option to specify backfill duration
# Details

**Describe the solution you'd like:**

With prowlarr activated option backfill it can happen, that the exporter times out or prowlarr crashes, because of too many items in history. 
It would be nice to introduce an environment variable or cli flag to specify an amount of days to request, like you can do on the prowlarr stats page itself.

<!-- Note: A clear and concise description of what you want to happen. -->

the added environment variable or cli flag should specify the amount of days to request. 
so from that exportarr should request prowlarr with the backwards calculated amount of days from now, specified in the cli or environment variable

Pull request:
Add --backfill & --backfill-since-date options for Prowlarr.
** NOTE: depends on two prior PRs, review #114 and #115 first.**
<!--
Before you open the request please review the following guidelines and tips to help it be more easily integrated:

- Describe the scope of your change - i.e. what the change does.
- Describe any known limitations with your change.
- Please run any tests or examples that can exercise your modified code.

Thank you for contributing! We will try to test and integrate the change as soon as we can. There is no need to bump or check in on a pull request (it will clutter the discussion of the request).

Also don't be worried if the request is closed or not integrated sometimes our priorities might not match the priorities of the pull request. Don't fret, the open source community thrives on forks and GitHub makes it easy to keep your changes in a forked repo.
-->

**Description of the change**

Per title, creates some controls on how backfill is executed. If`--backfill` is set and/or `--backfill-since-date` is set, Prowlarr will backfill, starting from the beginning of time, or from the backfill date respectively.

Note that this is a change from prior behavior, `enable-additional-metrics` is no longer involved here. I think this is probably ok, breaking change wise, since the prior behavior hasn't yet seen a release.

**Benefits**

Allows control over backfills to prevent timeouts when attempting to backfill.

**Possible drawbacks**

Using a date rather than a duration helps protect against broken metrics, but conceivably a user good change their backfill date, in which case, they would see an issue in prometheus where aggregation metrics (`increase()`, `rate()`, etc) over the restart period may assume a jump in metrics which isn't actually present. This would require conscious effort, and only impacts that point in time.

**Applicable issues**

<!-- Enter any applicable Issues here (You can reference an issue using #) -->
- fixes #109 

**Additional information**

<!-- If there's anything else that's important and relevant to your pull request, mention that information here.-->


Patch:
diff --git a/README.md b/README.md
index b8a0b4f..6e181b3 100644
--- a/README.md
+++ b/README.md
@@ -146,9 +146,15 @@ Visit http://127.0.0.1:9709/metrics to see Radarr metrics
 ./exportarr prowlarr \
   --port 9710 \
   --url http://x.x.x.x:9696 \
-  --api-key amlmndfb503rfqaa5ln5hj5qkmu3hy18
+  --api-key amlmndfb503rfqaa5ln5hj5qkmu3hy18 \
+  --backfill \
+  --backfill-since-date 2023-03-01
 ```
 
+Backfill options are optional.
+
+Visit http://127.0.0.1:9710/metrics to see Prowlarr metrics
+
 #### Readarr (Experimental)
 
 ```sh
@@ -160,7 +166,7 @@ Visit http://127.0.0.1:9709/metrics to see Radarr metrics
   --api-key amlmndfb503rfqaa5ln5hj5qkmu3hy18
 ```
 
-Visit http://127.0.0.1:9710/metrics to see Prowlarr metrics
+Visit http://127.0.0.1:9711/metrics to see Readarr metrics
 
 ## Configuration
 
@@ -169,18 +175,25 @@ Visit http://127.0.0.1:9710/metrics to see Prowlarr metrics
 | `PORT`                       | `--port` or `-p`               | The port exportarr will listen on                              |           | ✅       |
 | `URL`                        | `--url` or `-u`                | The full URL to Sonarr, Radarr, or Lidarr                      |           | ✅       |
 | `APIKEY`                     | `--api-key` or `-a`            | API Key for Sonarr, Radarr or Lidarr                           |           | ❌       |
-| `APIKEY_FILE`                | `--api-key-file`               | API Key file location for Sonarr, Radarr or Lidarr             |           | ❌       |
+| `API_KEY_FILE`                | `--api-key-file`               | API Key file location for Sonarr, Radarr or Lidarr             |           | ❌       |
 | `CONFIG`                     | `--config` or `-c`             | Path to Sonarr, Radarr or Lidarr's `config.xml` (advanced)     |           | ❌       |
 | `INTERFACE`                  | `--interface` or `-i`          | The interface IP exportarr will listen on                      | `0.0.0.0` | ❌       |
 | `LOG_LEVEL`                  | `--log-level` or `-l`          | Set the default Log Level                                      | `INFO`    | ❌       |
 | `DISABLE_SSL_VERIFY`         | `--disable-ssl-verify`         | Set to `true` to disable SSL verification                      | `false`   | ❌       |
-| `BASIC_AUTH_PASSWORD`        | `--basic-auth-password`        | Set to your basic auth password                                |           | ❌       |
-| `BASIC_AUTH_USERNAME`        | `--basic-auth-username`        | Set to your basic auth username                                |           | ❌       |
+| `AUTH_PASSWORD`        | `--auth-password`        | Set to your basic or form auth password                                |           | ❌       |
+| `AUTH_USERNAME`        | `--auth-username`        | Set to your basic or form auth username                                |           | ❌       |
+| `FORM_AUTH`            | `--form-auth`            | Use Form Auth instead of basic auth                                        | `false` | ❌       |
 | `ENABLE_ADDITIONAL_METRICS`  | `--enable-additional-metrics`  | Set to `true` to enable gathering of additional metrics (slow) | `false`   | ❌       |
 | `ENABLE_UNKNOWN_QUEUE_ITEMS` | `--enable-unknown-queue-items` | Set to `true` to enable gathering unknown queue items          | `false`   | ❌       |
+| `PROWLARR__BACKFILL`         | `--backfill`                   | Set to `true` to enable backfill of historical metrics         | `false`   | ❌                |
+| `PROWLARR__BACKFILL_SINCE_DATE` | `--backfill-since-date`      | Set a date from which to start the backfill                    | `1970-01-01` (epoch) | ❌                |
 
 ### Prowlarr Backfill
 
-The prowlarr collector is a little different than other collectors as it's hitting an actual "stats" endpoint, collecting counters of events that happened in a small time window, rather than getting all-time statistics like the other collectors. This means that by default, when you start the prowlarr collector, collected stats will start from that moment (all counters will start from zero). `ENABLE_ADDITIONAL_METRIC` will tell the collector that it should backfill all-time metrics the first time it's queried. The actual API call this makes is *really* slow. On my prowlarr instance which is only a few months old, this took more than 5 seconds, so if you have a very active or very old prowlarr server this might time out, and need to be turned off.
+The prowlarr collector is a little different than other collectors as it's hitting an actual "stats" endpoint, collecting counters of events that happened in a small time window, rather than getting all-time statistics like the other collectors. This means that by default, when you start the prowlarr collector, collected stats will start from that moment (all counters will start from zero).
+
+To backill all Prowlarr Data, either use `PROWLARR__BACKFILL` or `--backfill`.
+
+Note that the first request can be extremely slow, depending on how long your prowlarr instance has been running. You can also specify a start date to limit the backfill if the backfill is timing out:
 
-Note that if you have to turn off backfill that your counters will reset to zero every time you restart the collector. This isn't a huge problem, prometheus expects counters to reset on restart, so normal aggregations like `rate()`, `increase()`, `irate()`, etc will all still work fine. The only thing that will be tough will be getting all-time stats (you can still do it, but you'll have to do something like `increase(stat_you_want_to_see[<timeserverhasbeenalive>])`, which will likely be slow)
\ No newline at end of file
+`PROWLARR__BACKFILL_DATE_SINCE=2023-03-01` or `--backfill-date-since=2023-03-01`
\ No newline at end of file
diff --git a/internal/collector/lidarr/music.go b/internal/collector/lidarr/music.go
index 8ce7fdf..e76f05d 100644
--- a/internal/collector/lidarr/music.go
+++ b/internal/collector/lidarr/music.go
@@ -19,10 +19,10 @@ type lidarrCollector struct {
 	albumsMetric           *prometheus.Desc // Total number of albums
 	albumsMonitoredMetric  *prometheus.Desc // Total number of monitored albums
 	albumsGenresMetric     *prometheus.Desc // Total number of albums by genre
+	albumsMissingMetric    *prometheus.Desc // Total number of missing albums
 	songsMetric            *prometheus.Desc // Total number of songs
 	songsMonitoredMetric   *prometheus.Desc // Total number of monitored songs
 	songsDownloadedMetric  *prometheus.Desc // Total number of downloaded songs
-	songsMissingMetric     *prometheus.Desc // Total number of missing songs
 	songsQualitiesMetric   *prometheus.Desc // Total number of songs by quality
 	errorMetric            *prometheus.Desc // Error Description for use with InvalidMetric
 }
@@ -72,6 +72,12 @@ func NewLidarrCollector(c *config.Config) *lidarrCollector {
 			[]string{"genre"},
 			prometheus.Labels{"url": c.URLLabel()},
 		),
+		albumsMissingMetric: prometheus.NewDesc(
+			"lidarr_albums_missing_total",
+			"Total number of missing albums",
+			nil,
+			prometheus.Labels{"url": c.URLLabel()},
+		),
 		songsMetric: prometheus.NewDesc(
 			"lidarr_songs_total",
 			"Total number of songs",
@@ -90,12 +96,6 @@ func NewLidarrCollector(c *config.Config) *lidarrCollector {
 			nil,
 			prometheus.Labels{"url": c.URLLabel()},
 		),
-		songsMissingMetric: prometheus.NewDesc(
-			"lidarr_songs_missing_total",
-			"Total number of missing songs",
-			nil,
-			prometheus.Labels{"url": c.URLLabel()},
-		),
 		songsQualitiesMetric: prometheus.NewDesc(
 			"lidarr_songs_quality_total",
 			"Total number of downloaded songs by quality",
@@ -119,10 +119,10 @@ func (collector *lidarrCollector) Describe(ch chan<- *prometheus.Desc) {
 	ch <- collector.albumsMetric
 	ch <- collector.albumsMonitoredMetric
 	ch <- collector.albumsGenresMetric
+	ch <- collector.albumsMissingMetric
 	ch <- collector.songsMetric
 	ch <- collector.songsMonitoredMetric
 	ch <- collector.songsDownloadedMetric
-	ch <- collector.songsMissingMetric
 	ch <- collector.songsQualitiesMetric
 }
 
@@ -198,9 +198,9 @@ func (collector *lidarrCollector) Collect(ch chan<- prometheus.Metric) {
 		}
 	}
 
-	songMissing := model.Missing{}
-	if err := c.DoRequest("wanted/missing", &songMissing); err != nil {
-		log.Errorw("Error getting missing", "error", err)
+	albumsMissing := model.Missing{}
+	if err := c.DoRequest("wanted/missing", &albumsMissing); err != nil {
+		log.Errorw("Error getting missing albums", "error", err)
 		ch <- prometheus.NewInvalidMetric(collector.errorMetric, err)
 		return
 	}
@@ -209,9 +209,9 @@ func (collector *lidarrCollector) Collect(ch chan<- prometheus.Metric) {
 	ch <- prometheus.MustNewConstMetric(collector.artistsMonitoredMetric, prometheus.GaugeValue, float64(artistsMonitored))
 	ch <- prometheus.MustNewConstMetric(collector.artistsFileSizeMetric, prometheus.GaugeValue, float64(artistsFileSize))
 	ch <- prometheus.MustNewConstMetric(collector.albumsMetric, prometheus.GaugeValue, float64(albums))
+	ch <- prometheus.MustNewConstMetric(collector.albumsMissingMetric, prometheus.GaugeValue, float64(albumsMissing.TotalRecords))
 	ch <- prometheus.MustNewConstMetric(collector.songsMetric, prometheus.GaugeValue, float64(songs))
 	ch <- prometheus.MustNewConstMetric(collector.songsDownloadedMetric, prometheus.GaugeValue, float64(songsDownloaded))
-	ch <- prometheus.MustNewConstMetric(collector.songsMissingMetric, prometheus.GaugeValue, float64(songMissing.TotalRecords))
 
 	if len(artistGenres) > 0 {
 		for genre, count := range artistGenres {
diff --git a/internal/collector/prowlarr/stats.go b/internal/collector/prowlarr/stats.go
index 6f45a80..8e08afc 100644
--- a/internal/collector/prowlarr/stats.go
+++ b/internal/collector/prowlarr/stats.go
@@ -116,12 +116,9 @@ type prowlarrCollector struct {
 }
 
 func NewProwlarrCollector(c *config.Config) *prowlarrCollector {
-	var lastStatUpdate time.Time
-	if c.EnableAdditionalMetrics {
-		// If additional metrics are enabled, backfill the cache.
-		lastStatUpdate = time.Time{}
-	} else {
-		lastStatUpdate = time.Now()
+	lastStatUpdate := time.Now()
+	if c.Prowlarr.Backfill || !c.Prowlarr.BackfillSinceTime.IsZero() {
+		lastStatUpdate = c.Prowlarr.BackfillSinceTime
 	}
 	return &prowlarrCollector{
 		config:             c,
diff --git a/internal/commands/prowlarr.go b/internal/commands/prowlarr.go
index 3c65a68..5b3755a 100644
--- a/internal/commands/prowlarr.go
+++ b/internal/commands/prowlarr.go
@@ -10,6 +10,9 @@ import (
 
 func init() {
 	rootCmd.AddCommand(prowlarrCmd)
+
+	prowlarrCmd.PersistentFlags().Bool("backfill", false, "Backfill Prowlarr")
+	prowlarrCmd.PersistentFlags().String("backfill-since-date", "", "Date from which to start Prowlarr Backfill")
 }
 
 var prowlarrCmd = &cobra.Command{
@@ -19,6 +22,10 @@ var prowlarrCmd = &cobra.Command{
 	RunE: func(cmd *cobra.Command, args []string) error {
 		conf.Arr = "prowlarr"
 		conf.ApiVersion = "v1"
+		conf.LoadProwlarrFlags(cmd.PersistentFlags())
+		if err := conf.Prowlarr.Validate(); err != nil {
+			return err
+		}
 		serveHttp(func(r *prometheus.Registry) {
 			r.MustRegister(
 				prowlarrCollector.NewProwlarrCollector(conf),
diff --git a/internal/config/config.go b/internal/config/config.go
index c9f63d5..96640fc 100644
--- a/internal/config/config.go
+++ b/internal/config/config.go
@@ -18,22 +18,24 @@ import (
 )
 
 type Config struct {
-	Arr                     string `koanf:"arr"`
-	LogLevel                string `koanf:"log-level" validate:"ValidateLogLevel"`
-	LogFormat               string `koanf:"log-format" validate:"in:console,json"`
-	URL                     string `koanf:"url" validate:"required|url"`
-	ApiKey                  string `koanf:"api-key" validate:"required|regex:([a-z0-9]{32})"`
-	ApiKeyFile              string `koanf:"api-key-file"`
-	ApiVersion              string `koanf:"api-version" validate:"required|in:v3,v4"`
-	XMLConfig               string `koanf:"config"`
-	Port                    int    `koanf:"port" validate:"required"`
-	Interface               string `koanf:"interface" validate:"required|ip"`
-	DisableSSLVerify        bool   `koanf:"disable-ssl-verify"`
-	AuthUsername            string `koanf:"auth-username"`
-	AuthPassword            string `koanf:"auth-password"`
-	FormAuth                bool   `koanf:"form-auth"`
-	EnableUnknownQueueItems bool   `koanf:"enable-unknown-queue-items"`
-	EnableAdditionalMetrics bool   `koanf:"enable-additional-metric"`
+	Arr                     string         `koanf:"arr"`
+	LogLevel                string         `koanf:"log-level" validate:"ValidateLogLevel"`
+	LogFormat               string         `koanf:"log-format" validate:"in:console,json"`
+	URL                     string         `koanf:"url" validate:"required|url"`
+	ApiKey                  string         `koanf:"api-key" validate:"required|regex:([a-z0-9]{32})"`
+	ApiKeyFile              string         `koanf:"api-key-file"`
+	ApiVersion              string         `koanf:"api-version" validate:"required|in:v3,v4"`
+	XMLConfig               string         `koanf:"config"`
+	Port                    int            `koanf:"port" validate:"required"`
+	Interface               string         `koanf:"interface" validate:"required|ip"`
+	DisableSSLVerify        bool           `koanf:"disable-ssl-verify"`
+	AuthUsername            string         `koanf:"auth-username"`
+	AuthPassword            string         `koanf:"auth-password"`
+	FormAuth                bool           `koanf:"form-auth"`
+	EnableUnknownQueueItems bool           `koanf:"enable-unknown-queue-items"`
+	EnableAdditionalMetrics bool           `koanf:"enable-additional-metric"`
+	Prowlarr                ProwlarrConfig `koanf:"prowlarr"`
+	k                       *koanf.Koanf
 }
 
 func (c *Config) UseBasicAuth() bool {
@@ -57,6 +59,7 @@ func (c *Config) URLLabel() string {
 	}
 	return c.URL
 }
+
 func LoadConfig(flags *flag.FlagSet) (*Config, error) {
 	k := koanf.New(".")
 
@@ -74,7 +77,10 @@ func LoadConfig(flags *flag.FlagSet) (*Config, error) {
 
 	// Environment
 	err = k.Load(env.Provider("", ".", func(s string) string {
-		return strings.Replace(strings.ToLower(s), "_", "-", -1)
+		s = strings.ToLower(s)
+		s = strings.Replace(s, "__", ".", -1)
+		s = strings.Replace(s, "_", "-", -1)
+		return s
 	}), nil)
 	if err != nil {
 		return nil, err
@@ -109,7 +115,7 @@ func LoadConfig(flags *flag.FlagSet) (*Config, error) {
 	if err := k.Unmarshal("", &out); err != nil {
 		return nil, err
 	}
-
+	out.k = k
 	return &out, nil
 }
 
@@ -146,3 +152,23 @@ func (c *Config) Messages() map[string]string {
 		"LogLevel.validateLogLevel": "Log Level must be one of: debug, info, warn, error, dpanic, panic, fatal",
 	}
 }
+
+func (c *Config) Translates() map[string]string {
+	return validate.MS{
+		"LogLevel":                "log-level",
+		"LogFormat":               "log-format",
+		"URL":                     "url",
+		"ApiKey":                  "api-key",
+		"ApiKeyFile":              "api-key-file",
+		"ApiVersion":              "api-version",
+		"XMLConfig":               "config",
+		"Port":                    "port",
+		"Interface":               "interface",
+		"DisableSSLVerify":        "disable-ssl-verify",
+		"AuthUsername":            "auth-username",
+		"AuthPassword":            "auth-password",
+		"FormAuth":                "form-auth",
+		"EnableUnknownQueueItems": "enable-unknown-queue-items",
+		"EnableAdditionalMetrics": "enable-additional-metric",
+	}
+}
diff --git a/internal/config/prowlarr.go b/internal/config/prowlarr.go
new file mode 100644
index 0000000..a3b6205
--- /dev/null
+++ b/internal/config/prowlarr.go
@@ -0,0 +1,58 @@
+package config
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/gookit/validate"
+	"github.com/knadh/koanf/providers/posflag"
+	"github.com/knadh/koanf/v2"
+	flag "github.com/spf13/pflag"
+)
+
+type ProwlarrConfig struct {
+	Backfill          bool   `koanf:"backfill"`
+	BackfillSinceDate string `koanf:"backfill-since-date" validate:"date"`
+	BackfillSinceTime time.Time
+}
+
+func (p ProwlarrConfig) Validate() error {
+	v := validate.Struct(p)
+	if !v.Validate() {
+		return v.Errors
+	}
+	if p.BackfillSinceDate != "" && p.BackfillSinceTime.IsZero() {
+		// Should be unreachable as long as we validate that the date is valid in LoadProwlarrFlags/Validate
+		return fmt.Errorf("backfill-since-date is not a valid date")
+	}
+	return nil
+}
+
+func (p ProwlarrConfig) Messages() map[string]string {
+	return validate.MS{
+		"backfill-since-date": "backfill-since-date is not a valid date",
+	}
+}
+
+func (p ProwlarrConfig) Translates() map[string]string {
+	return validate.MS{
+		"BackfillSinceDate": "backfill-since-date",
+	}
+}
+
+func (c *Config) LoadProwlarrFlags(flags *flag.FlagSet) error {
+	err := c.k.Load(posflag.Provider(flags, ".", c.k), nil, koanf.WithMergeFunc(func(src, dest map[string]interface{}) error {
+		dest["prowlarr"] = src
+		return nil
+	}))
+	if err != nil {
+		return err
+	}
+
+	err = c.k.Unmarshal("prowlarr", &c.Prowlarr)
+	c.Prowlarr.BackfillSinceTime = c.k.Time("prowlarr.backfill-since-date", "2006-01-02")
+	if err != nil {
+		return err
+	}
+	return nil
+}


Test patch:
diff --git a/internal/config/prowlarr_test.go b/internal/config/prowlarr_test.go
new file mode 100644
index 0000000..3d544a9
--- /dev/null
+++ b/internal/config/prowlarr_test.go
@@ -0,0 +1,46 @@
+package config
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestValidateProwlarr(t *testing.T) {
+	tm, _ := time.Parse("2006-01-02", "2021-01-01")
+	parameters := []struct {
+		name        string
+		config      *ProwlarrConfig
+		shouldError bool
+	}{
+		{
+			name: "good",
+			config: &ProwlarrConfig{
+				Backfill:          true,
+				BackfillSinceTime: tm,
+				BackfillSinceDate: "2021-01-01",
+			},
+		},
+		{
+			name: "bad-date",
+			config: &ProwlarrConfig{
+				Backfill:          true,
+				BackfillSinceTime: tm,
+				BackfillSinceDate: "2021-31-31",
+			},
+			shouldError: true,
+		},
+	}
+
+	for _, parameter := range parameters {
+		t.Run(parameter.name, func(t *testing.T) {
+			err := parameter.config.Validate()
+			if parameter.shouldError {
+				require.Error(t, err)
+			} else {
+				require.NoError(t, err)
+			}
+		})
+	}
+}