Repository: thanos-io/thanos

Issue #4118: Query Frontend: missing port in address
Created: 2021-04-29T06:14:52Z
Thanos, Prometheus and Golang version used:

thanos, version 0.20.0 (branch: HEAD, revision: 893188a6752e260cf54f3c082392a521b6c92f75)
  build user:       root@83239bffd915
  build date:       20210428-12:54:57
  go version:       go1.16.3
  platform:         linux/amd64

Object Storage Provider:
Irrelevant
What happened:
After an upgrade from Thanos 0.19.0 to 0.20.0, Thanos Ruler is logging following errors:
Apr 29 06:03:56 hostname thanos: level=error ts=2021-04-29T06:03:56.854131338Z caller=http.go:48 protocol=http http.component=server msg="failed to parse host port for http log decision" err="address metrics.example.com: missing port in address"
Apr 29 06:03:56 hostname thanos: level=error ts=2021-04-29T06:03:56.854439031Z caller=http.go:48 protocol=http http.component=server msg="failed to parse host port for http log decision" err="address metrics.example.com: missing port in address"

Following parameters are being used to start the service:
/usr/bin/thanos rule --data-dir=/var/lib/thanos/rule --grpc-address=0.0.0.0:10910 --http-address=0.0.0.0:10911 --rule-file=/etc/prometheus/thanos_alerts.yml --query=localhost:10913/thanos/query --alertmanagers.url=https://alerts.example.com --alert.query-url=https://metrics.example.com/all --web.external-prefix=/thanos/rule
If I change -alert.query-url parameter value to https://metrics.example.com:443/all, these errors are gone.
What you expected to happen:
It is perfectly fine for URL not to contain port number when default port is used. I would expect https://metrics.example.com/all to work properly.
How to reproduce it (as minimally and precisely as possible):
Run Thanos Ruler service with --alert.query-url=https://metrics.example.com/all flag.
Anything else we need to know:
No.

Duplicate issue #4334: thanos query switches between "No stores registered" and normally working state back and forth
Created: 2021-06-11T06:31:34Z
I have setup a thanos with sidecar, store and query.
Everything seems to be working as expected. Query shows two stores, a Sidecar and a Store.
I can query data as expected.
After some random time stores are gone and Query shows just "No stores registered".  Sometimes stores come back after some time, but it seems to be very random.
Logs on startup are
activeStores | component | level  | msg 

- 	storeset	info	adding new storeAPI to query storeset
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	info	adding new storeAPI to query storeset
- 	storeset	warning	update of store node failed
- 	storeset	info	removing store because it's unhealthy or does not exist
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	warning	update of store node failed
- 	storeset	info	adding new storeAPI to query storeset
- 	 - 	info	changing probe status
- 	query	info	listening for requests and metrics
- 	query	info	TLS is disabled.
- 	 - 	info	changing probe status
- 	query	info	listening for serving gRPC
- 	 - 	info	starting query node
- 	 - 	info	


at the point where the stores are gone I don't see any specific log. When enabling debug level I just see an endless stream of

activeStores | component | level  | msg 

2	storeset	debug	checked requested storeAPIs
 - 	storeset	debug	starting updating storeAPIs
2	storeset	debug	checked requested storeAPIs
 - 	storeset	debug	starting updating storeAPIs
2	storeset	debug	checked requested storeAPIs

I am a bit stuck here and looking for advise how to further analyze this.
Not sure if there is kind of a corner case bug in thanos, or something wrong with my config or an issue with our infrastructure or setup.
Config:
<image-entrypoint> query --http-address=0.0.0.0:9090 --log.format=json --log.level=debug --query.lookback-delta=15m --store=thanos-store.my-namespace.svc.cluster.local:10901 --store=prometheus.my-namespace.svc.cluster.local:10901

Version:
thanos --version
thanos, version 0.21.1 (branch: HEAD, revision: 3558f4a5adc8d4068f82528570a439d440f98b9d)
  build user:       circleci@eb60c35e294a
  build date:       20210604-12:24:45
  go version:       go1.16.4
  platform:         linux/amd64

Closing comment (2021-06-22T13:51:08Z): I looks like this could actually be related to #4118.
Thanos query is accessed via an oauth proxy in our setup. Setting -pass-host-header to false, might have solved the issue.
The fix is just hard to validate, so far I did not see the issue again.
@wiardvanrij thanks for your input on this issue 👍🏼