Repository: thanos-io/thanos

Issue #4878: Tools bucket verify -i index_know_issue does not parse all blocks
Created: 2021-11-19T10:18:00Z
Thanos, Prometheus and Golang version used:

thanos, version 0.23.1 (branch: HEAD, revision: 5327cd8b9bd56ce4912fe602197ad04ab2722b18)
  build user:       root@0acc901868e9
  build date:       20211005-12:08:29
  go version:       go1.16.8
  platform:         linux/amd64
Object Storage Provider:
S3
What happened:
When the command thanos tools bukcet verify is executed only the first block is verified at the index_issue step.
What you expected to happen:
All blocs of the bucket must be parsed.
How to reproduce it (as minimally and precisely as possible):
Run the command : thanos tools bucket verify --objstore.config-file=config.yml --log.level=debug -i -i index_known_issues and check if all blocks of the bucket are parsed.
Full logs to relevant components:
List of blocks from bucket  metrics-a01-dc1

$ s3cmd ls  s3://metrics-a01-dc1/
                          DIR  s3://metrics-a01-dc1/01ER2E8M3JTSNWY3NAH6RHFZ22/
                          DIR  s3://metrics-a01-dc1/01ER2H96B29Y8MQK9XQA6QJPNR/
                          DIR  s3://metrics-a01-dc1/01ESA8TF0Z2MSDH55BJ619QCH4/
                          DIR  s3://metrics-a01-dc1/01ESACAW94WXSTEF30A773TKF0/
                          DIR  s3://metrics-a01-dc1/01ETAHPD5KYCBZY76R726RC5YC/
                          DIR  s3://metrics-a01-dc1/01ETAMM17XN4VY5RT2HK1TFFTC/
                          DIR  s3://metrics-a01-dc1/01EVEJNP5G87S2ZSE8EJJ51JH5/
                          DIR  s3://metrics-a01-dc1/01EVENEF8VWVZY99KE4TDK90H5/
                          DIR  s3://metrics-a01-dc1/01EWJP35XXGX3TTH4ED50DY3BT/
                          DIR  s3://metrics-a01-dc1/01EWJS44TE0X07BWBVCZ5S5YH3/
                          DIR  s3://metrics-a01-dc1/01EXPVC904MFVSAGBG5YTJ8EDV/
                          DIR  s3://metrics-a01-dc1/01EXPYXV4RDKP5YNW3FXDQF1K2/
                          DIR  s3://metrics-a01-dc1/01EYTV0ZGVGF0N3VYVYRMNXR00/
                          DIR  s3://metrics-a01-dc1/01EYTXY68FTNEVPN49E6VE9D2R/
                          DIR  s3://metrics-a01-dc1/01EZZ4T9G1F7M0XG1T4K9WSTY7/
                          DIR  s3://metrics-a01-dc1/01EZZ85Q1VS38XD824M53VEA8V/
                          DIR  s3://metrics-a01-dc1/01F18A3J1JSEEEHYTXK12AP80K/
                          DIR  s3://metrics-a01-dc1/01F18DY65HDQZT1007Z5XFGP8G/
                          DIR  s3://metrics-a01-dc1/01F278Q16GCXSM2YJQPKACPMA5/
                          DIR  s3://metrics-a01-dc1/01F27C4H9PJBP2P7XF0BESRKZ7/
                          DIR  s3://metrics-a01-dc1/01F3B5KMMPKMZ9GC4QHF0PZBXE/
                          DIR  s3://metrics-a01-dc1/01F3B9CK2D7JPB0Q85EV19C55F/
                          DIR  s3://metrics-a01-dc1/01F4F5QH614CM4RXMD0T2PF2B7/
                          DIR  s3://metrics-a01-dc1/01F4F9B10PKJR9AQB6TCXQ9SXQ/
                          DIR  s3://metrics-a01-dc1/01F5K8TC9WH11321NH9DF6PDKW/
                          DIR  s3://metrics-a01-dc1/01F5KCGNATXSXX5TK8P7WJTMWS/
                          DIR  s3://metrics-a01-dc1/01F6QB3YSM5AC9QTW6T9J4RA3F/
                          DIR  s3://metrics-a01-dc1/01F6QETNYD5QYJWM6H0EQMH9FM/
                          DIR  s3://metrics-a01-dc1/01F7VDBX05X90FGHAG2ZSBVKV5/
                          DIR  s3://metrics-a01-dc1/01F7VGTXEQDV93VAHWG9VHJRW0/
                          DIR  s3://metrics-a01-dc1/01F8ZJ1K2RM1FQQQA7V0GPVES7/
                          DIR  s3://metrics-a01-dc1/01F8ZNJW6QB8NH1P4T6PEKYA8X/
                          DIR  s3://metrics-a01-dc1/01FA3MC0ATKDWCJ6BN52QR3J46/
                          DIR  s3://metrics-a01-dc1/01FA3R46KR3QTMHGC0YA60D4EC/
                          DIR  s3://metrics-a01-dc1/01FB7K1WHETTSBTXFF4K3B5P92/
                          DIR  s3://metrics-a01-dc1/01FB7PNXMR156WQZ8HGTQBPT2B/
                          DIR  s3://metrics-a01-dc1/01FCE9WF31K3VHEKP07B2HBYRV/
                          DIR  s3://metrics-a01-dc1/01FCEDB105QQTHNAHQZEWZZX9D/
                          DIR  s3://metrics-a01-dc1/01FDFP1R555BGB0XBBAQCK4A7T/
                          DIR  s3://metrics-a01-dc1/01FDFSNKRV3G3TE4SPGK9V5WFD/
                          DIR  s3://metrics-a01-dc1/01FEK3HHPEW2CVBKG2X519W5HW/
                          DIR  s3://metrics-a01-dc1/01FEKF181R40T3HZR87HC90S5G/
                          DIR  s3://metrics-a01-dc1/01FEKSJJ54CKQSWYXYB7BHX50V/
                          DIR  s3://metrics-a01-dc1/01FEKXAXEN9H4Z2858JRA6V99Y/
                          DIR  s3://metrics-a01-dc1/01FFQ5XJ8Z061TE47889N6Z84Q/
                          DIR  s3://metrics-a01-dc1/01FFQHJGSR783T8HTF67R84EGM/
                          DIR  s3://metrics-a01-dc1/01FFQW7RWVX59ZVMP75RJGKKTY/
                          DIR  s3://metrics-a01-dc1/01FFQZYZEAEBFWATHK6NJYGP99/
                          DIR  s3://metrics-a01-dc1/01FGV353WEC0B105HACDZ5CQ74/
                          DIR  s3://metrics-a01-dc1/01FGV8N4HABJVK9GBTB3E8HAHC/
                          DIR  s3://metrics-a01-dc1/01FGVK5VXAC7FS42DD25BTXS9G/
                          DIR  s3://metrics-a01-dc1/01FGVMZXBD3MDD4S6AZNKEQX82/
                          DIR  s3://metrics-a01-dc1/01FHZ689FWQWEER9KFREF5AW5F/
                          DIR  s3://metrics-a01-dc1/01FHZFPQ1P553VMWVPCKQX92P4/
                          DIR  s3://metrics-a01-dc1/01FHZR20ECJ25EDAZV96ZX1WCG/
                          DIR  s3://metrics-a01-dc1/01FHZTZFF5SDKGQ9DBS9FVDP9D/
                          DIR  s3://metrics-a01-dc1/01FK2YD7EWRE0K2Y0V4725T1X9/
                          DIR  s3://metrics-a01-dc1/01FK2YDN8VW3DH7RQYHQWTHCZ0/
                          DIR  s3://metrics-a01-dc1/01FK37YZGKR8362ZPFDMHG78B6/
                          DIR  s3://metrics-a01-dc1/01FK3HJAZFJFJSB9EDZVTM4B7C/
                          DIR  s3://metrics-a01-dc1/01FK3TDK7KP9DYQHEVWM8C8J7M/
                          DIR  s3://metrics-a01-dc1/01FK3XDPXK9FXX1MRNAKMKKY85/
                          DIR  s3://metrics-a01-dc1/01FM70TJAQYZZA8K7VV1JW1S4W/
                          DIR  s3://metrics-a01-dc1/01FM70TM9EV83QSDG6ZC2XDY4E/
                          DIR  s3://metrics-a01-dc1/01FM7A3QQ5ZWB2XSSMTZS2NMRD/
                          DIR  s3://metrics-a01-dc1/01FM7N1DX4HYVSFN330HHE82G8/
                          DIR  s3://metrics-a01-dc1/01FM7XR8A1EH4BBGNW4T432F1K/
                          DIR  s3://metrics-a01-dc1/01FM80PVZ3RXRWVCCRJ84RPA1X/
                          DIR  s3://metrics-a01-dc1/01FMC2QRF1JC0GWREHG4ZGAY4F/
                          DIR  s3://metrics-a01-dc1/01FMC2QVHCSD4FNJZZ5ASWH19A/
                          DIR  s3://metrics-a01-dc1/01FMC438AT78YAN3WQRPX2VBXN/
                          DIR  s3://metrics-a01-dc1/01FMC5EAT1PZ7MYV27NW4ARXWH/
                          DIR  s3://metrics-a01-dc1/01FMH7CYPE9VTE92HZEG4X76RW/
                          DIR  s3://metrics-a01-dc1/01FMH7D0X271D6RQKZD6ZF2NZ0/
                          DIR  s3://metrics-a01-dc1/01FMH8GZT2BEFTEG9Q6G9P4Y5Z/
                          DIR  s3://metrics-a01-dc1/01FMH9SKTW1QAC9GYVZAJJ9XM8/
                          DIR  s3://metrics-a01-dc1/01FMQC18GABHN44MR8YZQXPF31/
                          DIR  s3://metrics-a01-dc1/01FMQC19237HM06VA4VFC7XNSZ/
                          DIR  s3://metrics-a01-dc1/01FMQCHS28H9QZG6JFZQ2003D4/
                          DIR  s3://metrics-a01-dc1/01FMQCJ51VKZ37CW8W6GTPSC2A/
                          DIR  s3://metrics-a01-dc1/01FMQPJVRFJ2WTEBAJ9R3AAXNS/
                          DIR  s3://metrics-a01-dc1/01FMQQYDP99TT4127KA31ZV9EZ/
                          DIR  s3://metrics-a01-dc1/01FMRFSBQ61KSC3NPKH9YZD9GG/
                          DIR  s3://metrics-a01-dc1/01FMRFSCZZ7192QRXWQCF78057/
                          DIR  s3://metrics-a01-dc1/01FMRPN3338KN1T2W7F2QPN8GM/
                          DIR  s3://metrics-a01-dc1/01FMRPN4BVHFJKBV00H3CYKBHJ/
                          DIR  s3://metrics-a01-dc1/01FMRXGTJKABC2C0ZGR03SPWWZ/
                          DIR  s3://metrics-a01-dc1/01FMRXGVNBHACC13YG6BWCN75H/
                          DIR  s3://metrics-a01-dc1/01FMRXK8XT7574MC51M1DWH2QJ/
                          DIR  s3://metrics-a01-dc1/01FMRXK9NJPKEB3P5XMCDXCKS7/
                          DIR  s3://metrics-a01-dc1/01FMRXXNAGBW67RJHW3N2D0JMD/
                          DIR  s3://metrics-a01-dc1/01FMRXXP4211CQF2PBBVBTZ2BP/
                          DIR  s3://metrics-a01-dc1/01FMS4CHMW7YCK6XF2A4CCCSKX/
                          DIR  s3://metrics-a01-dc1/01FMS4CK54NKYQ8DGKDSTZADY2/
                          DIR  s3://metrics-a01-dc1/01FMSB895KP7MYFPK6AJE2Q49T/
                          DIR  s3://metrics-a01-dc1/01FMSB8A1HQ1QRDXXXBFVQYR0C/
                          DIR  s3://metrics-a01-dc1/01FMSJ3ZHG9Y7H2RWX05X2XA3B/
                          DIR  s3://metrics-a01-dc1/01FMSJ40N8MSNMQJ6AFBZVSVQN/
                          DIR  s3://metrics-a01-dc1/01FMSRZQ2357K745RXT4E7KJBC/
                          DIR  s3://metrics-a01-dc1/01FMSRZRK02ECGN1MDKVSQR7RP/
                          DIR  s3://metrics-a01-dc1/01FMSZVEF94MX21HCG92N1GX60/
                          DIR  s3://metrics-a01-dc1/01FMSZVFXW40XREXGT27JJ9DHB/
                          DIR  s3://metrics-a01-dc1/01FMT6Q5EYSN4KRSFEV6PMYY82/
                          DIR  s3://metrics-a01-dc1/01FMT6Q73Z3HT47V6N0VW4ZTBE/
                          DIR  s3://metrics-a01-dc1/01FMTDJWHPKH1D0ZFT5J31RKS2/
                          DIR  s3://metrics-a01-dc1/01FMTDJXRKET2FQBJECG396M0Z/
                          DIR  s3://metrics-a01-dc1/01FMTMEM0R28H55MT6E02S7CM5/
                          DIR  s3://metrics-a01-dc1/01FMTMENQBJ3DQA2Q8A4R3TA7X/
                          DIR  s3://metrics-a01-dc1/01FMTVABESW364ZYPJQK5CM6CH/
                          DIR  s3://metrics-a01-dc1/01FMTVACCG3J9BZH4J0C2K4PFS/
                          DIR  s3://metrics-a01-dc1/01FMV262KFF8AQWXFDKNF7KDA9/
                          DIR  s3://metrics-a01-dc1/01FMV2640ZVVBFCJH453KW8MV7/
                          DIR  s3://metrics-a01-dc1/01FMV91T4S606CXT43G4PYM64C/
                          DIR  s3://metrics-a01-dc1/01FMV91VNG5SAN583QA6XKVFCK/
                          DIR  s3://metrics-a01-dc1/01FMVFXHDQWGVSRWK0Q05224XB/
                          DIR  s3://metrics-a01-dc1/01FMVFXHXGHPBBA3ANWFK74Q79/
                          DIR  s3://metrics-a01-dc1/01FMVFZT2BEJ5475RBWR5JEWT2/
                          DIR  s3://metrics-a01-dc1/01FMVFZV3YYZKAJA0Z83YNTFKQ/
                          DIR  s3://metrics-a01-dc1/01FMVG8WNPBHFCFQE25HHZB7AE/
                          DIR  s3://metrics-a01-dc1/01FMVG8XBYFQMFXK4DE4GB36QV/
                          DIR  s3://metrics-a01-dc1/01FMVGJBHBQ16QAZG7XNBBYMRJ/
                          DIR  s3://metrics-a01-dc1/01FMVGJCDDXWX64GCZZA0ZVVF0/
                          DIR  s3://metrics-a01-dc1/01FMVPS8PPPKFV66NZAD5ASH86/
                          DIR  s3://metrics-a01-dc1/01FMVPS9WWM7T5MTX1CMVZQ5B7/
                          DIR  s3://metrics-a01-dc1/debug/




Logs

$ thanos tools bucket verify --objstore.config-file=/etc/default/thanos-storage-a01-dc1.yml   --log.level=debug -i index_known_issues
level=debug ts=2021-11-19T10:09:10.197063862Z caller=main.go:65 msg="maxprocs: Leaving GOMAXPROCS=[40]: CPU quota undefined"
level=info ts=2021-11-19T10:09:10.197284456Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2021-11-19T10:09:10.197802453Z caller=verify.go:137 verifiers=index_known_issues msg="Starting verify task"
level=info ts=2021-11-19T10:09:10.19783008Z caller=index_issue.go:32 verifiers=index_known_issues verifier=index_known_issues msg="started verifying issue" with-repair=false
level=debug ts=2021-11-19T10:09:10.197869924Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=info ts=2021-11-19T10:09:10.639636286Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=441.783304ms duration_ms=441 cached=124 returned=124 partial=0
level=debug ts=2021-11-19T10:11:01.458819956Z caller=index_issue.go:63 verifiers=index_known_issues verifier=index_known_issues stats="{TotalSeries:19549086 OutOfOrderSeries:0 OutOfOrderChunks:0 DuplicatedChunks:0 OutsideChunks:0 CompleteOutsideChunks:0 Issue347OutsideChunks:0 OutOfOrderLabels:0 SeriesMinLifeDuration:0s SeriesAvgLifeDuration:29h42m49.606s SeriesMaxLifeDuration:335h28m15.096s SeriesMinLifeDurationWithoutSingleSampleSeries:990ms SeriesAvgLifeDurationWithoutSingleSampleSeries:120h46m11.894s SeriesMaxLifeDurationWithoutSingleSampleSeries:335h28m15.096s SeriesMinChunks:1 SeriesAvgChunks:1 SeriesMaxChunks:5 TotalChunks:23562230 ChunkMinDuration:0s ChunkAvgDuration:24h39m10.429s ChunkMaxDuration:334h59m59.804s ChunkMinSize:171 ChunkAvgSize:7335 ChunkMaxSize:3758100833 SingleSampleSeries:14739304 SingleSampleChunks:14739304 LabelNamesCount:413 MetricLabelValuesCount:2070}" id=01F6QB3YSM5AC9QTW6T9J4RA3F
level=info ts=2021-11-19T10:11:01.875646949Z caller=verify.go:156 verifiers=index_known_issues msg="verify task completed"
level=info ts=2021-11-19T10:11:01.876009574Z caller=main.go:160 msg=exiting
level=debug ts=2021-11-19T10:11:01.87612335Z caller=main.go:65 msg="maxprocs: No GOMAXPROCS change to reset%!(EXTRA []interface {}=[])"



As you can see above , only the block 01F6QB3YSM5AC9QTW6T9J4RA3F is parsed.
Anything else we need to know:

Duplicate issue #4804: Add out-of-order blocks to bucket verify
Created: 2021-10-28T07:35:14Z
Is your proposal related to a problem?
When a Compactor halts due to an out-of-order (OOO) block, there are normally several OOO blocks that need to be tidied up before Compactor can safely run again. At the moment, Compactor only tells me about the first OOO block that it finds - so I need to go through quite a long cycle of identifying a broken block, deleting the broken block, and restarting compactor, multiple times before the storage account is clean.
Describe the solution you'd like
I would like thanos tools bucket verify to check for OOO blocks, and to tell me about all of the ones that it finds.
Describe alternatives you've considered
n.a.
Additional context
n.a.

Closing comment (2021-11-29T16:41:34Z): I've run verify again - the first time I ran it I didn't get a summary at the end, but this time I did. The final output is as follows:
level=debug ts=2021-11-29T15:50:24.250768867Z caller=index_issue.go:64 verifiers=index_known_issues verifier=index_known_issues stats="{TotalSeries:49402 OutOfOrderSeries:0 OutOfOrderChunks:0 DuplicatedChunks:0 OutsideChunks:0 CompleteOutsideChunks:0 Issue347OutsideChunks:0 OutOfOrderLabels:0 SeriesMinLifeDuration:2m44.631s SeriesAvgLifeDuration:1h58m21.913s SeriesMaxLifeDuration:1h59m30.219s SeriesMinLifeDurationWithoutSingleSampleSeries:2m44.631s SeriesAvgLifeDurationWithoutSingleSampleSeries:1h58m21.913s SeriesMaxLifeDurationWithoutSingleSampleSeries:1h59m30.219s SeriesMinChunks:1 SeriesAvgChunks:3 SeriesMaxChunks:4 TotalChunks:187632 ChunkMinDuration:2m44.631s ChunkAvgDuration:31m9.876s ChunkMaxDuration:1h59m0s ChunkMinSize:54 ChunkAvgSize:147 ChunkMaxSize:1251 SingleSampleSeries:0 SingleSampleChunks:0 LabelNamesCount:212 MetricLabelValuesCount:1319}" id=01FN2SC2A7B4K022T6FBWPWTJP

It looks to me like this is only actually verifying a single block, so my issue here is really just the same as #4878. If you agree I'm happy to close this issue in favour of that one.