Thanos processes cannot successfully restart while another thanos is uploading to bucket storage

This issue documents findings regarding one form of thanos crash loop.  Action items:
* [ ]  Add something like the following notes to the [troubleshooting/prometheus-is-down.md runbook](https://gitlab.com/gitlab-com/runbooks/blob/master/troubleshooting/prometheus-is-down.md), which is referenced by the alert that detects frequent thanos restarts.
* [ ]  File a bug report with the thanos community.  Potentially also work on a patch, if time allows.

----

### Problem statement

Thanos appears to have a race condition, such that whenever a thanos instance is uploading a new block directory to bucket storage, if any other thanos instance restarts for any reason, that instance will fail to startup until the upload completes.

This race condition is a thanos bug.  Automatic restart of the thanos service is the best mitigation we have until this bug is fixed.

Special case: If a thanos upload to the bucket aborts uncleanly, it could leave an incomplete directory in the bucket, which would cause this crash-loop to continue indefinitely.  Fixing that would require manually deleting the incomplete block's directory from the object storage bucket.

### Background

Part of Thanos' job is to use an object storage bucket as the primary place to store metrics data (and use the local filesystem as a persistent cache for that data).  Recently we have noticed the `thanos store` process temporarily entering a crash-loop due to an unhandled exception when scanning that bucket for new `meta.json` files.

Example log excerpt:

```
{
  "ts":"2020-01-10T17:35:29.319245979Z",
  "msg":"syncing blocks failed",
  "err":"iter: load meta: download meta.json: get file 01DY7VTGX0PWPHWWXFNZB6NA17/meta.json: storage: object doesn't exist"
}
```

This log message indicates that the object storage bucket has a "block directory" named `01DY7VTGX0PWPHWWXFNZB6NA17` that did not (yet) contain a `meta.json` file.  In this example, that missing `meta.json` file was created 30 minutes later, when `thanos compact` finished uploading data to that block's `chunks` subdirectory.

In general, when a new "block" (directory) is created in the object storage bucket, that dir's `meta.json` file is created as the last step after uploading all the data files and index file for that block.

Our current version of thanos (`v0.9.0`, the latest release) treats the absence of a `meta.json` file from a block directory as an error.  If thanos is starting up, it's a fatal error (hence the crash loop).  In contrast, if thanos is just doing one of its periodic refreshes of its catalog of bucket contents, then the error is treated as a non-fatal warning.

It can take many minutes between creating a new directory and creating its `meta.json` file, depending on the amount of data to upload and the available network bandwidth.  This is the main window of opportunity for thanos to hit this race condition and enter a crash loop.

### Potential solution

Thanos calls `SyncBlocks` on startup and again periodically.  Only the first call (via the `InitialSync` method) treats the absence of a `meta.json` file as fatal.  In contrast, the periodic calls to `SyncBlocks` just logs a warning and stops processing the bucket's remaining dirs.

If `SyncBlocks` skipped dirs that exist but do not yet have a `meta.json` file, this would:
* Prevent `InitialSync` from failing and causing the thanos process to exit shortly after startup (i.e. avoids the crash loop).
* Allow the periodic `SyncBlocks` calls to scan all viable bucket dirs, instead of aborting prematurely.  This should result in fresh data being more reliably available even during slow compaction runs.

Story points: 2.0