Logging validation metrics with `val_check_interval`
❓ Questions and Help

What is your question?
My training epoch is very long, so I'd like to run mini-validation loops in the middle of the epoch, which I can do easily by setting val_check_interval=N. However, I'd also like to log the values outputted by those mini-validation loops.
Is there a canonical way to log validation metrics within a training epoch while using val_check_interval in the Trainer?
Related issues:
#4980 -- However, I'm not trying to log step-wise validation metrics; I'm trying to log aggregate validation epoch metrics.
#4985 -- Similar, but I'm using the WandbLogger directly while this issue is more general.
#4652 -- Similar issue, but this is an older version of pytorch-lightning.
Code
The below is my code for the methods that involve logging. Unfortunately, I can't post the full repo; here's the relevant configuration (loaded from YAML) for the Trainer and Logger objects, which I've replicated in a Colab in the below comment.
trainer:
    gradient_clip_val: 0.1
    accumulate_grad_batches: 8
    num_sanity_val_steps: 2
    gpus: 1
    max_epochs: 1
    val_check_interval: 2000
    limit_val_batches: 100
    move_metrics_to_cpu: True
logging:
    save_dir: "sentcomp_logs/"
    project: "sent-comp"
    offline: False
    name: "bart_full_run"

The following is the actual logging code. For context, I am training a Seq2Seq model from HuggingFace.
# THIS WORKS FINE!
    def training_step(self, batch: BatchEncoding, batch_idx: int) -> torch.Tensor:
        outputs = self(**batch)
        loss = outputs[0]
        self.log("train_loss", loss, prog_bar=True, sync_dist=True)
        return loss

# DOES NOT WORK
    def validation_step(self, batch: BatchEncoding, batch_idx: int) -> Dict[str, torch.Tensor]:
        outputs = self(**batch)
        val_loss = outputs[0]
        predictions, targets = self.decode(batch) # returns lists of strings
        results = self.score(predictions, targets) # returns a dictionary
        self.log("val_loss", val_loss, sync_dist=True)
        self.log_dict(results, sync_dist=True)
        return results

Expected: Logs to my wandb project.
Actual: A warning wandb: WARNING Step must only increase in log calls.
It seems like the global step of  the validator is out of sync with my logging calls, as I get a message like
Step 249 < 2000; dropping { <my metrics> }
Step 499 < 4000; dropping { <my metrics> }
Step 749 < 6000; dropping { <my metrics> }

The global step numbers are definitely consistent with the val_check_interval provided (2000 * n). However, I am unsure why the validation logging step results in an attempt to log to steps in increments of 250. Maybe I need to somehow sync the validation logging step to the training logging step? Or is pytorch lightning supposed to take care of this interally?
Is this a bug, or am I doing something wrong/silly?
What have you tried?
Removing the log_dict call from validation steps and instead logging an aggregated dictionary in validation_epoch_end:
def validation_epoch_end(self, batches: List[Dict[str, torch.Tensor]]) -> None:
    metrics = {}         
    metrics[metric_key] = torch.stack([batch[metric_key] for batch in batches]).mean()  
    self.log_dict(metrics, sync_dist=True)

This does not change the overall behavior.
What's your environment?

OS: Ubuntu 18.04
Packaging: pip
Version: pytorch-lightning 1.1.0 (Master), pytorch 1.7.0, cuda 11.0