question: Calculation of inference time
context: Hi . I want to calculate the inference time of my model. I am not sure where to the code for measuring the time. I thought its better to do it inside predict_step of the LightningModule. Something like this
  class LitModule(pl.LightningModule):
          def __init__(self, hparams):
                 ....
                 self.starter, self.ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
                 .....
          ......
          ......
           def predict_step(self, batch, batch_idx, dataloader_idx=None):
                   torch.cuda.empty_cache()
                   minibatch_model_in, _ = batch
                   self.starter.record()
                   _ = self(minibatch_model_in)
                   self.ender.record()
                   # wait for gpu sync
                   torch.cuda.synchronize()
                   inference_time = self.starter.elapsed_time(self.ender)
                   return inference_time*1e-3
  
  And in the main funtion,
  inference_metrics = trainer.predict(model=pl_model, datamodule=pl_data)
  
  After removing the initial measurements (considering GPU warm-up) and taking mean of 200 samples, I get 0.0196 seconds.
  If I do the measurement outside the LightningModule then I get a different value. This is how I measured
  # Inferencing
  pl_data.setup()
  # Initializing cuda event loggers
  starter, ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
  metrics = []
  pl_model.eval()
  with torch.no_grad():
  for batch in pl_data.train_dataloader():
        torch.cuda.empty_cache()
        minibatch_model_in, _, _, _, _, _ = batch
        starter.record()
          _, _, _ = pl_model(minibatch_model_in)
        # wait for gpu sync
         torch.cuda.synchronize()
         curr_time = starter.elapsed_time(ender)
         metrics.append(curr_time)
  
  Using the above code, I get 0.027 seconds. Please someone tell me if I am doing something wrong. Which method is correct? Why is there a decrease in the inference time when measured within predict_step?
answer: Hi, the first snippet measures raw prediction time, whereas the second one doesn't seem to be on gpu at all (e.g. your data likely isn't on gpu) or (if it is) can already include the data preparation and host to device transfers.