question: Attribute is reset per batch in `dp` mode
context: ❓ Questions and Help
  Before asking:
  
  search the issues.
  search the docs.
  
  
  What is your question?
  I don't know whether this is a bug...
  As shown in the code below, I think the behavior of dp mode is unexpected? (The attribute is reset every batch)
  When using ddp mode, everything is fine. (The property will be initialized only once per GPU)
  Code
  import os
  import torch
  from torch.nn import functional as F
  from torch.utils.data import DataLoader
  from torchvision.datasets import MNIST
  from torchvision import transforms
  import pytorch_lightning as pl
  from pytorch_lightning import Trainer
  
  from argparse import Namespace
  
  
  class LitModel(pl.LightningModule):
  
      def __init__(self):
          super().__init__()
          self.l1 = torch.nn.Linear(28 * 28, 10)
          self._dummy_property = None
  
      @property
      def dummy_propery(self):
          if self._dummy_property is None:
              self._dummy_property = '*' * 30
              print('print only once per gpu')
          return self._dummy_property
  
      def forward(self, x):
          return torch.relu(self.l1(x.view(x.size(0), -1)))
  
      def training_step(self, batch, batch_idx):
          print(self._dummy_property)
          # Access every batch
          self.dummy_propery
          print(self._dummy_property)
  
          x, y = batch
          y_hat = self(x)
          loss = F.cross_entropy(y_hat, y)
          return pl.TrainResult(loss)
  
      def configure_optimizers(self):
          return torch.optim.Adam(self.parameters(), lr=0.02)
  
  
  train_loader = DataLoader(
      MNIST(
          os.getcwd(),
          download=True,
          transform=transforms.ToTensor()
      ),
      batch_size=128
  )
  trainer = pl.Trainer(gpus=2,
                       distributed_backend='dp',
                       max_epochs=2)
  
  model = LitModel()
  trainer.fit(model, train_loader)
  Output
  None
  print only once per gpu
  ******************************
  None
  print only once per gpu
  ******************************
  None
  print only once per gpu
  ******************************
  None
  print only once per gpu
  ******************************
  ...
  
     
  What's your environment?
  
  Version 0.9.0
answer: oh, this is actually a known problem and comes from DataParallel in PyTorch itself.
  See #565 and #1649 for reference.
  @ananyahjha93 has been working on a workaround but it seems to be super non trivial #1895