Repository: Lightning-AI/pytorch-lightning
Pull request: Modify LSFEnvironment to use more reliable environment variable
The environment variable `LSB_HOSTS` was being used to determine node rank and master address. This is not always populated. This PR updates LSFEnvironment to use `LSB_DJOB_RANKFILE` instead.

Fix #10216 

<!-- FILL IN or None -->

## Before submitting

- [x] Was this **discussed/approved** via a GitHub issue? (not for typos and docs)
- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [x] Did you make sure your **PR does only one thing**, instead of bundling different changes together?
- [x] Did you make sure to **update the documentation** with your changes? (if necessary)
- [x] Did you write any **new necessary tests**? (not for typos and docs)
- [x] Did you verify new and **existing tests pass** locally with your changes?
- [x] Did you list all the **breaking changes** introduced by this pull request?
- [x] Did you **update the [CHANGELOG](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/CHANGELOG.md)**? (not for typos, docs, test updates, or internal minor changes/refactorings)

<!-- In the CHANGELOG, separate each item in the unreleased section by a blank line to reduce collisions -->

## PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read [Review guidelines](https://github.com/PyTorchLightning/pytorch-lightning/wiki/Review-guidelines). In short, see the following bullet-list:

- [ ] Is this pull request ready for review? (if not, please submit in draft mode)
- [x] Check that all items from **Before submitting** are resolved
- [x] Make sure the title is self-explanatory and the description concisely explains the PR
- [ ] Add labels and milestones (and optionally projects) to the PR so it can be classified

## Did you have fun?

Make sure you had fun coding 🙃


Problem statement:
DDP initialization breaks on HPC system at larger scale
We are experiencing an issue with DDP at larger scales on our HPC system (Summit at OLCF - LSF scheduler). The specific threshold is at 14 nodes, where things suddenly aren't able to initialize anymore. It appears there is all of a sudden an inability to setup ranks across nodes properly, as depicted below in the output.

Each node as 6 GPUs. So, in total, we are trying to use 84 GPUs when things are suddenly unable to initialize. At 78 GPUs (or 13 nodes) things work as expected. 

Initialization output at 13 nodes (78 GPUs - this works as expected):

```
initializing ddp: GLOBAL_RANK: 72, MEMBER: 73/78
initializing ddp: GLOBAL_RANK: 36, MEMBER: 37/78
initializing ddp: GLOBAL_RANK: 10, MEMBER: 11/78
initializing ddp: GLOBAL_RANK: 19, MEMBER: 20/78
initializing ddp: GLOBAL_RANK: 32, MEMBER: 33/78
initializing ddp: GLOBAL_RANK: 12, MEMBER: 13/78
initializing ddp: GLOBAL_RANK: 41, MEMBER: 42/78
initializing ddp: GLOBAL_RANK: 74, MEMBER: 75/78
initializing ddp: GLOBAL_RANK: 27, MEMBER: 28/78
initializing ddp: GLOBAL_RANK: 21, MEMBER: 22/78
initializing ddp: GLOBAL_RANK: 13, MEMBER: 14/78
initializing ddp: GLOBAL_RANK: 73, MEMBER: 74/78
initializing ddp: GLOBAL_RANK: 25, MEMBER: 26/78
initializing ddp: GLOBAL_RANK: 24, MEMBER: 25/78
initializing ddp: GLOBAL_RANK: 28, MEMBER: 29/78
initializing ddp: GLOBAL_RANK: 67, MEMBER: 68/78
initializing ddp: GLOBAL_RANK: 69, MEMBER: 70/78
initializing ddp: GLOBAL_RANK: 68, MEMBER: 69/78
initializing ddp: GLOBAL_RANK: 47, MEMBER: 48/78
initializing ddp: GLOBAL_RANK: 56, MEMBER: 57/78
initializing ddp: GLOBAL_RANK: 71, MEMBER: 72/78
initializing ddp: GLOBAL_RANK: 53, MEMBER: 54/78
initializing ddp: GLOBAL_RANK: 66, MEMBER: 67/78
initializing ddp: GLOBAL_RANK: 70, MEMBER: 71/78
initializing ddp: GLOBAL_RANK: 55, MEMBER: 56/78
initializing ddp: GLOBAL_RANK: 49, MEMBER: 50/78
initializing ddp: GLOBAL_RANK: 44, MEMBER: 45/78
initializing ddp: GLOBAL_RANK: 46, MEMBER: 47/78
initializing ddp: GLOBAL_RANK: 57, MEMBER: 58/78
initializing ddp: GLOBAL_RANK: 51, MEMBER: 52/78
initializing ddp: GLOBAL_RANK: 45, MEMBER: 46/78
initializing ddp: GLOBAL_RANK: 54, MEMBER: 55/78
initializing ddp: GLOBAL_RANK: 50, MEMBER: 51/78
initializing ddp: GLOBAL_RANK: 59, MEMBER: 60/78
initializing ddp: GLOBAL_RANK: 52, MEMBER: 53/78
initializing ddp: GLOBAL_RANK: 43, MEMBER: 44/78
initializing ddp: GLOBAL_RANK: 58, MEMBER: 59/78
initializing ddp: GLOBAL_RANK: 42, MEMBER: 43/78
initializing ddp: GLOBAL_RANK: 48, MEMBER: 49/78
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All DDP processes registered. Starting ddp with 78 processes
```

Failed initialization at 14 nodes (84 GPUs - this hangs at this point):

```
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/84
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/84
```

Here is the code:

```
class LitAutoEncoder(pl.LightningModule):

    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(28*28, 64),
            nn.ReLU(),
            nn.Linear(64, 3)
        )
        self.decoder = nn.Sequential(
            nn.Linear(3, 64),
            nn.ReLU(),
            nn.Linear(64, 28*28)
        )

    def forward(self, x):
        # in lightning, forward defines the prediction/inference actions
        embedding = self.encoder(x)
        return embedding

    def training_step(self, batch, batch_idx):
        # training_step defined the train loop.
        # It is independent of forward
        x, y = batch
        x = x.view(x.size(0), -1)
        z = self.encoder(x)
        x_hat = self.decoder(z)
        loss = F.mse_loss(x_hat, x)

        # Logging to TensorBoard by default
        self.log('train_loss', loss)
        return loss

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
        return optimizer


# define datasets/dataloaders
dataset = MNIST(os.getcwd(), download=True, transform=transforms.ToTensor())
train_dl = DataLoader(dataset)


# train
model = LitAutoEncoder()
trainer = pl.Trainer(gpus="0,1,2,3,4,5", auto_select_gpus=True, num_nodes=14, max_epochs=3, accelerator='ddp')
trainer.fit(model, train_dl)
```

- PyTorch Lightning Version (e.g., 1.3.0): `1.4.9`
- PyTorch Version (e.g., 1.8): `1.9`
- Python version: `3.7`
- OS (e.g., Linux):
- CUDA/cuDNN version:
- GPU models and configuration:
- How you installed PyTorch (`conda`, `pip`, source): `source`
- If compiling from source, the output of `torch.__config__.show()`:
```
'PyTorch built with:\n  - GCC 7.3\n  - C++ Version: 201402\n  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n  - CPU capability usage: VSX\n  - CUDA Runtime 11.0\n  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80\n  - CuDNN 8.1.1  (built against CUDA 11.2)\n  - Magma 2.5.4\n  - Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CUDA_VERSION=11.0, CUDNN_VERSION=8.1.1, CXX_COMPILER=/gpfs/alpine/world-shared/stf007/davismj/open-ce-builds/rhel8-oce-1.4.0/python-env/conda-bld/pytorch-base_1633116212289/_build_env/bin/powerpc64le-conda_cos7-linux-gnu-c++, CXX_FLAGS=-fvisibility-inlines-hidden -fmessage-length=0 -mcpu=power8 -mtune=power8 -mpower8-fusion -mpower8-vector -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -I/gpfs/alpine/world-shared/stf007/davismj/open-ce-builds/rhel8-oce-1.4.0/python-env/conda-bld/pytorch-base_1633116212289/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include -fdebug-prefix-map=/gpfs/alpine/world-shared/stf007/davismj/open-ce-builds/rhel8-oce-1.4.0/python-env/conda-bld/pytorch-base_1633116212289/work=/usr/local/src/conda/pytorch-base-1.9.0 -fdebug-prefix-map=/gpfs/alpine/world-shared/stf007/davismj/open-ce-builds/rhel8-oce-1.4.0/python-env/conda-bld/pytorch-base_1633116212289/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol=/usr/local/src/conda-prefix -D__STDC_FORMAT_MACROS -I/gpfs/alpine/world-shared/stf007/davismj/open-ce-builds/rhel8-oce-1.4.0/python-env/conda-bld/pytorch-base_1633116212289/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include -I/sw/peak/cuda/11.0.3/include -I/gpfs/alpine/world-shared/stf007/davismj/open-ce-builds/rhel8-oce-1.4.0/python-env/conda-bld/pytorch-base_1633116212289/_build_env/include -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=open, TORCH_VERSION=1.9.0, USE_CUDA=1, USE_CUDNN=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKLDNN=OFF, USE_MPI=0, USE_NCCL=ON, USE_NNPACK=0, USE_OPENMP=1,
```


cc @awaelchli @rohitgr7

Hints from issue discussion:
@proutrc Hi, thanks for reporting.

You mention LSF. Could you verify that `trainer.training_type_plugin.cluster_environment` prints LSFEnvironment? If that's the case, what will `trainer.training_type_plugin.cluster_environment.global_rank()` print for you? It should be determined by the env variable `JSM_NAMESPACE_RANK`. 
@awaelchli something is indeed happening when going from 13 to 14 nodes on our system. See below:

13 nodes snippet (what we expect):
```
...
<pytorch_lightning.plugins.environments.lsf_environment.LSFEnvironment object at 0x2000a51c7650>
global_rank: 41
world_size: 78
<pytorch_lightning.plugins.environments.lsf_environment.LSFEnvironment object at 0x2000a51cc4d0>
global_rank: 56
world_size: 78
<pytorch_lightning.plugins.environments.lsf_environment.LSFEnvironment object at 0x2000a51c6810>
global_rank: 34
world_size: 78
<pytorch_lightning.plugins.environments.lsf_environment.LSFEnvironment object at 0x2000a4cb52d0>
global_rank: 25
world_size: 78
...
```

14 nodes snippet:
```
...
<pytorch_lightning.plugins.environments.lightning_environment.LightningEnvironment object at 0x2000a51bead0>
global_rank: 0
world_size: 84
<pytorch_lightning.plugins.environments.lightning_environment.LightningEnvironment object at 0x2000a51c6510>
global_rank: 0
...
```

At 14 nodes it shows `LightningEnvironment` instead of `LSFEnvironment`. In addition, everything is `global_rank:0`.. it does have the correct `world_size` though (84 for 14 nodes).
It looks like `LSB_HOSTS` is not always set, to show the list of hosts. It appears there could be a limit to the size of the list, where it stops using that variable at a certain size. I think we are hitting this limit.

I can confirm I see `LSB_HOSTS` when I use 13 nodes, but not when I use 14 nodes.

The variable to use is perhaps `LSB_MCPU_HOSTS`, instead of `LSB_HOSTS`.

Somewhat dated reference, but I bet it is still relative: https://www.ibm.com/support/pages/limitation-environment-variable-lsbhosts 
Thank you @proutrc for helping out here.
Right, given your hint, I'm finding the same, bit more detailed, information on the docs here: 
https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=variables-environment-variable-reference
(scroll down to the section for LSB_MCPU_HOSTS)

Quote:

> The environment variables LSB_HOSTS and LSB_MCPU_HOSTS both contain the same information, but the information is presented in different formats. LSB_MCPU_HOSTS uses a shorter format than LSB_HOSTS. As a general rule, sbatchd sets both these variables. However, for some parallel jobs, LSB_HOSTS is not set.
> 
> For parallel jobs, several CPUs are used, and the length of LSB_HOSTS can become very long. sbatchd needs to spend 
> a lot of time parsing the string. If the size of LSB_HOSTS exceeds 4096 bytes, LSB_HOSTS is ignored, and sbatchd sets only LSB_MCPU_HOSTS.
> 
> To verify the hosts and CPUs used for your dispatched job, check the value of LSB_HOSTS for single CPU jobs, and check the value of LSB_MCPU_HOSTS for parallel jobs.
> 


Their example there shows the format:

```
LSB_HOSTS= "hostA hostA hostA hostB hostB hostB"
LSB_MCPU_HOSTS="hostA 3 hostB 3" 
```

So, should we default to LSB_HOSTS in our plugin and if not set try to get the hosts from LSB_MCPU_HOSTS instead? When this happens, we can log a debug message. The fact that the variable contains the word "CPU" is a bit strange but I think for us this does not matter.
Could you help me verify that this fixes the issue by creating this custom cluster env:

```python
from pytorch_lightning.plugins.environments import ClusterEnvironment


class NewLSFEnvironment(LSFEnvironment):

    @staticmethod
    def is_using_lsf() -> bool:
        required_env_vars = ("LSB_JOBID", "LSB_MCPU_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE")
        return all(v in os.environ for v in required_env_vars)

    @staticmethod
    def _read_hosts():
        hosts_config = os.environ.get("LSB_MCPU_HOSTS", "")
        if not hosts_config:
            raise ValueError("Could not find hosts in environment variable LSB_MCPU_HOSTS")
        host_config = hosts_config.split()

        if len(host_config) % 2 != 0:
            raise ValueError(
                "Cannot parse hosts from LSB_MCPU_HOSTS environment variable. Expected format:"
                ' "<node0_name> <node0_num_procs> <node1_name> ..."'
            )
        return host_config[::2]

    def _get_master_address(self):
        return self._read_hosts()[0]
```

and adding it to your trainer like so:

```python
trainer = Trainer(plugins=NewLSFEnvironment())
```

The main change I made is switch to that env variable and adjusted the parsing.
I simulated it but can't run it for real. If you confirm, I can create a fix directly from this.

If things break down on your side, could you let me know the values of the two environment variables LSB_MCPU_HOSTS and LSB_HOSTS.
> Could you help me verify that this fixes the issue by creating this custom cluster env:
> 
> ```python
> from pytorch_lightning.plugins.environments import ClusterEnvironment
> 
> 
> class NewLSFEnvironment(LSFEnvironment):
> 
>     @staticmethod
>     def is_using_lsf() -> bool:
>         required_env_vars = ("LSB_JOBID", "LSB_MCPU_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE")
>         return all(v in os.environ for v in required_env_vars)
> 
>     @staticmethod
>     def _read_hosts():
>         hosts_config = os.environ.get("LSB_MCPU_HOSTS", "")
>         if not hosts_config:
>             raise ValueError("Could not find hosts in environment variable LSB_MCPU_HOSTS")
>         host_config = hosts_config.split()
> 
>         if len(host_config) % 2 != 0:
>             raise ValueError(
>                 "Cannot parse hosts from LSB_MCPU_HOSTS environment variable. Expected format:"
>                 ' "<node0_name> <node0_num_procs> <node1_name> ..."'
>             )
>         return host_config[::2]
> 
>     def _get_master_address(self):
>         return self._read_hosts()[0]
> ```
> 
> and adding it to your trainer like so:
> 
> ```python
> trainer = Trainer(plugins=NewLSFEnvironment())
> ```
> 
> The main change I made is switch to that env variable and adjusted the parsing. I simulated it but can't run it for real. If you confirm, I can create a fix directly from this.
> 
> If things break down on your side, could you let me know the values of the two environment variables LSB_MCPU_HOSTS and LSB_HOSTS.

When trying this override in my small program (listed above) it complains about LSFEnvironment not being defined. I thought maybe you meant `import LSFEnviornment` at the top, instead of `import ClusterEnvironment`,  but that produces another error. At any rate, I could be doing something silly (likely perhaps), but can you provide a full example of this override?
Thank you very much for trying and the patience.
Here is the full example (with correct imports, sorry about that :)) and with your code combined:

```python

import os

import torch
import torch.nn as nn
import torch.nn.functional as F
import pytorch_lightning as pl
from torch.utils.data import DataLoader
from torchvision.transforms import transforms

from pl_examples.basic_examples.mnist_datamodule import MNIST
from pytorch_lightning.plugins import DDPPlugin


class LitAutoEncoder(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(28 * 28, 64), nn.ReLU(), nn.Linear(64, 3))
        self.decoder = nn.Sequential(nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 28 * 28))

    def forward(self, x):
        # in lightning, forward defines the prediction/inference actions
        embedding = self.encoder(x)
        return embedding

    def training_step(self, batch, batch_idx):
        # training_step defined the train loop.
        # It is independent of forward
        x, y = batch
        x = x.view(x.size(0), -1)
        z = self.encoder(x)
        x_hat = self.decoder(z)
        loss = F.mse_loss(x_hat, x)

        # Logging to TensorBoard by default
        self.log("train_loss", loss)
        return loss

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
        return optimizer


from pytorch_lightning.plugins.environments import LSFEnvironment


class NewLSFEnvironment(LSFEnvironment):
    @staticmethod
    def is_using_lsf() -> bool:
        required_env_vars = ("LSB_JOBID", "LSB_MCPU_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE")
        return all(v in os.environ for v in required_env_vars)

    @staticmethod
    def _read_hosts():
        hosts_config = os.environ.get("LSB_MCPU_HOSTS", "")
        if not hosts_config:
            raise ValueError("Could not find hosts in environment variable LSB_MCPU_HOSTS")
        host_config = hosts_config.split()

        if len(host_config) % 2 != 0:
            raise ValueError(
                "Cannot parse hosts from LSB_MCPU_HOSTS environment variable. Expected format:"
                ' "<node0_name> <node0_num_procs> <node1_name> ..."'
            )
        return host_config[::2]

    def _get_master_address(self):
        return self._read_hosts()[0]


if __name__ == "__main__":
    # define datasets/dataloaders
    dataset = MNIST(os.getcwd(), download=True, transform=transforms.ToTensor())
    train_dl = DataLoader(dataset)

    # simulate locally
    # os.environ["LSB_JOBID"] = "1234"
    # os.environ["LSB_MCPU_HOSTS"] = "localhost 2 localhost 2"
    # os.environ["JSM_NAMESPACE_LOCAL_RANK"] = "1"
    # os.environ["JSM_NAMESPACE_RANK"] = "1"
    # os.environ["JSM_NAMESPACE_SIZE"] = "4"

    # train
    model = LitAutoEncoder()

    # simulate locally
    # trainer = pl.Trainer(
    #     num_processes=2,
    #     num_nodes=2,
    #     max_epochs=3,
    #     plugins=DDPPlugin(cluster_environment=NewLSFEnvironment()),
    # )

    trainer = pl.Trainer(
        gpus="0,1,2,3,4,5",
        auto_select_gpus=True,
        num_nodes=14,
        max_epochs=3,
        plugins=DDPPlugin(cluster_environment=NewLSFEnvironment()),
    )
    print("LSB_MCPU_HOSTS:", os.environ["LSB_MCPU_HOSTS"])
    trainer.fit(model, train_dl)

```


> Thank you very much for trying and the patience. Here is the full example (with correct imports, sorry about that :)) and with your code combined:
> 
> ```python
> import os
> 
> import torch
> import torch.nn as nn
> import torch.nn.functional as F
> import pytorch_lightning as pl
> from torch.utils.data import DataLoader
> from torchvision.transforms import transforms
> 
> from pl_examples.basic_examples.mnist_datamodule import MNIST
> from pytorch_lightning.plugins import DDPPlugin
> 
> 
> class LitAutoEncoder(pl.LightningModule):
>     def __init__(self):
>         super().__init__()
>         self.encoder = nn.Sequential(nn.Linear(28 * 28, 64), nn.ReLU(), nn.Linear(64, 3))
>         self.decoder = nn.Sequential(nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 28 * 28))
> 
>     def forward(self, x):
>         # in lightning, forward defines the prediction/inference actions
>         embedding = self.encoder(x)
>         return embedding
> 
>     def training_step(self, batch, batch_idx):
>         # training_step defined the train loop.
>         # It is independent of forward
>         x, y = batch
>         x = x.view(x.size(0), -1)
>         z = self.encoder(x)
>         x_hat = self.decoder(z)
>         loss = F.mse_loss(x_hat, x)
> 
>         # Logging to TensorBoard by default
>         self.log("train_loss", loss)
>         return loss
> 
>     def configure_optimizers(self):
>         optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
>         return optimizer
> 
> 
> from pytorch_lightning.plugins.environments import LSFEnvironment
> 
> 
> class NewLSFEnvironment(LSFEnvironment):
>     @staticmethod
>     def is_using_lsf() -> bool:
>         required_env_vars = ("LSB_JOBID", "LSB_MCPU_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE")
>         return all(v in os.environ for v in required_env_vars)
> 
>     @staticmethod
>     def _read_hosts():
>         hosts_config = os.environ.get("LSB_MCPU_HOSTS", "")
>         if not hosts_config:
>             raise ValueError("Could not find hosts in environment variable LSB_MCPU_HOSTS")
>         host_config = hosts_config.split()
> 
>         if len(host_config) % 2 != 0:
>             raise ValueError(
>                 "Cannot parse hosts from LSB_MCPU_HOSTS environment variable. Expected format:"
>                 ' "<node0_name> <node0_num_procs> <node1_name> ..."'
>             )
>         return host_config[::2]
> 
>     def _get_master_address(self):
>         return self._read_hosts()[0]
> 
> 
> if __name__ == "__main__":
>     # define datasets/dataloaders
>     dataset = MNIST(os.getcwd(), download=True, transform=transforms.ToTensor())
>     train_dl = DataLoader(dataset)
> 
>     # simulate locally
>     # os.environ["LSB_JOBID"] = "1234"
>     # os.environ["LSB_MCPU_HOSTS"] = "localhost 2 localhost 2"
>     # os.environ["JSM_NAMESPACE_LOCAL_RANK"] = "1"
>     # os.environ["JSM_NAMESPACE_RANK"] = "1"
>     # os.environ["JSM_NAMESPACE_SIZE"] = "4"
> 
>     # train
>     model = LitAutoEncoder()
> 
>     # simulate locally
>     # trainer = pl.Trainer(
>     #     num_processes=2,
>     #     num_nodes=2,
>     #     max_epochs=3,
>     #     plugins=DDPPlugin(cluster_environment=NewLSFEnvironment()),
>     # )
> 
>     trainer = pl.Trainer(
>         gpus="0,1,2,3,4,5",
>         auto_select_gpus=True,
>         num_nodes=14,
>         max_epochs=3,
>         plugins=DDPPlugin(cluster_environment=NewLSFEnvironment()),
>     )
>     print("LSB_MCPU_HOSTS:", os.environ["LSB_MCPU_HOSTS"])
>     trainer.fit(model, train_dl)
> ```

It looks like we are switching to the DDPPlugin now, ` plugins=DDPPlugin(cluster_environment=NewLSFEnvironment())` , which is causing other types of issues for me.

Error with DDPPlugin method:

```
pytorch_lightning.utilities.exceptions.MisconfigurationException: Horovod does not support setting num_nodes / num_gpus explicitly. Use horovodrun / mpirun to configure the number of processes.
Traceback (most recent call last):
  File "auto-test.py", line 99, in <module>
    plugins=DDPPlugin(cluster_environment=NewLSFEnvironment()),
  File "/gpfs/alpine/stf007/scratch/rprout/deep-learning-hpc-project-template/torch-1.9/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/env_vars_connector.py", line 40, in insert_env_defaults
    return fn(self, **kwargs)
  File "/gpfs/alpine/stf007/scratch/rprout/deep-learning-hpc-project-template/torch-1.9/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 371, in __init__
    plugins,
  File "/gpfs/alpine/stf007/scratch/rprout/deep-learning-hpc-project-template/torch-1.9/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 152, in __init__
    self.set_distributed_mode()
  File "/gpfs/alpine/stf007/scratch/rprout/deep-learning-hpc-project-template/torch-1.9/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 740, in set_distributed_mode
    self._set_horovod_backend()
  File "/gpfs/alpine/stf007/scratch/rprout/deep-learning-hpc-project-template/torch-1.9/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 821, in _set_horovod_backend
    self.check_horovod()
  File "/gpfs/alpine/stf007/scratch/rprout/deep-learning-hpc-project-template/torch-1.9/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 856, in check_horovod
    "Horovod does not support setting num_nodes / num_gpus explicitly. Use "
```
This seems mostly explanatory. Just wanted to confirm this is the recommended path, as it appears to be changing some underlying things (i.e. `Horovod does not support setting num_nodes / num_gpus explicitly`). I can of course try without `num_nodes`. Just not sure I understand _all_ the implied changes here yet. 

On another note, when I do this (not using DDPPlugin) it hangs and never runs the model (but it does seem to initialize all ranks on the GPUs beyond our previous limit):

```
trainer = pl.Trainer(
    gpus="0,1,2,3,4,5",
    auto_select_gpus=True,
    num_nodes=14,
    max_epochs=3,
    accelerator='ddp',
    plugins=[NewLSFEnvironment()],
)
```
If the latter method, without `DDPPlugin`, is suppose to work we may have an issue still. If I need to switch to `DDPlugin` method, I can continue down that path. Just wanted to confirm these changes and what is implied. 

Thanks!
@awaelchli Hopefully my last question made sense. 

Ultimately, it appeared the new env variable and parsing worked. I could see it seemingly initialize the GPUs if I _don't_ use the `DDPPlugin`, but the model never runs (just hangs there).

The switching to the `DDPlugin` caused some different behavior (seems like it doesn't like the trainer setup, as it is, with this plugin).  

Please let me know if I can do or provide anything else. 
@proutrc apologies for the delay. I'm struggling to simulate what you are seeing because I can't try it on a LSF cluster myself.

@ajtritt who is the original contributor of the LSF Environment plugin, could you review the changes in my PR over here #10354 and if you still have access to LSF maybe verify if I broke anything, that would be very appreciated.

Commit message: [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Patch to review:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c2b50c1d41f5b..cfc8edbd60927 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -71,6 +71,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Moved `batch_to_device` method from `Accelerator` to `TrainingTypePlugin` ([#10649](https://github.com/PyTorchLightning/pytorch-lightning/pull/10649))
 
 
+- Change `LSFEnvironment` to use `LSB_DJOB_RANKFILE` environment variable instead of `LSB_HOSTS` for determining node rank and master address ([#10825](https://github.com/PyTorchLightning/pytorch-lightning/pull/10825))
+
 -
 
 ### Deprecated
diff --git a/pytorch_lightning/plugins/environments/lsf_environment.py b/pytorch_lightning/plugins/environments/lsf_environment.py
index c25d068ae01bb..5e1f514a3174d 100644
--- a/pytorch_lightning/plugins/environments/lsf_environment.py
+++ b/pytorch_lightning/plugins/environments/lsf_environment.py
@@ -24,21 +24,24 @@ class LSFEnvironment(ClusterEnvironment):
     """An environment for running on clusters managed by the LSF resource manager.
 
     It is expected that any execution using this ClusterEnvironment was executed
-    using the Job Step Manager i.e. ``jsrun``.
+    using the Job Step Manager i.e. jsrun.
 
-    This plugin expects the following environment variables.
+    This plugin expects the following environment variables:
 
-    LSB_JOBID:
-        The LSF assigned job ID
+    LSB_JOBID
+      The LSF assigned job ID
 
-    LSB_HOSTS:
-        The hosts used in the job. This string is expected to have the format "batch <rank_0_host> ...."
+    LSB_DJOB_RANKFILE
+      The OpenMPI compatibile rank file for the LSF job
 
-    JSM_NAMESPACE_LOCAL_RANK:
-        The node local rank for the task. This environment variable is set by jsrun
+    JSM_NAMESPACE_LOCAL_RANK
+      The node local rank for the task. This environment variable is set by jsrun
 
-    JSM_NAMESPACE_SIZE:
-        The world size for the task. This environment variable is set by jsrun
+    JSM_NAMESPACE_SIZE
+      The world size for the task. This environment variable is set by jsrun
+
+    JSM_NAMESPACE_RANK
+      The global rank for the task. This environment variable is set by jsrun
     """
 
     def __init__(self):
@@ -51,73 +54,102 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
+        os.environ["MASTER_ADDR"] = str(self._main_address)
+        log.debug(f"MASTER_ADDR: {os.environ['MASTER_ADDR']}")
+        os.environ["MASTER_PORT"] = str(self._main_port)
+        log.debug(f"MASTER_PORT: {os.environ['MASTER_PORT']}")
+
+        tmp = ("main_address", "main_port", "world_size", "local_rank", "node_rank", "global_rank")
+        self._rep = ",".join("{}={}".format(s, getattr(self, "_" + s)) for s in tmp)
+
+    def _read_hosts(self):
+        var = "LSB_DJOB_RANKFILE"
+        try:
+            rankfile = os.environ[var]
+        except KeyError:
+            raise ValueError("Could not find environment variable LSB_DJOB_RANKFILE")
+        if not rankfile:
+            raise ValueError("Environment variable LSB_DJOB_RANKFILE is empty")
+        with open(rankfile) as f:
+            ret = [line.strip() for line in f]
+        return ret
+
+    def _get_main_address(self):
+        """A helper for getting the master address."""
+        hosts = self._read_hosts()
+        return hosts[1]
 
-    @staticmethod
-    def detect() -> bool:
-        """Returns ``True`` if the current process was launched using the jsrun command."""
-        required_env_vars = {"LSB_JOBID", "LSB_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE"}
-        return required_env_vars.issubset(os.environ.keys())
+    def _get_main_port(self):
+        """A helper for getting the master port.
 
-    def world_size(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_SIZE`."""
-        var = "JSM_NAMESPACE_SIZE"
-        world_size = os.environ.get(var)
-        if world_size is None:
-            raise ValueError(
-                f"Cannot determine world size from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
-            )
-        return int(world_size)
+        Use the LSF job ID so all ranks can compute the master port
+        """
+        # check for user-specified master port
+        port = os.environ.get("MASTER_PORT")
+        if not port:
+            var = "LSB_JOBID"
+            jobid = os.environ.get(var)
+            if not jobid:
+                raise ValueError("Could not find job id -- expected in environment variable %s" % var)
+            else:
+                port = int(jobid)
+                # all ports should be in the 10k+ range
+                port = int(port) % 1000 + 10000
+            log.debug("calculated master port")
+        else:
+            log.debug("using externally specified master port")
+        return port
 
-    def set_world_size(self, size: int) -> None:
-        log.debug("LSFEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.")
+    def _get_global_rank(self):
+        """A helper function for getting the global rank.
 
-    def global_rank(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_RANK"
         global_rank = os.environ.get(var)
         if global_rank is None:
             raise ValueError(
-                f"Cannot determine global rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine global rank -- expected in %s "
+                "-- make sure you run your executable with jsrun" % var
             )
         return int(global_rank)
 
-    def set_global_rank(self, rank: int) -> None:
-        log.debug("LSFEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.")
+    def _get_local_rank(self):
+        """A helper function for getting the local rank.
 
-    def local_rank(self):
-        """The local rank is read from the environment variable `JSM_NAMESPACE_LOCAL_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_LOCAL_RANK"
         local_rank = os.environ.get(var)
         if local_rank is None:
             raise ValueError(
-                f"Cannot determine local rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine local rank -- expected in %s " "-- make sure you run your executable with jsrun" % var
             )
         return int(local_rank)
 
-    def node_rank(self):
-        """The node rank is determined by the position of the current hostname in the list of hosts stored in the
-        environment variable `LSB_HOSTS`."""
+    def _get_world_size(self):
+        """A helper function for getting the world size.
+
+        Read this from the environment variable JSM_NAMESPACE_SIZE
+        """
+        var = "JSM_NAMESPACE_SIZE"
+        world_size = os.environ.get(var)
+        if world_size is None:
+            raise ValueError(
+                "Cannot determine local rank -- expected in %s " "-- make sure you run your executable with jsrun" % var
+            )
+        return int(world_size)
+
+    def _get_node_rank(self):
+        """A helper function for getting the node rank."""
         hosts = self._read_hosts()
-        count = {}
+        count = dict()
         for host in hosts:
             if "batch" in host or "login" in host:
                 continue
@@ -125,38 +157,49 @@ def node_rank(self):
                 count[host] = len(count)
         return count[socket.gethostname()]
 
+    def __str__(self):
+        return self._rep
+
     @staticmethod
-    def _read_hosts():
-        hosts = os.environ.get("LSB_HOSTS")
-        if not hosts:
-            raise ValueError("Could not find hosts in environment variable LSB_HOSTS")
-        hosts = hosts.split()
-        if len(hosts) < 2:
-            raise ValueError(
-                'Cannot parse hosts from LSB_HOSTS environment variable. Expected format: "batch <rank_0_host> ..."'
-            )
-        return hosts
+    def detect():
+        """Detect if running in an LSF environment."""
+        env_vars = ["LSB_JOBID", "LSB_DJOB_RANKFILE", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE"]
+        flags = [v in os.environ for v in env_vars]
+        return any(flags)
+
+    def creates_processes_externally(self):
+        """LSF creates subprocesses -- i.e. PyTorch Lightning does not need to spawn them."""
+        return True
 
-    def _get_main_address(self) -> str:
-        hosts = self._read_hosts()
-        return hosts[1]
+    @property
+    def main_address(self):
+        """Master address is read from an OpenMPI host rank file in the environment variable *LSB_DJOB_RANKFILE*"""
+        return self._main_address
 
-    @staticmethod
-    def _get_main_port() -> int:
-        """A helper function for accessing the main port.
+    @property
+    def main_port(self):
+        """Master port is calculated from the LSF job ID."""
+        return self._main_port
 
-        Uses the LSF job ID so all ranks can compute the main port.
-        """
-        # check for user-specified main port
-        port = os.environ.get("MASTER_PORT")
-        if not port:
-            jobid = os.environ.get("LSB_JOBID")
-            if not jobid:
-                raise ValueError("Could not find job id in environment variable LSB_JOBID")
-            port = int(jobid)
-            # all ports should be in the 10k+ range
-            port = int(port) % 1000 + 10000
-            log.debug(f"calculated LSF main port: {port}")
-        else:
-            log.debug(f"using externally specified main port: {port}")
-        return int(port)
+    def world_size(self):
+        """World size is read from the environment variable JSM_NAMESPACE_SIZE."""
+        return self._world_size
+
+    def local_rank(self):
+        """World size is read from the environment variable JSM_NAMESPACE_LOCAL_RANK."""
+        return self._local_rank
+
+    def node_rank(self):
+        """Node rank is determined by the position of the current hostname in the OpenMPI host rank file stored in
+        LSB_DJOB_RANKFILE."""
+        return self._node_rank
+
+    def global_rank(self):
+        """World size is read from the environment variable JSM_NAMESPACE_RANK."""
+        return self._global_rank
+
+    def set_world_size(self, size: int) -> None:
+        log.debug("SLURMEnvironment.set_world_size was called, but setting " "world size is not allowed. Ignored.")
+
+    def set_global_rank(self, rank: int) -> None:
+        log.debug("SLURMEnvironment.set_global_rank was called, but setting " "global rank is not allowed. Ignored.")
diff --git a/tests/deprecated_api/test_remove_1-7.py b/tests/deprecated_api/test_remove_1-7.py
index 0065d5947dc26..099b8b278e377 100644
--- a/tests/deprecated_api/test_remove_1-7.py
+++ b/tests/deprecated_api/test_remove_1-7.py
@@ -509,8 +509,7 @@ def master_port(self):
         (TorchElasticEnvironment, "is_using_torchelastic"),
     ],
 )
-@mock.patch.dict(os.environ, {"LSB_HOSTS": "batch 10.10.10.0 10.10.10.1", "LSB_JOBID": "1234"})
-def test_v1_7_0_cluster_environment_detection(cls, method_name):
+def test_v1_7_0_cluster_environment_detection(cls, method_name, tmp_path):
     class MyClusterEnvironment(cls):
         @staticmethod
         def is_using_kubeflow():
@@ -524,7 +523,19 @@ def is_using_lsf():
         def is_using_torchelastic():
             pass
 
-    with pytest.deprecated_call(
-        match=f"MyClusterEnvironment.{method_name}` has been deprecated in v1.6 and will be removed in v1.7"
-    ):
-        MyClusterEnvironment()
+    hosts = "batch\n10.10.10.0\n10.10.10.1\n10.10.10.2\n10.10.10.3"
+    p = tmp_path / "lsb_djob_rankfile"
+    p.write_text(hosts)
+    environ = {
+        "LSB_DJOB_RANKFILE": str(p),
+        "LSB_JOBID": "1234",
+        "JSM_NAMESPACE_SIZE": "4",
+        "JSM_NAMESPACE_RANK": "3",
+        "JSM_NAMESPACE_LOCAL_RANK": "1",
+    }
+    with mock.patch.dict(os.environ, environ):
+        with mock.patch("socket.gethostname", return_value="10.10.10.2"):
+            with pytest.deprecated_call(
+                match=f"MyClusterEnvironment.{method_name}` has been deprecated in v1.6 and will be removed in v1.7"
+            ):
+                MyClusterEnvironment()
diff --git a/tests/plugins/environments/test_lsf_environment.py b/tests/plugins/environments/test_lsf_environment.py
index 57b3a16f1230a..ac9a8f6bbeffd 100644
--- a/tests/plugins/environments/test_lsf_environment.py
+++ b/tests/plugins/environments/test_lsf_environment.py
@@ -12,78 +12,106 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import os
-from unittest import mock
+from unittest.mock import patch
 
 import pytest
 
 from pytorch_lightning.plugins.environments import LSFEnvironment
 
 
-@mock.patch.dict(os.environ, {"LSB_HOSTS": "batch 10.10.10.0 10.10.10.1", "LSB_JOBID": "1234"})
-def test_missing_lsb_hosts():
-    """Test an error when the lsb hosts list cannot be found."""
-    del os.environ["LSB_HOSTS"]
-    with pytest.raises(ValueError, match="Could not find hosts in environment variable LSB_HOSTS"):
-        LSFEnvironment()
+def _make_rankfile(tmp_path):
+    hosts = "batch\n10.10.10.0\n10.10.10.1\n10.10.10.2\n10.10.10.3"
+    p = tmp_path / "lsb_djob_rankfile"
+    p.write_text(hosts)
+    return str(p)
 
 
-@mock.patch.dict(os.environ, {"LSB_HOSTS": "batch 10.10.10.0 10.10.10.1", "LSB_JOBID": "1234"})
-def test_missing_lsb_job_id():
+def test_missing_lsb_djob_rankfile():
+    """Test an error when the LSB_DJOB_RANKFILE cannot be found."""
+    with patch.dict(os.environ, {"LSB_DJOB_RANKFILE": "", "LSB_JOBID": "1234"}):
+        del os.environ["LSB_DJOB_RANKFILE"]
+        with pytest.raises(ValueError, match="Could not find environment variable LSB_DJOB_RANKFILE"):
+            LSFEnvironment()
+
+
+def test_empty_lsb_djob_rankfile():
+    """Test an error when the LSB_DJOB_RANKFILE is not populated."""
+    with patch.dict(os.environ, {"LSB_DJOB_RANKFILE": "", "LSB_JOBID": "1234"}):
+        with pytest.raises(ValueError, match="Environment variable LSB_DJOB_RANKFILE is empty"):
+            LSFEnvironment()
+
+
+def test_missing_lsb_job_id(tmp_path):
     """Test an error when the job id cannot be found."""
-    del os.environ["LSB_JOBID"]
-    with pytest.raises(ValueError, match="Could not find job id in environment variable LSB_JOBID"):
-        LSFEnvironment()
+    with patch.dict(os.environ, {"LSB_DJOB_RANKFILE": _make_rankfile(tmp_path), "LSB_JOBID": "1234"}):
+        del os.environ["LSB_JOBID"]
+        with pytest.raises(ValueError, match="Could not find job id -- expected in environment variable LSB_JOBID"):
+            LSFEnvironment()
 
 
-@mock.patch.dict(os.environ, {"MASTER_PORT": "4321", "LSB_JOBID": "1234", "LSB_HOSTS": "batch 10.10.10.0 10.10.10.1"})
-def test_manual_main_port_and_address():
+def test_manual_main_port_and_address(tmp_path):
     """Test a user can set the port manually through the MASTER_PORT env variable."""
-    env = LSFEnvironment()
-    assert env.main_port == 4321
+    environ = {
+        "LSB_DJOB_RANKFILE": _make_rankfile(tmp_path),
+        "LSB_JOBID": "1234",
+        "JSM_NAMESPACE_SIZE": "4",
+        "JSM_NAMESPACE_RANK": "3",
+        "JSM_NAMESPACE_LOCAL_RANK": "1",
+    }
+    with patch.dict(os.environ, environ):
+        with patch("socket.gethostname", return_value="10.10.10.2"):
+            env = LSFEnvironment()
+            assert env.main_port == 10234
 
 
-@mock.patch.dict(
-    os.environ,
-    {
-        "LSB_HOSTS": "batch 10.10.10.0 10.10.10.1 10.10.10.2 10.10.10.3",
+def test_attributes_from_environment_variables(tmp_path):
+    """Test that the LSF environment takes the attributes from the environment variables."""
+    environ = {
+        "LSB_DJOB_RANKFILE": _make_rankfile(tmp_path),
         "LSB_JOBID": "1234",
         "JSM_NAMESPACE_SIZE": "4",
         "JSM_NAMESPACE_RANK": "3",
         "JSM_NAMESPACE_LOCAL_RANK": "1",
-    },
-)
-def test_attributes_from_environment_variables():
-    """Test that the LSF environment takes the attributes from the environment variables."""
-    env = LSFEnvironment()
-    assert env.creates_processes_externally
-    assert env.main_address == "10.10.10.0"
-    assert env.main_port == 10234
-    assert env.world_size() == 4
-    assert env.global_rank() == 3
-    assert env.local_rank() == 1
-    env.set_global_rank(100)
-    assert env.global_rank() == 3
-    env.set_world_size(100)
-    assert env.world_size() == 4
-    assert LSFEnvironment.detect()
-
-
-@mock.patch("socket.gethostname", return_value="host2")
-@mock.patch.dict(os.environ, {"LSB_HOSTS": "batch host0 host1 host2 host3", "LSB_JOBID": "1234"})
-def test_node_rank(_):
-    env = LSFEnvironment()
-    assert env.node_rank() == 2
+    }
+    with patch.dict(os.environ, environ):
+        with patch("socket.gethostname", return_value="10.10.10.2"):
+            env = LSFEnvironment()
+            assert env.creates_processes_externally
+            assert env.main_address == "10.10.10.0"
+            assert env.main_port == 10234
+            assert env.world_size() == 4
+            assert env.global_rank() == 3
+            assert env.local_rank() == 1
+            env.set_global_rank(100)
+            assert env.global_rank() == 3
+            env.set_world_size(100)
+            assert env.world_size() == 4
+            assert LSFEnvironment.detect()
+
+
+def test_node_rank(tmp_path):
+    environ = {
+        "LSB_DJOB_RANKFILE": _make_rankfile(tmp_path),
+        "LSB_JOBID": "1234",
+        "JSM_NAMESPACE_SIZE": "4",
+        "JSM_NAMESPACE_RANK": "3",
+        "JSM_NAMESPACE_LOCAL_RANK": "1",
+    }
+    with patch.dict(os.environ, environ):
+        with patch("socket.gethostname", return_value="10.10.10.2"):
+            env = LSFEnvironment()
+            assert env.node_rank() == 2
 
 
 def test_detect():
     """Test the detection of a LSF environment configuration."""
-    with mock.patch.dict(os.environ, {}):
+    with patch.dict(os.environ, {}):
         assert not LSFEnvironment.detect()
 
-    with mock.patch.dict(
+    with patch.dict(
         os.environ,
         {
-            "LSB_HOSTS": "",
+            "LSB_DJOB_RANKFILE": "",
             "LSB_JOBID": "",
             "JSM_NAMESPACE_SIZE": "",
             "JSM_NAMESPACE_LOCAL_RANK": "",


Code review comments:
diff_hunk: @@ -51,112 +54,152 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
+        os.environ["MASTER_ADDR"] = str(self._main_address)
+        log.debug(f"MASTER_ADDR: {os.environ['MASTER_ADDR']}")
+        os.environ["MASTER_PORT"] = str(self._main_port)
+        log.debug(f"MASTER_PORT: {os.environ['MASTER_PORT']}")
+
+        tmp = ("main_address", "main_port", "world_size", "local_rank", "node_rank", "global_rank")
path: pytorch_lightning/plugins/environments/lsf_environment.py
text: @user1:
I would prefer to have this directly within __str__

diff_hunk: @@ -51,112 +54,152 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
+        os.environ["MASTER_ADDR"] = str(self._main_address)
+        log.debug(f"MASTER_ADDR: {os.environ['MASTER_ADDR']}")
+        os.environ["MASTER_PORT"] = str(self._main_port)
+        log.debug(f"MASTER_PORT: {os.environ['MASTER_PORT']}")
+
+        tmp = ("main_address", "main_port", "world_size", "local_rank", "node_rank", "global_rank")
+        self._rep = ",".join("{}={}".format(s, getattr(self, "_" + s)) for s in tmp)
+
+    def _read_hosts(self):
+        var = "LSB_DJOB_RANKFILE"
+        try:
+            rankfile = os.environ[var]
+        except KeyError:
+            raise ValueError("Could not find environment variable LSB_DJOB_RANKFILE")
+        if not rankfile:
+            raise ValueError("Environment variable LSB_DJOB_RANKFILE is empty")
+        with open(rankfile) as f:
path: pytorch_lightning/plugins/environments/lsf_environment.py
text: @user1:
Silly question, but should we use fs = getfilesystem(rankfile), fs.open to be consistent with the rest of the codebase ?

@author:
I'm not familiar with `getfilesystem`. Can you elaborate?

@user1:
This might be an overkill, but here is how it works.


```py
from pytorch_lightning.utilities.cloud_io import get_filesystem


fs = get_filesystem(rankfile)
with fs.open(rankfile) as f:
     ...

diff_hunk: @@ -51,112 +54,152 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
+        os.environ["MASTER_ADDR"] = str(self._main_address)
+        log.debug(f"MASTER_ADDR: {os.environ['MASTER_ADDR']}")
+        os.environ["MASTER_PORT"] = str(self._main_port)
+        log.debug(f"MASTER_PORT: {os.environ['MASTER_PORT']}")
+
+        tmp = ("main_address", "main_port", "world_size", "local_rank", "node_rank", "global_rank")
+        self._rep = ",".join("{}={}".format(s, getattr(self, "_" + s)) for s in tmp)
+
+    def _read_hosts(self):
+        var = "LSB_DJOB_RANKFILE"
+        try:
+            rankfile = os.environ[var]
+        except KeyError:
+            raise ValueError("Could not find environment variable LSB_DJOB_RANKFILE")
+        if not rankfile:
+            raise ValueError("Environment variable LSB_DJOB_RANKFILE is empty")
+        with open(rankfile) as f:
+            ret = [line.strip() for line in f]
+        return ret
+
+    def _get_main_address(self):
+        """A helper for getting the master address."""
+        hosts = self._read_hosts()
+        return hosts[1]
 
-    @staticmethod
-    def detect() -> bool:
-        """Returns ``True`` if the current process was launched using the jsrun command."""
-        required_env_vars = {"LSB_JOBID", "LSB_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE"}
-        return required_env_vars.issubset(os.environ.keys())
+    def _get_main_port(self):
+        """A helper for getting the master port.
 
-    def world_size(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_SIZE`."""
-        var = "JSM_NAMESPACE_SIZE"
-        world_size = os.environ.get(var)
-        if world_size is None:
-            raise ValueError(
-                f"Cannot determine world size from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
-            )
-        return int(world_size)
+        Use the LSF job ID so all ranks can compute the master port
+        """
+        # check for user-specified master port
+        port = os.environ.get("MASTER_PORT")
+        if not port:
+            var = "LSB_JOBID"
+            jobid = os.environ.get(var)
path: pytorch_lightning/plugins/environments/lsf_environment.py
text: @user1:
```suggestion
            jobid = os.environ.get(var, None)
```

diff_hunk: @@ -51,112 +54,152 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
+        os.environ["MASTER_ADDR"] = str(self._main_address)
+        log.debug(f"MASTER_ADDR: {os.environ['MASTER_ADDR']}")
+        os.environ["MASTER_PORT"] = str(self._main_port)
+        log.debug(f"MASTER_PORT: {os.environ['MASTER_PORT']}")
+
+        tmp = ("main_address", "main_port", "world_size", "local_rank", "node_rank", "global_rank")
+        self._rep = ",".join("{}={}".format(s, getattr(self, "_" + s)) for s in tmp)
+
+    def _read_hosts(self):
+        var = "LSB_DJOB_RANKFILE"
+        try:
+            rankfile = os.environ[var]
+        except KeyError:
+            raise ValueError("Could not find environment variable LSB_DJOB_RANKFILE")
+        if not rankfile:
+            raise ValueError("Environment variable LSB_DJOB_RANKFILE is empty")
+        with open(rankfile) as f:
+            ret = [line.strip() for line in f]
+        return ret
+
+    def _get_main_address(self):
+        """A helper for getting the master address."""
+        hosts = self._read_hosts()
+        return hosts[1]
 
-    @staticmethod
-    def detect() -> bool:
-        """Returns ``True`` if the current process was launched using the jsrun command."""
-        required_env_vars = {"LSB_JOBID", "LSB_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE"}
-        return required_env_vars.issubset(os.environ.keys())
+    def _get_main_port(self):
+        """A helper for getting the master port.
 
-    def world_size(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_SIZE`."""
-        var = "JSM_NAMESPACE_SIZE"
-        world_size = os.environ.get(var)
-        if world_size is None:
-            raise ValueError(
-                f"Cannot determine world size from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
-            )
-        return int(world_size)
+        Use the LSF job ID so all ranks can compute the master port
+        """
+        # check for user-specified master port
+        port = os.environ.get("MASTER_PORT")
+        if not port:
+            var = "LSB_JOBID"
+            jobid = os.environ.get(var)
+            if not jobid:
+                raise ValueError("Could not find job id -- expected in environment variable %s" % var)
+            else:
+                port = int(jobid)
+                # all ports should be in the 10k+ range
+                port = int(port) % 1000 + 10000
+            log.debug("calculated master port")
+        else:
+            log.debug("using externally specified master port")
+        return port
 
-    def set_world_size(self, size: int) -> None:
-        log.debug("LSFEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.")
+    def _get_global_rank(self):
+        """A helper function for getting the global rank.
 
-    def global_rank(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_RANK"
         global_rank = os.environ.get(var)
         if global_rank is None:
             raise ValueError(
-                f"Cannot determine global rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine global rank -- expected in %s "
+                "-- make sure you run your executable with jsrun" % var
             )
         return int(global_rank)
 
-    def set_global_rank(self, rank: int) -> None:
-        log.debug("LSFEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.")
+    def _get_local_rank(self):
+        """A helper function for getting the local rank.
 
-    def local_rank(self):
-        """The local rank is read from the environment variable `JSM_NAMESPACE_LOCAL_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_LOCAL_RANK"
         local_rank = os.environ.get(var)
         if local_rank is None:
             raise ValueError(
-                f"Cannot determine local rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine local rank -- expected in %s " "-- make sure you run your executable with jsrun" % var
             )
         return int(local_rank)
 
-    def node_rank(self):
-        """The node rank is determined by the position of the current hostname in the list of hosts stored in the
-        environment variable `LSB_HOSTS`."""
+    def _get_world_size(self):
+        """A helper function for getting the world size.
+
+        Read this from the environment variable JSM_NAMESPACE_SIZE
+        """
+        var = "JSM_NAMESPACE_SIZE"
+        world_size = os.environ.get(var)
+        if world_size is None:
+            raise ValueError(
+                "Cannot determine local rank -- expected in %s " "-- make sure you run your executable with jsrun" % var
path: pytorch_lightning/plugins/environments/lsf_environment.py
text: @user1:
Let's use f"{var}" to be consistent with the rest of the codebase.

@author:
```suggestion
                f'Cannot determine local rank -- expected in {var} -- make sure you run your executable with jsrun'
```

@user1:
Great, you can commit them directly and resolve the comment :)

diff_hunk: @@ -51,112 +54,152 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
+        os.environ["MASTER_ADDR"] = str(self._main_address)
+        log.debug(f"MASTER_ADDR: {os.environ['MASTER_ADDR']}")
+        os.environ["MASTER_PORT"] = str(self._main_port)
+        log.debug(f"MASTER_PORT: {os.environ['MASTER_PORT']}")
+
+        tmp = ("main_address", "main_port", "world_size", "local_rank", "node_rank", "global_rank")
+        self._rep = ",".join("{}={}".format(s, getattr(self, "_" + s)) for s in tmp)
+
+    def _read_hosts(self):
+        var = "LSB_DJOB_RANKFILE"
+        try:
+            rankfile = os.environ[var]
+        except KeyError:
+            raise ValueError("Could not find environment variable LSB_DJOB_RANKFILE")
+        if not rankfile:
+            raise ValueError("Environment variable LSB_DJOB_RANKFILE is empty")
+        with open(rankfile) as f:
+            ret = [line.strip() for line in f]
+        return ret
+
+    def _get_main_address(self):
+        """A helper for getting the master address."""
+        hosts = self._read_hosts()
+        return hosts[1]
path: pytorch_lightning/plugins/environments/lsf_environment.py
text: @user1:
Should we validate that `batch` is hosts[0] ?

@author:
No. In fact I am going to remove this check. I am not certain that `batch` will always be used to launch nodes.

diff_hunk: @@ -51,112 +54,152 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
+        os.environ["MASTER_ADDR"] = str(self._main_address)
+        log.debug(f"MASTER_ADDR: {os.environ['MASTER_ADDR']}")
+        os.environ["MASTER_PORT"] = str(self._main_port)
+        log.debug(f"MASTER_PORT: {os.environ['MASTER_PORT']}")
+
+        tmp = ("main_address", "main_port", "world_size", "local_rank", "node_rank", "global_rank")
+        self._rep = ",".join("{}={}".format(s, getattr(self, "_" + s)) for s in tmp)
+
+    def _read_hosts(self):
+        var = "LSB_DJOB_RANKFILE"
+        try:
+            rankfile = os.environ[var]
+        except KeyError:
+            raise ValueError("Could not find environment variable LSB_DJOB_RANKFILE")
+        if not rankfile:
+            raise ValueError("Environment variable LSB_DJOB_RANKFILE is empty")
+        with open(rankfile) as f:
+            ret = [line.strip() for line in f]
+        return ret
+
+    def _get_main_address(self):
+        """A helper for getting the master address."""
+        hosts = self._read_hosts()
+        return hosts[1]
 
-    @staticmethod
-    def detect() -> bool:
-        """Returns ``True`` if the current process was launched using the jsrun command."""
-        required_env_vars = {"LSB_JOBID", "LSB_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE"}
-        return required_env_vars.issubset(os.environ.keys())
+    def _get_main_port(self):
+        """A helper for getting the master port.
 
-    def world_size(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_SIZE`."""
-        var = "JSM_NAMESPACE_SIZE"
-        world_size = os.environ.get(var)
-        if world_size is None:
-            raise ValueError(
-                f"Cannot determine world size from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
-            )
-        return int(world_size)
+        Use the LSF job ID so all ranks can compute the master port
+        """
+        # check for user-specified master port
+        port = os.environ.get("MASTER_PORT")
+        if not port:
+            var = "LSB_JOBID"
+            jobid = os.environ.get(var)
+            if not jobid:
+                raise ValueError("Could not find job id -- expected in environment variable %s" % var)
+            else:
+                port = int(jobid)
+                # all ports should be in the 10k+ range
+                port = int(port) % 1000 + 10000
+            log.debug("calculated master port")
+        else:
+            log.debug("using externally specified master port")
+        return port
 
-    def set_world_size(self, size: int) -> None:
-        log.debug("LSFEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.")
+    def _get_global_rank(self):
+        """A helper function for getting the global rank.
 
-    def global_rank(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_RANK"
         global_rank = os.environ.get(var)
         if global_rank is None:
             raise ValueError(
-                f"Cannot determine global rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine global rank -- expected in %s "
+                "-- make sure you run your executable with jsrun" % var
             )
         return int(global_rank)
 
-    def set_global_rank(self, rank: int) -> None:
-        log.debug("LSFEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.")
+    def _get_local_rank(self):
+        """A helper function for getting the local rank.
 
-    def local_rank(self):
-        """The local rank is read from the environment variable `JSM_NAMESPACE_LOCAL_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_LOCAL_RANK"
         local_rank = os.environ.get(var)
         if local_rank is None:
             raise ValueError(
-                f"Cannot determine local rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine local rank -- expected in %s " "-- make sure you run your executable with jsrun" % var
             )
         return int(local_rank)
 
-    def node_rank(self):
-        """The node rank is determined by the position of the current hostname in the list of hosts stored in the
-        environment variable `LSB_HOSTS`."""
+    def _get_world_size(self):
+        """A helper function for getting the world size.
+
+        Read this from the environment variable JSM_NAMESPACE_SIZE
+        """
+        var = "JSM_NAMESPACE_SIZE"
+        world_size = os.environ.get(var)
path: pytorch_lightning/plugins/environments/lsf_environment.py
text: @user1:
```suggestion
        world_size = os.environ.get(var, None)
```

diff_hunk: @@ -51,112 +54,152 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
+        os.environ["MASTER_ADDR"] = str(self._main_address)
+        log.debug(f"MASTER_ADDR: {os.environ['MASTER_ADDR']}")
+        os.environ["MASTER_PORT"] = str(self._main_port)
+        log.debug(f"MASTER_PORT: {os.environ['MASTER_PORT']}")
+
+        tmp = ("main_address", "main_port", "world_size", "local_rank", "node_rank", "global_rank")
+        self._rep = ",".join("{}={}".format(s, getattr(self, "_" + s)) for s in tmp)
+
+    def _read_hosts(self):
+        var = "LSB_DJOB_RANKFILE"
+        try:
+            rankfile = os.environ[var]
+        except KeyError:
+            raise ValueError("Could not find environment variable LSB_DJOB_RANKFILE")
+        if not rankfile:
+            raise ValueError("Environment variable LSB_DJOB_RANKFILE is empty")
+        with open(rankfile) as f:
+            ret = [line.strip() for line in f]
+        return ret
+
+    def _get_main_address(self):
+        """A helper for getting the master address."""
+        hosts = self._read_hosts()
+        return hosts[1]
 
-    @staticmethod
-    def detect() -> bool:
-        """Returns ``True`` if the current process was launched using the jsrun command."""
-        required_env_vars = {"LSB_JOBID", "LSB_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE"}
-        return required_env_vars.issubset(os.environ.keys())
+    def _get_main_port(self):
+        """A helper for getting the master port.
 
-    def world_size(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_SIZE`."""
-        var = "JSM_NAMESPACE_SIZE"
-        world_size = os.environ.get(var)
-        if world_size is None:
-            raise ValueError(
-                f"Cannot determine world size from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
-            )
-        return int(world_size)
+        Use the LSF job ID so all ranks can compute the master port
+        """
+        # check for user-specified master port
+        port = os.environ.get("MASTER_PORT")
+        if not port:
+            var = "LSB_JOBID"
+            jobid = os.environ.get(var)
+            if not jobid:
+                raise ValueError("Could not find job id -- expected in environment variable %s" % var)
+            else:
+                port = int(jobid)
+                # all ports should be in the 10k+ range
+                port = int(port) % 1000 + 10000
+            log.debug("calculated master port")
+        else:
+            log.debug("using externally specified master port")
+        return port
 
-    def set_world_size(self, size: int) -> None:
-        log.debug("LSFEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.")
+    def _get_global_rank(self):
+        """A helper function for getting the global rank.
 
-    def global_rank(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_RANK"
         global_rank = os.environ.get(var)
         if global_rank is None:
             raise ValueError(
-                f"Cannot determine global rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine global rank -- expected in %s "
+                "-- make sure you run your executable with jsrun" % var
             )
         return int(global_rank)
 
-    def set_global_rank(self, rank: int) -> None:
-        log.debug("LSFEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.")
+    def _get_local_rank(self):
+        """A helper function for getting the local rank.
 
-    def local_rank(self):
-        """The local rank is read from the environment variable `JSM_NAMESPACE_LOCAL_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_LOCAL_RANK"
         local_rank = os.environ.get(var)
         if local_rank is None:
             raise ValueError(
-                f"Cannot determine local rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine local rank -- expected in %s " "-- make sure you run your executable with jsrun" % var
             )
         return int(local_rank)
 
-    def node_rank(self):
-        """The node rank is determined by the position of the current hostname in the list of hosts stored in the
-        environment variable `LSB_HOSTS`."""
+    def _get_world_size(self):
+        """A helper function for getting the world size.
+
+        Read this from the environment variable JSM_NAMESPACE_SIZE
+        """
+        var = "JSM_NAMESPACE_SIZE"
+        world_size = os.environ.get(var)
+        if world_size is None:
+            raise ValueError(
+                "Cannot determine local rank -- expected in %s " "-- make sure you run your executable with jsrun" % var
+            )
+        return int(world_size)
+
+    def _get_node_rank(self):
+        """A helper function for getting the node rank."""
path: pytorch_lightning/plugins/environments/lsf_environment.py
text: @author:
```suggestion
        """A helper function for getting the node rank.
        
        Node rank is determined by the position of the current node in the hosts
        used in the job. This is calculated by reading all hosts from LSB_DJOB_RANKFILE
        and finding this nodes hostname in the list.
        """
```

@user1:
Great !

diff_hunk: @@ -51,112 +54,152 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
path: pytorch_lightning/plugins/environments/lsf_environment.py
text: @user1:
Let's move this within a private method.

@author:
Is this what you wanted moved to a private method?

@user1:
Yes, I think thought we could move the master_address / master port to assignment to a method to clean a bit the code.

@user1:
Neat !

diff_hunk: @@ -51,112 +54,152 @@ def __init__(self):
             )
         self._main_address = self._get_main_address()
         self._main_port = self._get_main_port()
-        log.debug(f"MASTER_ADDR: {self._main_address}")
-        log.debug(f"MASTER_PORT: {self._main_port}")
-
-    @property
-    def creates_processes_externally(self) -> bool:
-        return True
-
-    @property
-    def main_address(self) -> str:
-        """The main address is read from a list of hosts contained in the environment variable `LSB_HOSTS`."""
-        return self._main_address
-
-    @property
-    def main_port(self) -> int:
-        """The main port gets calculated from the LSF job ID."""
-        return self._main_port
+        self._local_rank = self._get_local_rank()
+        self._global_rank = self._get_global_rank()
+        self._world_size = self._get_world_size()
+        self._node_rank = self._get_node_rank()
+
+        # set environment variables needed for initializing torch distributed process group
+        os.environ["MASTER_ADDR"] = str(self._main_address)
+        log.debug(f"MASTER_ADDR: {os.environ['MASTER_ADDR']}")
+        os.environ["MASTER_PORT"] = str(self._main_port)
+        log.debug(f"MASTER_PORT: {os.environ['MASTER_PORT']}")
+
+        tmp = ("main_address", "main_port", "world_size", "local_rank", "node_rank", "global_rank")
+        self._rep = ",".join("{}={}".format(s, getattr(self, "_" + s)) for s in tmp)
+
+    def _read_hosts(self):
+        var = "LSB_DJOB_RANKFILE"
+        try:
+            rankfile = os.environ[var]
+        except KeyError:
+            raise ValueError("Could not find environment variable LSB_DJOB_RANKFILE")
+        if not rankfile:
+            raise ValueError("Environment variable LSB_DJOB_RANKFILE is empty")
+        with open(rankfile) as f:
+            ret = [line.strip() for line in f]
+        return ret
+
+    def _get_main_address(self):
+        """A helper for getting the master address."""
+        hosts = self._read_hosts()
+        return hosts[1]
 
-    @staticmethod
-    def detect() -> bool:
-        """Returns ``True`` if the current process was launched using the jsrun command."""
-        required_env_vars = {"LSB_JOBID", "LSB_HOSTS", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE"}
-        return required_env_vars.issubset(os.environ.keys())
+    def _get_main_port(self):
+        """A helper for getting the master port.
 
-    def world_size(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_SIZE`."""
-        var = "JSM_NAMESPACE_SIZE"
-        world_size = os.environ.get(var)
-        if world_size is None:
-            raise ValueError(
-                f"Cannot determine world size from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
-            )
-        return int(world_size)
+        Use the LSF job ID so all ranks can compute the master port
+        """
+        # check for user-specified master port
+        port = os.environ.get("MASTER_PORT")
+        if not port:
+            var = "LSB_JOBID"
+            jobid = os.environ.get(var)
+            if not jobid:
+                raise ValueError("Could not find job id -- expected in environment variable %s" % var)
+            else:
+                port = int(jobid)
+                # all ports should be in the 10k+ range
+                port = int(port) % 1000 + 10000
+            log.debug("calculated master port")
+        else:
+            log.debug("using externally specified master port")
+        return port
 
-    def set_world_size(self, size: int) -> None:
-        log.debug("LSFEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.")
+    def _get_global_rank(self):
+        """A helper function for getting the global rank.
 
-    def global_rank(self):
-        """The world size is read from the environment variable `JSM_NAMESPACE_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_RANK"
         global_rank = os.environ.get(var)
         if global_rank is None:
             raise ValueError(
-                f"Cannot determine global rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine global rank -- expected in %s "
+                "-- make sure you run your executable with jsrun" % var
             )
         return int(global_rank)
 
-    def set_global_rank(self, rank: int) -> None:
-        log.debug("LSFEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.")
+    def _get_local_rank(self):
+        """A helper function for getting the local rank.
 
-    def local_rank(self):
-        """The local rank is read from the environment variable `JSM_NAMESPACE_LOCAL_RANK`."""
+        Read this from the environment variable JSM_NAMESPACE_LOCAL_RANK
+        """
         var = "JSM_NAMESPACE_LOCAL_RANK"
         local_rank = os.environ.get(var)
         if local_rank is None:
             raise ValueError(
-                f"Cannot determine local rank from environment variable {var}."
-                " Make sure you run your executable with `jsrun`"
+                "Cannot determine local rank -- expected in %s " "-- make sure you run your executable with jsrun" % var
             )
         return int(local_rank)
 
-    def node_rank(self):
-        """The node rank is determined by the position of the current hostname in the list of hosts stored in the
-        environment variable `LSB_HOSTS`."""
+    def _get_world_size(self):
+        """A helper function for getting the world size.
+
+        Read this from the environment variable JSM_NAMESPACE_SIZE
+        """
+        var = "JSM_NAMESPACE_SIZE"
+        world_size = os.environ.get(var)
+        if world_size is None:
+            raise ValueError(
+                "Cannot determine local rank -- expected in %s " "-- make sure you run your executable with jsrun" % var
+            )
+        return int(world_size)
+
+    def _get_node_rank(self):
+        """A helper function for getting the node rank."""
         hosts = self._read_hosts()
-        count = {}
+        count = dict()
         for host in hosts:
             if "batch" in host or "login" in host:
                 continue
             if host not in count:
                 count[host] = len(count)
         return count[socket.gethostname()]
 
+    def __str__(self):
+        return self._rep
+
     @staticmethod
-    def _read_hosts():
-        hosts = os.environ.get("LSB_HOSTS")
-        if not hosts:
-            raise ValueError("Could not find hosts in environment variable LSB_HOSTS")
-        hosts = hosts.split()
-        if len(hosts) < 2:
-            raise ValueError(
-                'Cannot parse hosts from LSB_HOSTS environment variable. Expected format: "batch <rank_0_host> ..."'
-            )
-        return hosts
+    def detect():
+        """Detect if running in an LSF environment."""
+        env_vars = ["LSB_JOBID", "LSB_DJOB_RANKFILE", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE"]
+        flags = [v in os.environ for v in env_vars]
+        return any(flags)
path: pytorch_lightning/plugins/environments/lsf_environment.py
text: @user1:
```suggestion
        env_vars = ("LSB_JOBID", "LSB_DJOB_RANKFILE", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE")
        return any(f in os.environ for f in env_vars)
```