ProcessExitedException: process 0 terminated with exit code 1

username: david_patsch
cooked: <p>The basic example i am trying to run:</p>
<blockquote>
<p>“”"<br>
Based on: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
<p>Correctness of code: <a href="https://stackoverflow.com/questions/66226135/how-to-parallelize-a-training-loop-ever-samples-of-a-batch-when-cpu-is-only-avai" class="inline-onebox" rel="noopener nofollow ugc">machine learning - How to parallelize a training loop ever samples of a batch when CPU is only available in pytorch? - Stack Overflow</a></p>
<p>Note: as opposed to the multiprocessing (torch.multiprocessing) package, processes can use<br>
different communication backends and are not restricted to being executed on the same machine.<br>
“”"<br>
import time</p>
<p>from typing import Tuple</p>
<p>import torch<br>
from torch import nn, optim<br>
import torch.distributed as dist<br>
import torch.multiprocessing as mp<br>
from torch.nn.parallel import DistributedDataParallel as DDP</p>
<p>import os</p>
<p>num_epochs = 5<br>
batch_size = 8<br>
Din, Dout = 10, 5<br>
data_x = torch.randn(batch_size, Din)<br>
data_y = torch.randn(batch_size, Dout)<br>
data = [(i<em>data_x, i</em>data_y) for i in range(num_epochs)]</p>
<p>class PerDeviceModel(nn.Module):<br>
“”"<br>
Toy example for a model ran in parallel but not distributed accross gpus<br>
(only processes with their own gpu or hardware)<br>
“”"<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.net1 = nn.Linear(Din, Din)<br>
self.relu = nn.ReLU()<br>
self.net2 = nn.Linear(Din, Dout)</p>
<pre><code>def forward(self, x):
    return self.net2(self.relu(self.net1(x)))
</code></pre>
<p>def setup_process(rank, world_size, backend=‘nccl’):<br>
“”"<br>
Initialize the distributed environment (for each process).</p>
<pre><code>gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.
"""
# set up the master's ip address so this child process can coordinate
# os.environ['MASTER_ADDR'] = '127.0.0.1'
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'

# - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
# if torch.cuda.is_available():
#     backend = 'nccl'
# Initializes the default distributed process group, and this will also initialize the distributed package.
dist.init_process_group(backend, rank=rank, world_size=world_size)
</code></pre>
<p>def cleanup():<br>
“”" Destroy a given process group, and deinitialize the distributed package “”"<br>
dist.destroy_process_group()</p>
<p>def get_batch(batch: Tuple[torch.Tensor, torch.Tensor], rank):<br>
x, y = batch<br>
if torch.cuda.is_available():<br>
x, y = x.to(rank), y.to(rank)<br>
else:<br>
x, y = x.share_memory_(), y.share_memory_()<br>
return x, y</p>
<p>def get_ddp_model(model: nn.Module, rank):<br>
“”"<br>
Moves the underlying storage to shared memory.</p>
<pre><code>    This is a no-op if the underlying storage is already in shared memory
    and for CUDA tensors. Tensors in shared memory cannot be resized.

:return:

TODO: does this have to be done outside or inside the process? my guess is that it doesn't matter because
1) if its on gpu once it's on the right proc it moves it to cpu with id rank via mdl.to(rank)
2) if it's on cpu then mdl.share_memory() or data.share_memory() is a no op if it's already in shared memory o.w.
"""
# if gpu avail do the standard of creating a model and moving the model to the GPU with id rank
if torch.cuda.is_available():
# create model and move it to GPU with id rank
    model = model.to(rank)
    ddp_model = DDP(model, device_ids=[rank])
else:
# if we want multiple cpu just make sure the model is shared properly accross the cpus with shared_memory()
# note that op is a no op if it's already in shared_memory
    model = model.share_memory()
    ddp_model = DDP(model)  # I think removing the devices ids should be fine...?
return ddp_model
# return OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()
</code></pre>
<p>def run_parallel_training_loop(rank, world_size):<br>
“”"<br>
Distributed function to be implemented later.</p>
<pre><code>This is the function that is actually ran in each distributed process.

Note: as DDP broadcasts model states from rank 0 process to all other processes in the DDP constructor,
you don’t need to worry about different DDP processes start from different model parameter initial values.
"""
setup_process(rank, world_size)
print()
print(f"Start running DDP with model parallel example on rank: {rank}.")
print(f'current process: {mp.current_process()}')
print(f'pid: {os.getpid()}')

# get ddp model
model = PerDeviceModel()
ddp_model = get_ddp_model(model, rank)

# do training
for batch_idx, batch in enumerate(data):
    x, y = get_batch(batch, rank)
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    outputs = ddp_model(x)
    # Gradient synchronization communications take place during the backward pass and overlap with the backward computation.
    loss_fn(outputs, y).backward()  # When the backward() returns, param.grad already contains the synchronized gradient tensor.
    optimizer.step()  # TODO how does the optimizer know to do the gradient step only once?

print()
print(f"End running DDP with model parallel example on rank: {rank}.")
print(f'End current process: {mp.current_process()}')
print(f'End pid: {os.getpid()}')
# Destroy a given process group, and deinitialize the distributed package
cleanup()
</code></pre>
<p>def main():<br>
print()<br>
print(‘running main()’)<br>
print(f’current process: {mp.current_process()}‘)<br>
print(f’pid: {os.getpid()}’)<br>
# args<br>
if torch.cuda.is_available():<br>
world_size = torch.cuda.device_count()<br>
else:<br>
world_size = mp.cpu_count()</p>
<pre><code>print(f'world_size={world_size}')
mp.spawn(run_parallel_training_loop, args=(world_size,), nprocs=world_size)
</code></pre>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
print(‘starting <strong>main</strong>’)<br>
start = time.time()<br>
main()<br>
print(f’execution length = {time.time() - start}')<br>
print(‘Done!\a\n’)</p>
</blockquote>
<p>The error i get:</p>
<p>I tried reinstalling all nvidia-drivers, but the issue persists</p>
<blockquote>
<hr>
<p>ProcessExitedException                    Traceback (most recent call last)<br>
 in <br>
151     print(‘starting <strong>main</strong>’)<br>
152     start = time.time()<br>
 → 153     main()<br>
154     print(f’execution length = {time.time() - start}')<br>
155     print(‘Done!\a\n’)</p>
<p> in main()<br>
146<br>
147     print(f’world_size={world_size}')<br>
 → 148     mp.spawn(run_parallel_training_loop, args=(world_size,), nprocs=world_size)<br>
149<br>
150 if <strong>name</strong> == “<strong>main</strong>”:</p>
<p>~/anaconda3/envs/leaf/lib/python3.8/site-packages/torch/multiprocessing/spawn.py in spawn(fn, args, nprocs, join, daemon, start_method)<br>
228                ’ torch.multiprocessing.start_process(…)’ % start_method)<br>
229         warnings.warn(msg)<br>
 → 230     return start_processes(fn, args, nprocs, join, daemon, start_method=‘spawn’)</p>
<p>~/anaconda3/envs/leaf/lib/python3.8/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)<br>
186<br>
187     # Loop on join until it returns True or raises an exception.<br>
 → 188     while not context.join():<br>
189         pass<br>
190</p>
<p>~/anaconda3/envs/leaf/lib/python3.8/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)<br>
137                 )<br>
138             else:<br>
 → 139                 raise ProcessExitedException(<br>
140                     “process %d terminated with exit code %d” %<br>
141                     (error_index, exitcode),</p>
<p>ProcessExitedException: process 0 terminated with exit code 1</p>
</blockquote>

username: Anti_Lippasaar
cooked: <p>Post copy/pasted from another thread (let us know if it works for you, it did for me).</p>
<blockquote>
<p>It took me quite some time to figure it out. Hopefully this is useful for someone.<br>
In my case I am using Jupyter Notebook which was causing the exact same error. I wrote a small blog post about it with code examples to a fully working PyTorch Multiprocessing with Queue.<br>
<a href="https://dataiskey.eu/jupyter-notebook-pytorch-multiprocessing/" rel="noopener nofollow ugc">https://dataiskey.eu/jupyter-notebook-pytorch-multiprocessing/</a><br>
Github with just the code example:<br>
<a href="https://github.com/olympus999/jupyter-notebook-pytorch-multiprocessing-queue" class="inline-onebox" rel="noopener nofollow ugc">GitHub - olympus999/jupyter-notebook-pytorch-multiprocessing-queue: A simple workaround to run Pytorch multiprocessing in Jupyter Notebook. Also showing performance difference between normal Queue (not sharing memory) and Pytorch queue (sharing memory).</a></p>
</blockquote>

username: SM2023
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I tried this code, but it gave me the error , I search ed the error but no specific solution came to my mind.</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader
from transformers import TextDataset,DataCollatorForLanguageModeling
#from transformers import  AutoModelWithLMHead
from transformers import  AutoModelForCausalLM
from transformers import AdamW, get_linear_schedule_with_warmup
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import numpy as np
import random
import torch
from torch.utils.data import Dataset, DataLoader
from transformers import GPT2Tokenizer, GPT2LMHeadModel, AdamW, get_linear_schedule_with_warmup
from tqdm import tqdm, trange
import gc
import math
import os
import time
import datetime
import torch
import torch.distributed as dist
import sys
## the directory include the package from INVIDIA
#sys.path.append('/home/momenisa/GPU_ZIP_Apex/apex-master/apex/')
#from apex import amp
#from apex.parallel import DistributedDataParallel as DDP
import torch.multiprocessing as mp
from torch.utils.data.distributed import DistributedSampler
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.distributed import init_process_group, destroy_process_group
import os
import random
######################
PathData='/home//NLP_Projects/CaseSummary_resolutionProject/Results_GPT_2/model_v4200_k_bs=16_lr=5e-05_epochs=20/'
pretrained_model='/home///GPT_2/'

weight_decay=0
learning_rate=5e-5
adam_epsilon=1e-8
warmup_steps = 1e2
lr=5e-5
Max_length=400

################################################
class GPT2Dataset(Dataset):

    def __init__(self, txt_list, tokenizer, gpt2_type=pretrained_model, max_length=400):

        self.tokenizer = tokenizer
        self.input_ids = []
        self.attn_masks = []

        for txt in txt_list:

            encodings_dict = tokenizer('&lt;|startoftext|&gt;'+ txt + '&lt;|endoftext|&gt;', truncation=True, max_length=max_length, padding="max_length")

            self.input_ids.append(torch.tensor(encodings_dict['input_ids']))
            self.attn_masks.append(torch.tensor(encodings_dict['attention_mask']))

    def __len__(self):
        return len(self.input_ids)

    def __getitem__(self, idx):
        return self.input_ids[idx], self.attn_masks[idx] 

######################################################3
def ddp_setup(rank, world_size):
    """
    Args:
        rank: Unique identifier of each process
        world_size: Total number of processes
    """
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "12355"
    init_process_group(backend="nccl", rank=rank, world_size=world_size)
#########################################################

def main(rank: int, world_size: int, save_every: int, total_epochs: int, batch_size: int):
        
    print(PathData)
    
    print(rank)

    ### defined variable ###############
    seed_val = 42
    random.seed(seed_val)
    np.random.seed(seed_val)
    torch.manual_seed(seed_val)
    torch.cuda.manual_seed_all(seed_val)

    ###############################
    
    ddp_setup(rank, world_size)
    
    ###############################
    
    tokenizer = GPT2Tokenizer.from_pretrained(pretrained_model, bos_token='&lt;|startoftext|&gt;', eos_token='&lt;|endoftext|&gt;', pad_token='&lt;|pad|&gt;') #gpt2-small

    model = GPT2LMHeadModel.from_pretrained(pretrained_model)

    model.resize_token_embeddings(len(tokenizer))

    ## loading traina and tets dataset
    print(PathData)
    trains_titles=pd.read_csv(PathData+'/'+'traindata.csv')
    valid_titles=pd.read_csv(PathData+'/'+'validdata.csv')

    train_dataset = GPT2Dataset(trains_titles, tokenizer, max_length=Max_length)

    Val_dataset = GPT2Dataset(valid_titles, tokenizer, max_length=Max_length)

    ################# define optimizer and scheduler#########################

    # Note: AdamW is a class from the huggingface library (as opposed to pytorch) 
    optimizer = AdamW(model.parameters(), lr = learning_rate,eps = adam_epsilon)

    total_steps = len(train_dataloader) * epochs

    # Create the learning rate scheduler.
    # This changes the learning rate as the training loop progresses
    scheduler = get_linear_schedule_with_warmup(optimizer, 
                                                num_warmup_steps = warmup_steps, 
                                                num_training_steps = total_steps)

    ############################## train_loader and validation_loader ######################3
    train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                                       batch_size=batch_size,
                                                        pin_memory=True,
                                                        shuffle=False,
                                                       sampler=DistributedSampler(train_dataset))


    validation_loader= torch.utils.data.DataLoader(dataset=Val_dataset,
                                                       batch_size=batch_size,
                                                        pin_memory=True,
                                                        shuffle=False,
                                                       sampler=DistributedSampler(Val_dataset))
   
    training_steps_per_epoch=len(train_loader)
    total_num_training_steps = int(training_steps_per_epoch*total_epochs)

  ######################## applying DDP on the model for training ############################
    model=model.to(gpu_id)
    model = DDP(model, device_ids=[gpu_id])
    # ========================================
    #               Training
    # ========================================

    print("")
    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))
    print('Training...')
        
    training_stats = []


    for epoch_i in range(0, total_epochs):

        ##########################################
        train_loader.sampler.set_epoch(epochs)
        b_sz = len(next(iter(train_loader))[0])
        print(f"[GPU{self.gpu_id}] Epoch {epoch} | Batchsize: {b_sz} | Steps: {len(self.train_data)}")
        train_loader.sampler.set_epoch(epoch)
        ##########################################

        t0 = time.time()

        total_train_loss = 0

        model.train()

        for step, batch in enumerate(train_loader):

            #################################
            b_input_ids = batch[0].to(gpu_id)
            b_labels = batch[0].to(gpu_id)
            b_masks = batch[1].to(gpu_id)
            #################################

            optimizer.zero_grad()        

            outputs = model(  b_input_ids,
                             labels=b_labels, 
                              attention_mask = b_masks,
                              token_type_ids=None
                            )

            loss = outputs[0]  
            batch_loss = loss.item()
            total_train_loss += batch_loss
            loss.backward()
            optimizer.step()
            scheduler.step()

        # Calculate the average loss over all of the batches.
        avg_train_loss = total_train_loss / len(train_dataloader)  

        del total_train_loss
        del batch_loss
        
        # Measure how long this epoch took.
        training_time = format_time(time.time() - t0)
        # ========================================
        #               Validation
        # ========================================

        print("")
        print("Running Validation...")

        avg_val_loss_1=[]
        t0 = time.time()
        #################### is this section corrcet for validation ???????????????????? #############
        model.eval()
        model = DDP(model, device_ids=[gpu_id])
        ########################################3
        total_eval_loss = 0
        nb_eval_steps = 0
        
        ########################################
        validation_loader.sampler.set_epoch(epochs)
        b_sz = len(next(iter(validation_loader))[0])
        print(f"[GPU{gpu_id}] Epoch {epoch} | Batchsize: {b_sz} | Steps: {len(validation_loader)}")
        validation_loader.sampler.set_epoch(epoch)
        ###########################################
        
        # Evaluate data for one epoch
        for batch in validation_loader:

            b_input_ids = batch[0].to(gpu_id)
            b_labels = batch[0].to(gpu_id)
            b_masks = batch[1].to(gpu_id)

            with torch.no_grad():        
                outputs  = model(b_input_ids,attention_mask = b_masks,labels=b_labels)
                loss = outputs[0]  
            batch_loss = loss.item()
            total_eval_loss += batch_loss        

        avg_val_loss = total_eval_loss / len(validation_dataloader)

        perplexity=math.exp(avg_val_loss)

        avg_val_loss_1.append(avg_val_loss)

        validation_time = format_time(time.time() - t0)    

        del total_eval_loss 


        print("  Validation Loss: {0:.2f}".format(avg_val_loss))
        print("  Validation took: {:}".format(validation_time))

        # Record all statistics from this epoch.
        training_stats.append(
            {
                'epoch': epoch_i + 1,
                'Training Loss': avg_train_loss,
                'Valid. Loss': avg_val_loss,
                'Training Time': training_time,
                'Validation Time': validation_time,
                'perplexity': perplexity
            }
        )
        gc.collect()
        
        ################### saving the model ########################

        if gpu_id == 0 and epoch_i % save_every == 0:

            Pathmodel=Results_Path+'/'+'savemodel_epoch='+str(epoch_i)

            ss=os.path.isdir(Pathmodel)
            if ss==False:
                os.makedirs(Pathmodel)

            ckp = model.module.state_dict()
            torch.save(ckp, Pathmodel+'/ "checkpoint.pt"')
    ############ save the results #####################3          
    Path_2=pt_save_directory+'/'+'training_stats='+str(0)+".csv"
    torch.save(training_stats,Path_2)

    #### is a good place to put the destrop process ??????????????????? ###########
    destroy_process_group()
    #############################
if __name__ == '__main__':
    import sys
    total_epochs=21
    save_every=5
    batch_size=32
    world_size = torch.cuda.device_count()
    print(world_size)
    mp.spawn(main, args=(world_size, save_every, total_epochs, batch_size), nprocs=world_size,join=True)

</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8bfa9d1b898c74c9489063a9e3ff9c5d614c8b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8bfa9d1b898c74c9489063a9e3ff9c5d614c8b3" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8bfa9d1b898c74c9489063a9e3ff9c5d614c8b3.png" alt="Capture" data-base62-sha1="o4OQODsY1buhyqQBkSRTt8N2NOj" width="690" height="337" data-dominant-color="F9D9D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">983×481 26.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The stacktrace doesn’t show the actual error message so I don’t know what exactly is failing. Try to execute your script in the terminal directly instead of inside a notebook to see if a better error message would be raised.</p>