Distributed parallel error:TypeError: unsupported operand type(s) for +: 'int' and 'CrossEntropyLoss'

username: nikiguo93
cooked: <p>the error message comes from:<br>
return next(self._sampler_iter)  # may raise StopIteration<br>
File “/home/nikiguo93/.conda/envs/branch-env/lib/python3.7/site-packages/torch/utils/data/sampler.py”, line 229, in <strong>iter</strong><br>
for idx in self.sampler:<br>
python3.7/site-packages/torch/utils/data/distributed.py", line 98, in <strong>iter</strong><br>
g.manual_seed(self.seed + self.epoch)<br>
TypeError: unsupported operand type(s) for +: ‘int’ and ‘CrossEntropyLoss’<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 6341) of binary: /home/usrname/.conda/envs/branch-env/bin/python…</p>
<p>can anybody help me with that?</p>

username: ptrblck
cooked: <p>Based on the error message you are trying to add an <code>int</code> to an object, which will fail, as seen here:</p>
<pre><code class="lang-python">criterion = nn.CrossEntropyLoss()
1 + criterion
# TypeError: unsupported operand type(s) for +: 'int' and 'CrossEntropyLoss'
</code></pre>
<p>so check where this line of code is executed and make sure to call the <code>criterion</code> with valid tensor inputs instead.</p>

username: nikiguo93
cooked: <p>Hi, thanks for your information. Solved now. and another question is that, when I used :<br>
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 file.py,<br>
and executing this line of code:<br>
if args.local_rank == 0:<br>
torch.save()<br>
there is error message saying that: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0, pytorch ddp.py’ not found.<br>
do you have some ideas about how to fix it?</p>

username: ptrblck
cooked: <p>I haven’t seen this error message before so could you show the content of <code>file.py</code>?</p>

username: nikiguo93
cooked: <p>import torch<br>
import gc<br>
gc.collect()<br>
torch.cuda.empty_cache()</p>
<p>import os<br>
import numpy as np<br>
from PIL import Image<br>
from torchvision import transforms,models, utils<br>
from sklearn import preprocessing<br>
import argparse<br>
import random<br>
from torch.utils.tensorboard import SummaryWriter<br>
from collections import defaultdict,OrderedDict<br>
from copy import deepcopy<br>
from torch.optim import Adam as Adam<br>
from torch.nn import CrossEntropyLoss as CrossEntropyLoss<br>
from torch.utils.data import Dataset, DataLoader<br>
import torch<br>
import random<br>
from torch.utils.data.distributed import DistributedSampler<br>
from apex.parallel import DistributedDataParallel as DDP<br>
from apex.parallel import convert_syncbn_model<br>
from apex import amp</p>
<p><span class="hashtag">#reproducibility</span><br>
random.seed(0)<br>
np.random.seed(0)<br>
torch.manual_seed(0)<br>
torch.cuda.manual_seed(0)<br>
torch.backends.cudnn.deterministic = True</p>
<p>parser = argparse.ArgumentParser()<br>
parser.add_argument(‘–sub_idx’, default=3, type=int)<br>
parser.add_argument(‘–categories’, default=‘51’, type=int)<br>
parser.add_argument(‘–writerpath’, default=‘/home/nikiguo93/python_examples/DataLoader/places365/runs/’, type=str)<br>
parser.add_argument(‘–modelname’, default=‘resnet50’, type=str)<br>
parser.add_argument(‘–arch’, default=‘resnet’, type=str)<br>
parser.add_argument(‘–lr’, default= 5e-6, type=float)<br>
parser.add_argument(‘–MaxEpoch’, default=150, type=int)<br>
parser.add_argument(‘–mtl’, default= ‘True’, type=str)<br>
parser.add_argument(“–local_rank”, default=os.getenv(‘LOCAL_RANK’, 0), type=int)</p>
<p>parser.add_argument(‘–dirpath’, default= ‘/scratch/nikiguo93/results_relu_fc_bk3_bk4/’, type=str)</p>
<p>args = parser.parse_args()</p>
<p>print(args.mtl)</p>
<p>writer = SummaryWriter(args.writerpath + ‘train_places365_mtl’ + str(args.mtl))</p>
<p>‘’’<br>
num_of_gpu = torch.cuda.device_count()<br>
print(num_of_gpu)<br>
os.environ[‘CUDA_VISIBLE_DEVICES’] = ‘0,1,2,3’<br>
os.environ[‘NCCL_DEBUG’]=‘INFO’<br>
os.environ[‘NCCL_DEBUG_SUBSYS’]=‘ALL’<br>
os.environ[‘TORCH_DISTRIBUTED_DEBUG’] = ‘INFO’</p>
<h1>
<a name="set-up-distributed-device-1" class="anchor" href="#set-up-distributed-device-1"></a>set up distributed device</h1>
<p>rank = int(os.environ[‘RANK’])<br>
local_rank = int(os.environ[‘LOCAL_RANK’])<br>
torch.cuda.set_device(rank % torch.cuda.device_count())<br>
torch.distributed.init_process_group(backend=‘nccl’ if torch.distributed.is_nccl_available() else ‘gloo’)<br>
device = torch.device(‘cuda’,local_rank)</p>
<p>print(f"[init] == local rank: {local_rank}, global rank: {rank} ==")<br>
‘’’</p>
<p>args.gpu = args.local_rank<br>
torch.cuda.set_device(args.gpu)<br>
torch.distributed.init_process_group(backend=‘nccl’,init_method=‘env://’)</p>
<p>class Places365Dataset(Dataset):</p>
<pre><code>def __init__(self,image_path,file_list,transform,train=True):
    
    train_ImageNames = []
    self.train = train
    self.transform = transform
    
    for FileName in os.listdir(image_path + 'train'):
        if FileName in file_list:
            for ImgName in os.listdir(image_path+'train/'+FileName):
                image_name = os.path.join(image_path,'train/',FileName,ImgName)
                train_ImageNames.append(image_name) 
    train_ImageNames.sort()
    
    test_ImageNames = []
    
    for FileName in os.listdir(image_path + 'val'):
        if FileName in file_list:
            for ImgName in os.listdir(image_path+'val/'+FileName):
                image_name = os.path.join(image_path,'val/',FileName,ImgName)
                test_ImageNames.append(image_name) 
    test_ImageNames.sort()
    
    
    train_label = [train_ImageNames[i].split('/')[5] for i in range(len(train_ImageNames))]
    test_label = [test_ImageNames[i].split('/')[5] for i in range(len(test_ImageNames))]
    
    

    b_train=random.sample(range(0, 999), 400)
    self.train_img_names = []
    self.train_lbl = []
    for i in range(len(train_ImageNames)):
        res = i%1000
        if res in b_train:
            self.train_img_names.append(train_ImageNames[i])
            self.train_lbl.append(train_label[i])
            
    b_test=random.sample(range(0, 99), 50)
    self.test_img_names = []
    self.test_lbl = []
    for i in range(len(test_ImageNames)):
        res = i%100
        if res in b_test:
            self.test_img_names.append(test_ImageNames[i])
            self.test_lbl.append(test_label[i])

    le = preprocessing.LabelEncoder()
    self.train_lbl = le.fit_transform(self.train_lbl)
    self.test_lbl = le.fit_transform(self.test_lbl)
    self.train_lbl = torch.as_tensor(self.train_lbl)
    self.test_lbl = torch.as_tensor(self.test_lbl)
    
        


def __len__(self):
    if self.train:
        return len(self.train_img_names)
    else:
        return len(self.test_img_names)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
        
    if self.train:    
        TheImage = self.transform(Image.open(self.train_img_names[idx]).convert('RGB'))
        TheLabel = self.train_lbl[idx]
    else:
        TheImage = self.transform(Image.open(self.test_img_names[idx]).convert('RGB'))
        TheLabel = self.test_lbl[idx]
        
    sample = {'input': TheImage, 'label': TheLabel}

    return sample
</code></pre>
<p>image_path = ‘/scratch/nikiguo93/places365/’</p>
<p>file_list = [‘mountain_snowy’,‘bakery-shop’,‘lighthouse’,‘beach’,‘ball_pit’,‘dining_room’,‘highway’,‘bridge’,‘creek’,<br>
‘waterfall’,‘forest-broadleaf’,‘nursery’,‘campsite’,‘car_interior’,‘toyshop’,‘shoe_shop’,‘bookstore’,‘coast’,‘ocean’,‘corridor’,<br>
‘skyscraper’,‘water_tower’,‘closet’,‘landing_deck’,‘florist_shop-indoor’,‘park’,‘pasture’,‘pantry’,‘swimming_pool-outdoor’,<br>
‘supermarket’,‘conference_room’,‘corn_field’,‘movie_theater-indoor’,‘laundromat’,‘staircase’,‘street’,‘home_theater’,‘utility_room’,<br>
‘auditorium’,‘river’,‘castle’,‘butchers_shop’,‘windmill’,‘desert-sand’,‘tower’,‘cockpit’,‘office’,‘kitchen’,‘carrousel’,<br>
‘playground’,‘wheat_field’]</p>
<p>normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],<br>
std=[0.229, 0.224, 0.225])</p>
<p>transform = transforms.Compose([<br>
transforms.Resize(224),<br>
transforms.ToTensor(),<br>
normalize,<br>
])</p>
<p>PLACES_Traindataset = Places365Dataset(image_path,file_list,transform,train=True)<br>
PLACES_Testdataset = Places365Dataset(image_path,file_list,transform,train=False)</p>
<p>train_sampler = torch.utils.data.distributed.DistributedSampler(PLACES_Traindataset,shuffle=True,)<br>
val_sampler = torch.utils.data.distributed.DistributedSampler(PLACES_Testdataset)</p>
<p>TraindataLoader = DataLoader(PLACES_Traindataset, batch_size=64,num_workers=8,pin_memory=True,sampler=train_sampler)<br>
TestdataLoader = DataLoader(PLACES_Testdataset, batch_size=64,num_workers=8,shuffle=False,sampler=val_sampler)</p>
<p>del PLACES_Traindataset<br>
del PLACES_Testdataset</p>
<p>def load_model(model,modelpath):</p>
<pre><code>if os.path.isfile(modelpath):
    print("=&gt; loading checkpoint '{}'".format(modelpath))
    checkpoint = torch.load(modelpath)
    

state_dict = checkpoint['best_model']
model_dict = model.state_dict()
new_state_dict = OrderedDict()
matched_layers, discarded_layers = [], []

for k, v in state_dict.items():
    if k.startswith('module.'):
        k = k[7:] # discard module.

    if k in model_dict and model_dict[k].size() == v.size():
            new_state_dict[k] = v
            matched_layers.append(k)
else:
    discarded_layers.append(k)

model_dict.update(new_state_dict)
model.load_state_dict(model_dict)
model.eval().cuda()
</code></pre>
<p>class CustomResNet(torch.nn.Module):<br>
def <strong>init</strong>(self,modelname):<br>
super(CustomResNet, self).<strong>init</strong>()<br>
resnet50 = models.<strong>dict</strong><a>modelname</a><br>
self.features_1= torch.nn.ModuleList(resnet50.children())[:-2]<br>
self.features_2= torch.nn.ModuleList(resnet50.children())[:-3]<br>
self.features_3= torch.nn.ModuleList(resnet50.children())[:-4]<br>
self.features_4= torch.nn.ModuleList(resnet50.children())[:-5]</p>
<pre><code>    self.part_1 = torch.nn.Sequential(
              *self.features_1,
              torch.nn.ReLU(),
              torch.nn.Flatten(),
              torch.nn.Linear(100352,256)
             ) 

    self.part_2 = torch.nn.Sequential(
              *self.features_2,
              torch.nn.ReLU(),
              torch.nn.Flatten(),
              torch.nn.Linear(200704,256)
             ) 
    

    
    self.regression = torch.nn.Sequential(
                      torch.nn.ReLU(),
                      torch.nn.Linear(512,1700))
    
    self.classification = torch.nn.Sequential(
                    *self.features_1,
                    torch.nn.AdaptiveAvgPool2d((1, 1)),
                    torch.nn.Flatten(),
                    torch.nn.Linear(2048,1654))

    
       
def forward(self, x):
    
    x1 = self.part_1(x)
    x2 = self.part_2(x)
    x_eeg = torch.cat([x1,x2],axis=1)
    
    #x_eeg = x_eeg.view(x_eeg.size(0),-1)
    x_eeg = self.regression(x_eeg)
    x_img = self.classification(x)
    
    return x_eeg,x_img
</code></pre>
<p>if args.mtl == ‘True’:<br>
print(args.mtl)<br>
mtl = CustomResNet(args.modelname)<br>
modelpath = args.dirpath + ‘real_zeeg_sub’ + str(args.sub_idx) + ‘_resnet50.pth’<br>
load_model(model = mtl,modelpath=modelpath)<br>
model_ft = mtl.classification<br>
del mtl<br>
num_ftrs=model_ft[10].in_features<br>
model_ft[10] = torch.nn.Linear(num_ftrs, args.categories,bias = True)<br>
else:<br>
model_ft = models.resnet50(pretrained=True)<br>
num_ftrs=model_ft.fc.in_features<br>
model_ft.fc = torch.nn.Linear(num_ftrs, 1654,bias = True)<br>
modelpath = args.dirpath + ‘finetune_classification_resnet50_all.pth’<br>
load_model(model_ft,modelpath)<br>
num_ftrs=model_ft.fc.in_features<br>
model_ft.fc = torch.nn.Linear(num_ftrs, args.categories,bias = True)</p>
<p><span class="hashtag">#freeze</span> part of the resnet<br>
count = 0<br>
for child in model_ft.children():<br>
count+=1<br>
if count &lt; 9:<br>
for param in child.parameters():<br>
param.requires_grad = False</p>
<p>model_ft.cuda()</p>
<p><span class="hashtag">#parameters</span> to optimized<br>
params_to_update = model_ft.parameters()<br>
print(“Params to learn:”)<br>
for name,param in model_ft.named_parameters():<br>
if param.requires_grad == True:<br>
print(“\t”,name)</p>
<p><span class="hashtag">#define</span> the optimizer<br>
<span class="hashtag">#optimizer_ft</span> = SGD(params_to_update, lr = 0.01,momentum=0.09,weight_decay=0.0005)<br>
optimizer_ft = Adam(params_to_update, lr=args.lr)</p>
<p>model_ft,optimizer_ft = amp.initialize(model_ft,optimizer_ft,opt_level=‘O2’)<br>
model_ft = DDP(model_ft,delay_allreduce=True)</p>
<p>def reduce_value(value, average=True):<br>
world_size = torch.distributed.get_world_size()<br>
if world_size &lt; 2:<br>
return value</p>
<pre><code>with torch.no_grad():
    torch.distributed.all_reduce(value)
    if average:
        value /= world_size
return value
</code></pre>
<p>def train(TraindataLoader,optimizer,model,criterion,ep):</p>
<pre><code>model.train()
train_loss = 0
num_correct = 0.0
batch_size = 0
#set sampler
TraindataLoader.sampler.set_epoch(ep)

for i_batch, sample_batched in enumerate(TraindataLoader):
    local_X = sample_batched['input'].cuda()
    local_Y = sample_batched['label'].cuda()

    local_X =  torch.autograd.Variable(local_X)
    local_Y =  torch.autograd.Variable(local_Y)

        
    # compute gradient and do SGD step
    optimizer.zero_grad()
    
    # compute the model output
    with torch.set_grad_enabled(True):
        yhat = model(local_X)
        print(yhat.size())
        print(local_Y.size())
        
        #yhat= torch.reshape(yhat,[yhat.size()[0],17,100])
        # calculate loss
        loss = criterion(yhat, local_Y)
        
        '''
        pred_category,indx_pred = yhat.max(1)
        num_correct += indx_pred.eq(local_Y).sum().item()
        
        
        # credit assignment
        loss.backward()
        '''
        #############################################################
        with amp.scale_loss(loss, optimizer) as scaled_loss:
            scaled_loss.backward()

        #############################################################

        #calculate loss
        loss = reduce_value(loss,average=True)
        train_loss += loss.item()*yhat.size()[0]
        batch_size += yhat.size()[0]

        # update model weights
        optimizer.step()

        #measure accuracy
        indx_pred = torch.max(yhat,dim=1)[1]
        num_correct += torch.eq(indx_pred,local_Y).sum()
        
    #scheduler.step()
num_correct = reduce_value(num_correct,average=True).item()
acc = num_correct/batch_size
avg_trainloss = train_loss/batch_size

torch.cuda.synchronize()
            
return avg_trainloss,acc            
</code></pre>
<p>def test(TestdataLoader, model,criterion):</p>
<pre><code>test_loss = 0
num_correct = 0.0
batch_size = 0

# switch to evaluate mode
model.eval()

with torch.no_grad():
    for i_batch, sample_batched in enumerate(TestdataLoader):
        local_X = sample_batched['input'].cuda()
        local_Y = sample_batched['label'].cuda()
 
        local_X =  torch.autograd.Variable(local_X)
        local_Y =  torch.autograd.Variable(local_Y)
        
        # compute the model output
        yhat = model(local_X)
        
        #yhat= torch.reshape(yhat,[yhat.size()[0],17,100])

        # calculate loss
        loss = criterion(yhat, local_Y)
        loss = reduce_value(loss,average=True)
        batch_size += yhat.size()[0]
        test_loss += loss.item()*yhat.size()[0]
        
        #measure accuracy
        indx_pred = torch.max(yhat,dim=1)[1]
        num_correct += torch.eq(indx_pred,local_Y).sum()
        
        #batch_size += yhat.size()[0]
    num_correct = reduce_value(num_correct,average=True).item()    
    acc = num_correct/batch_size    
    avg_testloss = test_loss/batch_size

torch.cuda.synchronize()    
    
return avg_testloss,acc
</code></pre>
<p>MaxEpoch = args.MaxEpoch<br>
min_loss = 50000<br>
best_model = None<br>
best_epoch = -1<br>
train_batches = 64<br>
test_batches = 64</p>
<h1>
<a name="define-the-optimization-2" class="anchor" href="#define-the-optimization-2"></a>define the optimization</h1>
<p>criterion = CrossEntropyLoss().cuda()<br>
<span class="hashtag">#scheduler</span> = torch.optim.lr_scheduler.StepLR(optimizer_ft, step_size=1000, gamma=0.1)</p>
<p>def adjust_learning_rate(optimizer, epoch, init_lr):<br>
“”“Sets the learning rate to the initial LR decayed by 10 every 30 epochs”“”<br>
lr = init_lr * (0.1 ** (epoch // 30))<br>
for param_group in optimizer.param_groups:<br>
param_group[‘lr’] = lr</p>
<p>for epoch in range(MaxEpoch):<br>
[avg_trainloss,acc_train] = train(TraindataLoader,optimizer_ft,model_ft,criterion,epoch)<br>
writer.add_scalar(“Loss/train”, avg_trainloss, epoch+1)<br>
writer.add_scalar(“acc_train/train”, acc_train, epoch+1)<br>
[avg_testloss,acc_test] = test(TestdataLoader, model_ft,criterion)<br>
<span class="hashtag">#scheduler</span>.step()</p>
<pre><code>if avg_testloss &lt;= min_loss:
    min_loss = avg_testloss
    best_model = deepcopy(model_ft.state_dict())
    best_epoch = epoch+1
writer.add_scalar("Loss/test", avg_testloss, epoch+1)
writer.add_scalar("acc_test/test", acc_test, epoch+1)
writer.flush()
</code></pre>
<p>print("Best Epoch is: ", best_epoch , " with test loss: ", min_loss)<br>
model_state_dict= model_ft.state_dict()<br>
results_dir = ‘/scratch/nikiguo93/places365/results/’</p>
<h1>
<a name="save-model-3" class="anchor" href="#save-model-3"></a>save model</h1>
<p>if args.local_rank == 0:<br>
torch.save({<br>
‘epoch’: epoch,<br>
‘model_state_dict’: model_state_dict,<br>
‘best_model’: best_model,<br>
‘optimizer_state_dict’: optimizer_ft.state_dict(),<br>
‘loss’: avg_trainloss,<br>
}, results_dir + ‘places365_mtl_’ + str(args.mtl) + ‘.pth’)</p>
<p>writer.close()</p>

username: nikiguo93
cooked: <p>FutureWarning,<br>
WARNING:torch.distributed.run:</p>
<hr>
<p>Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.</p>
<hr>
<h1>
<a name="homenikiguo93condaenvsbranch-envbinpython-cant-open-file-torch_ddppy-errno-2-no-such-file-or-directory-homenikiguo93condaenvsbranch-envbinpython-cant-open-file-torch_ddppy-errno-2-no-such-file-or-directory-errortorchdistributedelasticmultiprocessingapifailed-exitcode-2-local_rank-0-pid-120134-of-binary-homenikiguo93condaenvsbranch-envbinpython-traceback-most-recent-call-last-file-homenikiguo93condaenvsbranch-envlibpython37runpypy-line-193-in-_run_module_as_main-__main__-mod_spec-file-homenikiguo93condaenvsbranch-envlibpython37runpypy-line-85-in-_run_code-execcode-run_globals-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlaunchpy-line-193-in-module-main-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlaunchpy-line-189-in-main-launchargs-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlaunchpy-line-174-in-launch-runargs-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedrunpy-line-713-in-run-cmd_args-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlauncherapipy-line-131-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlauncherapipy-line-261-in-launch_agent-failuresresultfailures-torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#homenikiguo93condaenvsbranch-envbinpython-cant-open-file-torch_ddppy-errno-2-no-such-file-or-directory-homenikiguo93condaenvsbranch-envbinpython-cant-open-file-torch_ddppy-errno-2-no-such-file-or-directory-errortorchdistributedelasticmultiprocessingapifailed-exitcode-2-local_rank-0-pid-120134-of-binary-homenikiguo93condaenvsbranch-envbinpython-traceback-most-recent-call-last-file-homenikiguo93condaenvsbranch-envlibpython37runpypy-line-193-in-_run_module_as_main-__main__-mod_spec-file-homenikiguo93condaenvsbranch-envlibpython37runpypy-line-85-in-_run_code-execcode-run_globals-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlaunchpy-line-193-in-module-main-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlaunchpy-line-189-in-main-launchargs-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlaunchpy-line-174-in-launch-runargs-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedrunpy-line-713-in-run-cmd_args-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlauncherapipy-line-131-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-homenikiguo93condaenvsbranch-envlibpython37site-packagestorchdistributedlauncherapipy-line-261-in-launch_agent-failuresresultfailures-torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a>/home/nikiguo93/.conda/envs/branch-env/bin/python: can’t open file ‘torch_ddp.py’: [Errno 2] No such file or directory<br>
/home/nikiguo93/.conda/envs/branch-env/bin/python: can’t open file ‘torch_ddp.py’: [Errno 2] No such file or directory<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 2) local_rank: 0 (pid: 120134) of binary: /home/nikiguo93/.conda/envs/branch-env/bin/python<br>
Traceback (most recent call last):<br>
File “/home/nikiguo93/.conda/envs/branch-env/lib/python3.7/runpy.py”, line 193, in _run_module_as_main<br>
“<strong>main</strong>”, mod_spec)<br>
File “/home/nikiguo93/.conda/envs/branch-env/lib/python3.7/runpy.py”, line 85, in _run_code<br>
exec(code, run_globals)<br>
File “/home/nikiguo93/.conda/envs/branch-env/lib/python3.7/site-packages/torch/distributed/launch.py”, line 193, in <br>
main()<br>
File “/home/nikiguo93/.conda/envs/branch-env/lib/python3.7/site-packages/torch/distributed/launch.py”, line 189, in main<br>
launch(args)<br>
File “/home/nikiguo93/.conda/envs/branch-env/lib/python3.7/site-packages/torch/distributed/launch.py”, line 174, in launch<br>
run(args)<br>
File “/home/nikiguo93/.conda/envs/branch-env/lib/python3.7/site-packages/torch/distributed/run.py”, line 713, in run<br>
)(*cmd_args)<br>
File “/home/nikiguo93/.conda/envs/branch-env/lib/python3.7/site-packages/torch/distributed/launcher/api.py”, line 131, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/nikiguo93/.conda/envs/branch-env/lib/python3.7/site-packages/torch/distributed/launcher/api.py”, line 261, in launch_agent<br>
failures=result.failures,<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2>
<a name="torch_ddppy-failed-2" class="anchor" href="#torch_ddppy-failed-2"></a>torch_ddp.py FAILED</h2>
<h2>
<a name="failures-1-time-2023-06-09_022820-host-g010curtazedatfu-berlinde-rank-1-local_rank-1-exitcode-2-pid-120135-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-3" class="anchor" href="#failures-1-time-2023-06-09_022820-host-g010curtazedatfu-berlinde-rank-1-local_rank-1-exitcode-2-pid-120135-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-3"></a>Failures:<br>
[1]:<br>
time      : 2023-06-09_02:28:20<br>
host      : <a href="http://g010.curta.zedat.fu-berlin.de" rel="noopener nofollow ugc">g010.curta.zedat.fu-berlin.de</a><br>
rank      : 1 (local_rank: 1)<br>
exitcode  : 2 (pid: 120135)<br>
error_file: &lt;N/A&gt;<br>
traceback : To enable traceback see: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 2.0 documentation</a>
</h2>
<h1>
<a name="root-cause-first-observed-failure-0-time-2023-06-09_022820-host-g010curtazedatfu-berlinde-rank-0-local_rank-0-exitcode-2-pid-120134-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4" class="anchor" href="#root-cause-first-observed-failure-0-time-2023-06-09_022820-host-g010curtazedatfu-berlinde-rank-0-local_rank-0-exitcode-2-pid-120134-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4"></a>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2023-06-09_02:28:20<br>
host      : <a href="http://g010.curta.zedat.fu-berlin.de" rel="noopener nofollow ugc">g010.curta.zedat.fu-berlin.de</a><br>
rank      : 0 (local_rank: 0)<br>
exitcode  : 2 (pid: 120134)<br>
error_file: &lt;N/A&gt;<br>
traceback : To enable traceback see: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 2.0 documentation</a>
</h1>

username: nikiguo93
cooked: <p>Hi my code looks like that and you can find the error message after the code</p>

username: ptrblck
cooked: <p>Your code is not properly formatted so quite hard to read.<br>
However, it also doesn’t contain any <code>ddp.py</code> mention so it’s unclear where the error is being raised from.</p>