Multiple exits distributed data parallel model issue

Question:
Hello PyTorch Community,
Hope everyone is well!
Recently we got some extra GPU’s added to my labs machine since we need to profile certain models for a research project which really quickly overwhelm a single GPU so data parallelisation seemed to be the obvious solution.
Before I explain my issue please note that the model runs on a single GPU (Although, the problem might be in the model but only cause an error in the more “delicate” distributed case).
To be honest I have no idea what is really going on… This is my first time working with the DDP and I have had a dozen or so different DDP related errors which I worked through and finally I ended up with this one which is a model related problem weirdly only occurring on the DDP case.
def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    dist.init_process_group("nccl", rank=rank, world_size=world_size)

def cleanup():
    dist.destroy_process_group()

def main(rank,s,bs,epochs,world_size):
    setup(rank, world_size)

    dataloader = prepare(s,rank, world_size, bs)

    ddp_model = DDP(my_net(s).to(rank), device_ids=[rank], output_device=rank, find_unused_parameters=False)

    size = len(dataloader.dataset)

    optimizer = torch.optim.SGD(ddp_model.parameters(), lr=0.01, momentum=0.9)
    critireon = nn.CrossEntropyLoss()

    ddp_model.train()

    for epoch in range(epochs):
        dataloader.sampler.set_epoch(epoch)

        for step, (X, y) in enumerate(dataloader):
            X, y = X.to(rank), y.to(rank)

            preds = []
            for e in ddp_model.module.exits:
                preds.append(ddp_model(X, e))

            losses = []
            for y_hat in preds:
                losses.append(critireon(y_hat,y))

            loss = losses[0] * 0.15 + losses[1] * 0.15 +losses[2] * 0.6

            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

            if rank == 0 and step % 50 == 0:
                current = step * len(X)
                for indx,l in enumerate(losses):
                    tloss = l.item()
                    print(f"loss of exit {indx}: {tloss:>7f}  [{current:>5d}/{size:>5d}]")
                print('\n')

    cleanup()

def data_parallel():
    s = int(argv[1])
    bs = int(argv[2])
    epochs = int(argv[3])

    world_size = 3
    mp.spawn(
        main,
        args=(s,bs,epochs,world_size,),
        nprocs=world_size,
        join=True
    )

The above is the DDP code I wrote for training the model. The training loop exactly the same as the single GPU case (posted below for reference as well as the model).
Note: As you can guess form the above code, the model implements an early existing mechanism. Although the current model, inception V1, does have this early exits to prevent gradient issues, here we are interested in their performance and also the rest of the models have been augmented in a similar way with early exists so its trivial for the model to train in this way. Please also note that the current implementation (reference the model code below) is not very optimal but that is beyond the scope of this .
The error I am getting while is the following (stderr with detect_anomaly==True):
[W python_anomaly_mode.cpp:104] Warning: Error detected in CudnnBatchNormBackward0. Traceback of forward call that caused the error:
  File "<string>", line 1, in <module>
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
    return self._bootstrap(parent_sentinel)
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/george/edge ai/train_googlenet.py", line 42, in main
    preds.append(ddp_model(X, e))
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/george/edge ai/google_net.py", line 326, in forward
    x = l(x)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/george/edge ai/google_net.py", line 92, in forward
    x = self.bn(x)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
    return F.batch_norm(
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/functional.py", line 2282, in batch_norm
    return torch.batch_norm(
 (function _print_stack)
Traceback (most recent call last):
  File "/home/george/edge ai/train_googlenet.py", line 173, in <module>
    data_parallel()
  File "/home/george/edge ai/train_googlenet.py", line 70, in data_parallel
    mp.spawn(
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/george/edge ai/train_googlenet.py", line 50, in main
    loss.backward()
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128]] is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Although I understand what the issue is, I have but the slightest clue what is causing this since it seems like something which should be also affecting the single GPU model but it does not… Bellow you can find the model arch, the distributed dataloader and the single_GPU training loop. Sorry for the wall of code just making sure everything one might need to help me is present.
Line with the BatchNorm is highlighted in the code with a comment.
Thanks a lot for everyone helping users in the forum. You are all heros.
All the best,
George
EDIT:
I failed to mention this since I have had many issues over the course of this parallelisation journey of mine but I have tried this and I got the following error:
[W python_anomaly_mode.cpp:104] Warning: Error detected in torch::autograd::AccumulateGrad. Traceback of forward call that caused the error:
  File "<string>", line 1, in <module>
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
    return self._bootstrap(parent_sentinel)
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/george/edge ai/train_googlenet.py", line 25, in main
    ddp_model = DDP(my_net(s).to(rank), device_ids=[rank], output_device=rank, find_unused_parameters=True)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 587, in __init__
    self._ddp_init_helper(parameters, expect_sparse_gradient, param_to_name_mapping)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 632, in _ddp_init_helper
    self.reducer = dist.Reducer(
 (function _print_stack)
Traceback (most recent call last):
  File "/home/george/edge ai/train_googlenet.py", line 173, in <module>
    data_parallel()
  File "/home/george/edge ai/train_googlenet.py", line 70, in data_parallel
    mp.spawn(
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/george/edge ai/train_googlenet.py", line 50, in main
    loss.backward()
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
Parameter at index 186 has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration. You can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print parameter names for further debugging.

When using a static graph I got the following
RuntimeError: Your training graph has changed in this iteration, e.g., one parameter is unused in first iteration, but then got used in the second iteration. this is not compatible with static_graph set to True.

So I assume this also leads to a dead end unless, I am missing something.
A possible solution might be hidden in this:
Please make sure model parameters are not shared across multiple concurrent forward-backward passes

Although I have no clue about which/ or how parameters are shared across the models since each forward backward pass is separate with each other at least in the code I implemented (not sure about the PyTorch backend but I assume some parameter sharing must be going on for the synch of gradients).

Edit:

With debugger on I get the following more specific message

Parameter at index 185 with name e1_fc2.weight has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration.
def prepare(size, rank, world_size, batch_size=32, pin_memory=False, num_workers=0):
    t = transforms.Compose([transforms.Resize((size+10, size+10)),
                        transforms.RandomCrop((size, size)),
                        transforms.RandomHorizontalFlip(p=0.5),
                        transforms.ToTensor()])

    dataset = datasets.CIFAR10(
        root='data',
        train=True,
        download=True,
        transform=t
    )

    sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank, shuffle=False, drop_last=False)

    dataloader = DataLoader(dataset, batch_size=batch_size,
                         pin_memory=pin_memory, num_workers=num_workers,
                        drop_last=False, shuffle=False, sampler=sampler)

    return dataloader

class BasicConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, **kwargs):
        super(BasicConv2d, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs)
        self.bn = nn.BatchNorm2d(out_channels, eps=0.001)

    def forward(self, x):
        x = self.conv(x)

##### THIS CAUSES THE ERROR
        x = self.bn(x)

        x = F.relu(x)
        return x

class my_inception(nn.Module):
    def __init__(self, in_size, g1out, g2hid, g2out, g3hid, g3out, g4out):
        super(my_inception, self).__init__()

        self.g1c1x1 = BasicConv2d(in_size, g1out, kernel_size=1, stride=1, padding=0)

        self.g2c1x1 = BasicConv2d(in_size, g2hid, kernel_size=1, stride=1, padding=0)
        self.g2c3x3 = BasicConv2d(g2hid, g2out, kernel_size=3, stride=1, padding=1)

        self.g3c1x1 = BasicConv2d(in_size, g3hid, kernel_size=1, stride=1, padding=0)
        self.g3c5x5 = BasicConv2d(g3hid, g3out, kernel_size=5, stride=1, padding=2)

        self.g4mp = nn.MaxPool2d(kernel_size=3, stride=1, padding=1)
        self.g4c1x1 = BasicConv2d(in_size, g4out, kernel_size=1, stride=1, padding=0)

    def forward(self, x):
        g1_out = self.g1c1x1(x)

        g2_out = self.g2c1x1(x)
        g2_out = self.g2c3x3(g2_out)

        g3_out = self.g3c1x1(x)
        g3_out = self.g3c5x5(g3_out)

        g4_out = self.g4mp(x)
        g4_out = self.g4c1x1(g4_out)

        return torch.cat([g1_out, g2_out, g3_out, g4_out], dim=1)

class my_net(nn.Module):
    def __init__(self, img_size, num_out=10):
        super(my_net, self).__init__()

            ### CORE ###
        self.cc11 = BasicConv2d(3, 64, kernel_size=7, stride=2, padding=3)

        self.cmp1 = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        self.clrn1 = nn.LocalResponseNorm(2)

        self.cc21 = BasicConv2d(64, 192, kernel_size=1, stride=1, padding=1)

        self.cc22 = BasicConv2d(192, 192, kernel_size=3, stride=1, padding=0)

        self.cmp2 = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        self.clrn2 = nn.LocalResponseNorm(2)

        self.cinc3a = my_inception(192, 64, 96, 128, 16, 32, 32)

        self.cinc3b = my_inception(256, 128, 128, 192, 32, 96, 64)

        self.cmp3 = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        self.cinc4a = my_inception(480, 192, 96, 208, 16, 48, 64)

        self.cinc4b = my_inception(512, 160, 112, 224, 24, 64, 64)

        self.cinc4c = my_inception(512, 128, 128, 256, 24, 64, 64)

        self.cinc4d = my_inception(512, 112, 144, 288, 32, 64, 64)

        self.cinc4e = my_inception(528, 256, 160, 320, 32, 128, 128)

        self.cmp4 = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        self.cinc5a = my_inception(832, 256, 160, 320, 32, 128, 128)

        self.cinc5b = my_inception(832, 384, 192, 384, 48, 128, 128)

        if img_size == 224:
            self.cap = nn.AvgPool2d(kernel_size=7, stride=1, padding=0)
            self.cfl = nn.Flatten()
            self.cdo1 = nn.Dropout(0.4)
            self.cfc1 = nn.Linear(1024, 10)
        elif img_size == 512:
            self.cap = nn.AvgPool2d(kernel_size=14, stride=1, padding=0)
            self.cfl = nn.Flatten()
            self.cdo1 = nn.Dropout(0.4)
            self.cfc1 = nn.Linear(9216, 10)
        elif img_size == 1024:
            self.cap = nn.AvgPool2d(kernel_size=28, stride=1, padding=0)
            self.cfl = nn.Flatten()
            self.cdo1 = nn.Dropout(0.4)
            self.cfc1 = nn.Linear(25600, 10)

        self.core = [
            self.cc11,
            self.cmp1 ,
            self.clrn1 ,
            self.cc21,
            self.cc22 ,
            self.cmp2 ,
            self.clrn2,
            self.cinc3a ,
            self.cinc3b,

            self.cmp3,

            self.cinc4a,
            self.cinc4b,
            self.cinc4c,
            self.cinc4d,
            self.cinc4e,

            self.cmp4,

            self.cinc5a,
            self.cinc5b,

            self.cap,

            self.cfl,
            self.cdo1,
            self.cfc1,
        ]
            # exit 0
        if img_size == 224:
            self.e0_ap = nn.AvgPool2d(kernel_size=5, stride=3, padding=0)
            self.e0_conv = BasicConv2d(512, 128, kernel_size=1, stride=1, padding=0)
            self.e0_flat = nn.Flatten()
            self.e0_fc1 = nn.Linear(2048, 1024)
        elif img_size == 512:
            self.e0_ap = nn.AvgPool2d(kernel_size=10, stride=3, padding=0)
            self.e0_conv = BasicConv2d(512, 128, kernel_size=1, stride=1, padding=0)
            self.e0_flat = nn.Flatten()
            self.e0_fc1 = nn.Linear(8192, 1024)
        elif img_size == 1024:
            self.e0_ap = nn.AvgPool2d(kernel_size=20, stride=3, padding=0)
            self.e0_conv = BasicConv2d(512, 128, kernel_size=1, stride=1, padding=0)
            self.e0_flat = nn.Flatten()
            self.e0_fc1 = nn.Linear(28800, 1024)

        self.e0_r = nn.ReLU(True)
        self.e0_do = nn.Dropout(0.7)
        self.e0_fc2 = nn.Linear(1024, num_out)

        self.exit0 = [
            self.cc11,
            self.cmp1 ,
            self.clrn1 ,
            self.cc21,
            self.cc22 ,
            self.cmp2 ,
            self.clrn2,
            self.cinc3a ,
            self.cinc3b,

            self.cmp3,

            self.cinc4a,

            self.e0_ap,
            self.e0_conv,

            self.e0_flat,

            self.e0_fc1,
            self.e0_r,
            self.e0_do,
            self.e0_fc2,
        ]
            # exit 1
        if img_size == 224:
            self.e1_ap = nn.AvgPool2d(kernel_size=5, stride=3, padding=0)
            self.e1_conv = BasicConv2d(528, 128, kernel_size=1, stride=1, padding=0)
            self.e1_flat = nn.Flatten()
            self.e1_fc1 = nn.Linear(2048, 1024)
        elif img_size == 512:
            self.e1_ap = nn.AvgPool2d(kernel_size=10, stride=3, padding=0)
            self.e1_conv = BasicConv2d(528, 128, kernel_size=1, stride=1, padding=0)
            self.e1_flat = nn.Flatten()
            self.e1_fc1 = nn.Linear(8192, 1024)
        elif img_size == 1024:
            self.e1_ap = nn.AvgPool2d(kernel_size=20, stride=3, padding=0)
            self.e1_conv = BasicConv2d(528, 128, kernel_size=1, stride=1, padding=0)
            self.e1_flat = nn.Flatten()
            self.e1_fc1 = nn.Linear(28800, 1024)

        self.e1_r = nn.ReLU(True)
        self.e1_do = nn.Dropout(0.7)
        self.e1_fc2 = nn.Linear(1024, num_out)

        self.exit1 = [
            self.cc11,
            self.cmp1 ,
            self.clrn1 ,
            self.cc21,
            self.cc22 ,
            self.cmp2 ,
            self.clrn2,
            self.cinc3a ,
            self.cinc3b,

            self.cmp3,

            self.cinc4a,
            self.cinc4b,
            self.cinc4c,
            self.cinc4d,

            self.e1_ap,
            self.e1_conv,

            self.e1_flat,

            self.e1_fc1,
            self.e1_r,
            self.e1_do,
            self.e1_fc2,
        ]

        self.exits = [self.exit0,
                      self.exit1,
                      self.core
                     ]

    def forward(self, x, exit_point=-1):
        if isinstance(exit_point, int):
            layers = self.exits[exit_point]
        else:
            layers = exit_point

        for l in layers:
            if isinstance(l, nn.Flatten):
                x = torch.flatten(x, 1)
            else:
                x = l(x)

        return x

def single_GPU():
    def train_exits(dataloader, model, loss_fn, optimizer):
        size = len(dataloader.dataset)
        model.train()
        for batch, (X, y) in enumerate(dataloader):
            X, y = X.to(device), y.to(device)

            # Compute prediction error
            pred_core = []
            for e in model.exits:
                pred_core.append(model(X, e))

            losses = []
            for y_hat in pred_core:
                losses.append(loss_fn(y_hat,y))

            loss = losses[0] * 0.15 + losses[1] * 0.15 +losses[2] * 0.6

            # Backpropagation
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if batch % 50 == 0:
                current = batch * len(X)
                for indx,l in enumerate(losses):
                    tloss = l.item()
                    print(f"loss of exit {indx}: {tloss:>7f}  [{current:>5d}/{size:>5d}]")
                print('\n')

     # omitted for simplicity
    def test_exits(dataloader, model, loss_fn=nn.CrossEntropyLoss()):

    s = int(argv[1])
    bs = int(argv[2])
    epochs = int(argv[3])

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    model = my_net(s).to(device)

    model.train()

    optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
    critireon = nn.CrossEntropyLoss()

    log_name = f'google logs size {s} batch_size {bs} epochs {epochs}.txt'

    for t in range(epochs):
        with open(log_name, 'a') as f:
            f.write(f'epoch: {t} \n')
        print('EPOCH:', t)
        train, test = get_data(s, batch_size=bs)
        train_exits(train, model, critireon, optimizer)
        test_exits(test, model)

        torch.save(model.state_dict(), f'google size {s} epochs {t} bs {bs}.pt')
        try:
            os.remove(f'google size {s} epochs {t-1} bs {bs}.pt')
        except:
            pass

Context:
Thanks for posting the question @George_Diamantop From the code snippets it seems like you are doing some early exits, which means not all parameters will participate in the calculation of model output and loss right? Can you try use find_unused_parameters=True when initializing DDP?
Currently, find_unused_parameters=True must be passed into torch.nn.parallel.DistributedDataParallel() initialization if there are parameters that may be unused in the forward pass.


Question:
I have been trying to minimize my memory footprint while training with DDP and came across ZeroRedundancyOptimizer. I implemented it similarly to that from the example, by simply wrapping an AdamW class into it. This is the (rough) example of the training loop I am using that is called from torch.multiprocessing.spawn.
device = f'cuda:{gpu}'
torch.cuda.set_device(gpu)  # From 0 to N-1 for the number of GPUs

net = net.to(device)
net = torch.nn.parallel.DistributedDataParallel(net, device_ids=[rank],
                                                gradient_as_bucket_view=True,
                                                find_unused_parameters=True)
scaler = GradScaler()
optimizer = ZeroRedundancyOptimizer(net.parameters(),
                                    optim=torch.optim.AdamW, lr=learning_rate,)

for sample, ground_truth in train_loader:
    optimizer.zero_grad(set_to_none=True)

    with autocast():
        pred = net(sample)
        loss = loss_fn(pred, ground_truth)

    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

However, I am getting the error Optimizer state has not been consolidated. Returning the local state Please call "consolidate_state_dict()" beforehand if you meant to save the global state.
Am I doing something wrong with this implementation?
I am running torch == 1.8.1 and cudatoolkit==10.2

Context: cc @mrshenli @rvarm1 if you have any insights on what might be wrong there?


Distributed Data Parallel doesn’t remove hooks

Question:
The Docs on the DDP class say:
Forward and backward hooks defined on module and its submodules
won’t be invoked anymore, unless the hooks are initialized in the forward() method.

However, they are working for me. Did I misinterpret the warning? Does it mean something else?
Is there a reason I should be careful when using hooks with DDP? In my original code I use it to modify the output of some layers. But here’s a simplified minimum reproducible example:
import torch.nn as nn
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import os

class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        self.model = nn.Sequential(nn.Conv2d(1, 1, 1), nn.Conv2d(1, 1, 1))

    def forward(self, x):
        self.model(x)

def hook_fn(mod, inp, out):
    if dist.is_initialized():
        print(
            f"The forward hook is not removed in rank : {dist.get_rank()}")

def run_model(rank):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group("nccl", rank=rank, world_size=2)
    model = SimpleCNN().to(rank)
    for mod in model.modules():
        if 'Conv' in mod.__class__.__name__:
            mod.register_forward_hook(hook_fn)

    ddp_mod = nn.parallel.DistributedDataParallel(model, device_ids=[rank])
    for _ in range(2):
        ddp_mod(torch.randn(1, 1, 10, 10))
    dist.destroy_process_group()

def run():
    mp.spawn(run_model, nprocs=2, join=True)

if __name__ == '__main__':
    run()

And the corresponding output:
The forward hook is not removed in rank : 0
The forward hook is not removed in rank : 1
The forward hook is not removed in rank : 0
The forward hook is not removed in rank : 1
The forward hook is not removed in rank : 0
The forward hook is not removed in rank : 0
The forward hook is not removed in rank : 1
The forward hook is not removed in rank : 1

Context: @ptrblck could you help please?


DDP Error: torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers

Question:
Hi,
Since last week I upgraded pytorch to 1.10.0, this error occurred. Here’s the log:

Blockquote
WARNING:torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15342 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15343 closing signal SIGHUP
Traceback (most recent call last):
File “/home/anaconda3/envs/aa/lib/python3.8/runpy.py”, line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File “/home/anaconda3/envs/aa/lib/python3.8/runpy.py”, line 87, in _run_code
exec(code, run_globals)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launch.py”, line 193, in
main()
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launch.py”, line 189, in main
launch(args)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launch.py”, line 174, in launch
run(args)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/run.py”, line 710, in run
elastic_launch(
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 252, in launch_agent
result = agent.run()
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py”, line 125, in wrapper
result = f(*args, **kwargs)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py”, line 709, in run
result = self._invoke_run(role)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py”, line 843, in _invoke_run
time.sleep(monitor_interval)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py”, line 60, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 15265 got signal: 1

Context: Hey @Kiuk_Chung, is there any BC-breaking change in 1.10?


DDP on 8 gpu work much worse then on single

Question:
So i read this thread and some other, googled and etc

    Training performance degrades with DistributedDataParallel distributed

    I’m training a conv model using DataParallel (DP) and DistributedDataParallel (DDP) modes. For DDP, I only use it on a single node and each process is one GPU.
My model has many BatchNorm2d layers. Given all other things the same, I observe that DP trains better than DDP (in classification accuracy). Even if I add SyncBN from pytorch 1.1, I still observe that DP > DDP+SyncBN > DDP without SyncBN in test accuracy.
I’m aware of the difference between DP and DDP’s handling of averaging/sum: Is av…

My model is seq to seq variable len input (tacotron 2).
This is how looks like loss

           6

Any suggestions? Any way how to debug it?
Also if i remove all batchnorm from model, it raise unused variable error, how it work?

Context:
Are you using any reference implementation as your code base or did you write the complete code yourself?
Could you compare your implementation to this one 39 written by NVIDIA?


Question:
Hi,
I’ve seen some discussions about DDP vs DP here but mainly focused around the learning rate. In my case both are taking the mean of the gradients from the GPU but I am seeing consistently somewhat worse performance in terms of loss and additional metrics from DDP than with DP. I am using same # of GPUs, same BS, same CrossEntropy Loss and other hyperparameters are kept the same as well. The epoch is set during every new epoch of DDP as well.
This is on a GPT-2 line language model so batch-norm cannot be the answer. In my understanding if the gradients are both averaged across GPUs and there is no batch-norm the two methods should be consistent except DDP is just much faster, however, obviously my understanding here is wrong I just haven’t found any explanation as to why this wouldn’t be the case.
Additionally, the seed is set for model initialization and many runs have been done as well.
Thanks!

Context:
johncookds:

In my understanding if the gradients are both averaged across GPUs and there is no batch-norm the two methods should be consistent except DDP is just much faster

I’m assuming you mean “DDP is just much slower” here? Do you have some sample code illustrating this performance gap? If so, it is much easier for us to troubleshoot why DDP might be much slower than DP in certain use cases.


Training performance degrades with DistributedDataParallel

Question:
I’m training a conv model using DataParallel (DP) and DistributedDataParallel (DDP) modes. For DDP, I only use it on a single node and each process is one GPU.
My model has many BatchNorm2d layers. Given all other things the same, I observe that DP trains better than DDP (in classification accuracy). Even if I add SyncBN from pytorch 1.1, I still observe that DP > DDP+SyncBN > DDP without SyncBN in test accuracy.
I’m aware of the difference between DP and DDP’s handling of averaging/sum: Is average the correct way for the gradient in DistributedDataParallel with multi nodes? 87
The LR and total batch size are the same for both DP, DDP+SyncBN, and DDP.
If I understand correctly, DP doesn’t do SyncBN, so DP should in theory achieve the same test accuracy as DDP (given small batch size per GPU)? If we assume larger effective batch size leads to better result, I should expect the following test performance ranking:
DDP+SyncBN > DP == DDP
but in practice, I observe: DP > DDP+SyncBN > DDP
Because DDP+SyncBN is 30% faster than DP, I really hope to solve the training gap so that I can take advantage of DDP’s superior speed. Thanks for any help!

Context:
Hi Jim,
From docs 39
DistributedDataParallel can be used in the following two ways:
(1) Single-Process Multi-GPU
(2) Multi-Process Single-GPU
Second method the highly recommended way to use DistributedDataParallel, with
multiple processes, each of which operates on a single GPU. This is
currently the fastest approach to do data parallel training using PyTorch
and applies to both single-node(multi-GPU) and multi-node data
parallel training.
Which one are you using ?


Concurrent CPU execution when Out-of-memory

Question:
Hi PyTorch community, I hope this is the right section to post my issue in.
I have a quite big dataset of biological (protein) sequences, around 68M, that I have to process with Rostlab/prot_t5_xl_uniref50 at main 2, a pre-trained NLP model that generates embeddings for downstream tasks. I have set up a decent workflow that is able to recover when my single V100 GPU goes OOM when the sequence is too big (even with batch size set to 1), effectively resorting to CPU and RAM for the calculations.
As of now, I used the suggested PyTorch implementation in the for loop:
oom = False
try:
    embedding = model(input_ids=input_ids, attention_mask=attention_mask)
except RuntimeError:
    oom = True
if oom:
    model.cpu()
    embedding = model(input_ids=input_ids.cpu(), attention_mask=attention_mask.cpu())

This approach solves the OOM issues, but I realized that has the side effect of having the GPU sitting around doing nothing while the big sequence is being processed, only to resume working with the next iterations.
Is there any possibility to send the problematic sequence to CPU while continuing to process the rest normally?
Things I’ve tried:

DataParallel is not the right answer:ValueError: Expected a non cpu device, but got: cpu

Python multiprocessing expects a pickleable Dataset: AttributeError: Can't pickle local object 'main.<locals>.SequenceDataset'

Context: If this is an inference task without the need to do things like broadcasting gradients, could you “outsource” the CPU inference to a subprocess subprocess — Subprocess management — Python 3.10.2 documentation 3 e.g., via POpen as a workaround?


Intel E810 RoCE NCCL unhandled system error

Question:
I was trying to start a distributed training across 2 machines using RoCE.
Environment:

Ubuntu 20.04
pytorch torch==1.10.1+cu113

The task is established by slurm:
#!/bin/bash
#SBATCH --gres=gpu:2
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=4

srun python train.py

And the error I met is:
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3
ncclSystemError: System call (socket, malloc, munmap, etc) failed.```

Line 957 in pytorch/ProcessGroupNCCL.cpp at v1.10.1 · pytorch/pytorch · GitHub 2 is:
C10D_NCCL_CHECK(ncclGroupEnd(), c10::nullopt);

Training works if I set ‘NCCL_IB_DISABLE’ = ‘1’. Which might indicate this might be a RoCE related issue.
I was wonder if I’m looking the correct place. And could anyone help me with this issue?

Context: Can you re-run the script with the environment variable NCCL_DEBUG=INFO to see if NCCL debug log reports any errors that could be helpful here?


Question:
Hi all!
I was wondering if there is a way to destroy a TCPStore, and close sockets associated with it.
I want to destroy the current process group (created with a TCPStore instance), and initialize a new one at the same port, but, as expected, I get an “Address already in use” error.
Thank you!

Context:
Thanks for posting the question @fot, Internally when constructing the TPCStore, we have a TCPStoreWorkerDaemon to bind the ports and do create creation, so when it goes out of scope and no reference to the TCPStore created, we will call the destructor automatically and release the port resources.
Could you elaborate more on how you get the error? If you did Ctrl-C on one process, there might be hanging processes still have TCPStore as it does not have master to send the destruction signals. In this case you might need to manually kill those hanging processes. You can do it by inspecting what ports your training_script binded to by using $(ps aux | grep training_script.py | grep -v grep | awk '{print $2}') , then kill them


Question:
Hi,
i got an error might relate to dataloader and shared memory, the error below would pop up randomly.
RuntimeError: falseINTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/MapAllocator.cpp:263, please report a bug to PyTorch. unable to open shared memory object </torch_43891_16> in read-write mode
I am using DDP torch.distributed.run to train a model on imagenet dataset with a multi-node multi-gpu setup. i’ve tried many possible solution such as setting ulimit -n <max value>, or using the command torch.multiprocessing.set_sharing_strategy('file_system'), but neither doesn’t work.
Each node has AMD CPU with 64 processors + 2 A100 GPUs + 250G RAM
I previously set num_workers=24 in data loader since the node has 64 processors in total but still failed
In the latest attemption i set num_workers=12 and ulimit -n <max value>, it works for now but the code is still running and i cannot make sure the error won’t pop up later.
May i ask would the num_workers related to this error? and also what would be a proper way to choose the num_workers if it is the case.
Thanks!

Context:
Thanks for your advice. I’ve looked the link you refer, unfortunately it seems not related to my issue.
Current i assume it is related to the PyTorch version. i previously worked on PyTorch 1.10+cuda11.3 on miniconda. Now I switch back to PyTorch 1.9+cuda11, it seems no such error. It is a work around but still can debug why it happens before.


Got nan loss when using dist.send and dist.recv

Question:
I have been trying use tcp connection and mp.spawn for using two GPUs for model parallel. But after execute some batches(about 4) in CIFAR10, the output of two sub-models becomes nan. Why this has happened? When I do the training in one gpu, it is fine.
code for send and recv
class RemoteSend(autograd.Function):
    @staticmethod
    def forward(ctx, input: torch.tensor, to_rank: int, from_rank: int,self_rank:int ):
        dist.send((torch.tensor(input.dim())*torch.tensor(1.0)).to(self_rank), to_rank)
        dist.send((torch.tensor(input.size())*torch.tensor(1.0)).to(self_rank), to_rank)
        print("forward send size",torch.tensor(input.size())*1.0)
        dist.send(input, to_rank)
        to_rank = torch.tensor(to_rank)
        from_rank = torch.tensor(from_rank)
        ctx.save_for_backward(to_rank, from_rank)
        return input

    @staticmethod
    def backward(ctx, grad_output):
        to_rank, from_rank = ctx.saved_tensors

        dist.recv(grad_output, int(from_rank))
        return grad_output

class RemoteReceive(autograd.Function):
    @staticmethod
    def forward(ctx, to_rank: int = 0, from_rank: int = 1,self_rank: int = 0):
        ##print(to_rank, from_rank)
        dim = torch.tensor(1.0).to(self_rank)
        dist.recv(dim, from_rank)
        #print("forward recv dim", dim)
        size = torch.rand(int(dim)).to(self_rank)
        dist.recv(size, from_rank)
        #print("forward recv size",size)
        x = torch.zeros(tuple(size.int())).to(self_rank)
        #print("try to get input with size", x.size())
        dist.recv(x, from_rank)
        #print("forward recv finish")
        to_rank = torch.tensor(to_rank)
        from_rank = torch.tensor(from_rank)
        ctx.save_for_backward(to_rank, from_rank)
        return x

    @staticmethod
    def backward(ctx, grad_output, to_rank: int = 0):
        to_rank, from_rank = ctx.saved_tensors
        dist.send(torch.tensor(grad_output.dim()*1.0), int(to_rank))
        dist.send(torch.tensor(grad_output.size())*1.0, int(to_rank))
        dist.send(grad_output, int(to_rank))

        return None

These are code that when you apply them it send tensors to other gpus in forward and get tensors in backward.
And training code are as below.
def main_worker(gpu,world_size):
    dist.init_process_group(backend='nccl', init_method='tcp://127.0.0.1:1224',
                            world_size=2, rank=gpu)
    transform = transforms.Compose(

        [transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
    trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                            download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=16,
                                            shuffle=True, num_workers=12)

    testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                        download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=16,
                                            shuffle=False, num_workers=12)
    model =torchvision.models.MobileNetV2(num_classes = 10)
    if gpu == 0:

        model1 = model.features[0:5]
        criterion = nn.CrossEntropyLoss().cuda(0)
        optimizer1 =torch.optim.SGD(model1.parameters(),lr = 0.1,weight_decay= 1e-4)
        model1.cuda(0)
        for i ,(images,target) in enumerate(trainloader):
            images = images.cuda(0,non_blocking =True)
            output =model1(images)
            #print(output)
            RemoteSend.apply(output,1,1,0)
            optimizer1.zero_grad()
            output.backward(output)
            optimizer1.step()
    if gpu ==1 :
        model2 = nn.Sequential(model.features[5:],Reshape(-1,1280),model.classifier)
        criterion = nn.CrossEntropyLoss().cuda(1)
        optimizer2 =torch.optim.SGD(model2.parameters(),lr = 0.1,weight_decay= 1e-4)
        model2.cuda(1)
        for i ,(images,target) in enumerate(trainloader):
            target = target.cuda(1,non_blocking =True)
            input = torch.tensor([16,32,4,4])
            input = RemoteReceive.apply(0,0,1)
            input = input.cuda(1)
            #print(input)
            output = model2(input)
            #print(output)
            loss = criterion(output, target)
            optimizer2.zero_grad()
            loss.backward()
            optimizer2.step()
            print(loss)

I divided MobileNetV2 into two parts.
The first 4 batches the model get normal resputs, after that, loss and output of model becomes nan.
Here are settings
Dataset:CIFAR10
lr:0.1
batchsize: 512
imagesize:32*32
I don’t believe these settings are related to nan, since I have tried on one GPU

Context:
Thanks for posting @Maxwell_Albert Looking into your problem, I couldn’t find anything suspicious yet. But I think potentially this might potentially be related to how the backward runs across two GPUs.
Since you want to do model parallel across two GPUs and split two models. Maybe you want to consider using RPC to do model parallel? RPC and Distributed Autograd could hook up the autograd graph implicitly without user writing custom send/recv functions, this might can resolve the nan loss issue that you met. Getting Started with Distributed RPC Framework — PyTorch Tutorials 1.10.1+cu102 documentation


Question:
I’m trying to train a model on 8 GPUs at once (on the same machine), using DistributedDataParallel. My issue is that this ends up using a lot of memory and I have to “cripple” my dataloader a bit in order even make it fit on the machine’s 64 GB of RAM.
The reason seems to be a combination of two things. First, my dataloader upon being initialized loads the entire dataset (about 3GB total) into RAM, so that loading batches later will be very fast. Second, I can see that a separate python process is created for each GPU that I train on (even with num_workers=0 in the dataloader), and each of these processes uses a substantial amount of RAM, presumably because each process loads its own copy of the entire dataset into RAM. This scales linearly with the number of GPUs I use.
I’m not sure the above fully explains the problem as the data is only 3GB, and yet each process uses about 6GB of RAM - unless perhaps the data is stored in RAM in a more bulky (e.g. lower-precision) format than on disk? Or perhaps the remaining RAM is because of other things being stored in RAM, like model weights or gradients (but then I thought those should live in GPU memory)?
I guess my main question is: is this normal / inevitable behavior? If the problem is indeed that 8 copies of the dataset end up getting stored in RAM, is there some way to avoid this and load it into RAM only once? Or is this issue caused by something else?

Context: Maybe this issues can answer you.


DDP’s trained loss is obviously not as good as DP’s

Question:
Hello everyone, I have a question that has been puzzling me for a long time.
The model I trained using DDP mode is much worse than the model trained by DP mode.
The specific settings are as follows: I keep the total batch_size the same. Each model runs 4 GPUs separately. training both use Adam optimizer. The learning rate is 1e-3.
I don’t know what’s wrong. Can anyone help me?

Context:
@ioyy on the onset, the difference between Data Parallel and Distributed Data Parallel is that one uses single process and the other multiple processes. Both implement the gradient synchronization process to assimilate gradients trained in different threads/machines.
Please share your code so that we can look further into it. Please ensure similar seeds in all machines in DDD


Question:
Hello,
We try to execute the distributed training on 32 nodes and each node can access 4 gpus. However, the code shows the RuntimeError: Socket Timeout for a specific epoch as follows:
Accuracy of the network on the 50000 test images: 0.6%
Max accuracy: 0.56%
Epoch: [3]  [ 0/78]  eta: 0:05:17  lr: 0.006401  loss: 6.8356 (6.8356)  time: 4.0712  data: 1.9933  max mem: 12282
Epoch: [3]  [10/78]  eta: 0:00:59  lr: 0.006401  loss: 6.8580 (6.8624)  time: 0.8734  data: 0.1814  max mem: 12282
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4286 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4287 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4289 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 2 (pid: 4288) of binary: /global/homes/z/zw241/.conda/envs/pt-1.9/bin/python
ERROR:torch.distributed.elastic.agent.server.api:Error waiting on exit barrier. Elapsed: 313.7377507686615 seconds
Traceback (most recent call last):
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/elastic/agent/server/api.py", line 904, in _exit_barrier
    barrier_timeout=self._exit_barrier_timeout,
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/elastic/utils/store.py", line 67, in barrier
    synchronize(store, data, rank, world_size, key_prefix, barrier_timeout)
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/elastic/utils/store.py", line 53, in synchronize
    agent_data = get_all(store, key_prefix, world_size)
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/elastic/utils/store.py", line 31, in get_all
    data = store.get(f"{prefix}{idx}")
RuntimeError: Socket Timeout
Traceback (most recent call last):
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/run.py", line 713, in run
    )(*cmd_args)
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/global/homes/z/zw241/.conda/envs/pt-1.9/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
    failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

We are not sure whether it is the issue of the platform or the issue of the pytorch library. We tried the 16 nodes and 8 nodes for the same training process, which looks good. Thanks for your help!

Context:
Thanks for posting @Zhe_Zhe. Can you explain more on your set up environment so that we can help more? Is this only happening in NCCL, what nccl version are you at?
Also, for good measure, you can try setting NCCL_DEBUG=INFO (also see Distributed communication package - torch.distributed — PyTorch 1.10.1 documentation) and check if you see anything out of the ordinary (perhaps some warnings or otherwise messages that indicate a problem with the topology).
There’s another github issue that have similar problem, you can check there as well to see if it helps Model parallel with DDP get `Socket Timeout` error when using NCCL, while GLOO works fine · Issue #25767 · pytorch/pytorch · GitHub 2


“Expected to have finished reduction” error when dropping layers with DDP

Question:
I want to do layerdrop. Unfortunately when I do so I get an error:
Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. [...]
I thought this could be caused by the fact that I was doing DDP and different workers were dropping different layers leading to problems when trying to sync gradients. However I set the seed so the different workers should be dropping the same layer at the same time and I still get the error.
Why is this happening?

Context:
Thanks for posting @divinho. From your description it seems that your model is dropping a particular layer and the loss calculation does not use that layer at all. If you want to train in such way with DDP, can you try use find_unused_parameters=True when initializing DDP?
Currently, find_unused_parameters=True must be passed into torch.nn.parallel.DistributedDataParallel() initialization if there are parameters that may be unused in the forward pass.


Confusing RuntimeError: CUDA error: device-side assert triggered

Question:
I’m trying to utilize my training/testing of my model using 4 GPUs by testing with a very simple CNN:
class SimpleCNN(nn.Module):
	def __init__(self, byte_size):
		super(SimpleCNN, self).__init__()
		self.l1 = nn.Sequential(nn.Linear(byte_size, int(byte_size/2)), nn.ReLU())
		self.l2 = nn.Sequential(nn.Linear(int(byte_size/2), int(byte_size/4)), nn.ReLU())
		self.l3 = nn.Sequential(nn.Linear(int(byte_size/4), 6))
	def forward(self, x):
		out = self.l1(x)
		out = self.l2(out)
		out = self.l3(out)
		return out

And utilizing this tutorial on moving my model to multiple gpus tutorial 1:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = SimpleCNN(BYTE_BLOCK_SIZE)
if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    model = nn.DataParallel(model)
model.to(device)

loss_function = nn.NLLLoss() # This is a convex loss function
optimizer = torch.optim.Adam(model.parameters(), lr=.03)

I get the following error (full trace):
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-86-3548497e2569> in <module>
      6     print("Let's use", torch.cuda.device_count(), "GPUs!")
      7     model = nn.DataParallel(model)
----> 8 model.to(device)
      9
     10 loss_function = nn.NLLLoss() # This is a convex loss function

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in to(self, *args, **kwargs)
    850             return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
    851
--> 852         return self._apply(convert)
    853
    854     def register_backward_hook(

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)
    528     def _apply(self, fn):
    529         for module in self.children():
--> 530             module._apply(fn)
    531
    532         def compute_should_use_set_data(tensor, tensor_applied):

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)
    528     def _apply(self, fn):
    529         for module in self.children():
--> 530             module._apply(fn)
    531
    532         def compute_should_use_set_data(tensor, tensor_applied):

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)
    528     def _apply(self, fn):
    529         for module in self.children():
--> 530             module._apply(fn)
    531
    532         def compute_should_use_set_data(tensor, tensor_applied):

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)
    550                 # `with torch.no_grad():`
    551                 with torch.no_grad():
--> 552                     param_applied = fn(param)
    553                 should_use_set_data = compute_should_use_set_data(param, param_applied)
    554                 if should_use_set_data:

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in convert(t)
    848                 return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
    849                             non_blocking, memory_format=convert_to_format)
--> 850             return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
    851
    852         return self._apply(convert)

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Note that when using the model only on the CPU the code works fine and I’m able to easily train the model with no warnings/errors.
Then I checked this post 3 because they had a similar error and when I did
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
torch.rand(1).cuda()

I still get the error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-91-e97f59385399> in <module>
      1 os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
----> 2 torch.rand(1).cuda()

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

So I couldn’t get a proper stacktrace even when CUDA_LAUNCH_BLOCKING=1 .
This was all done in Jupyter Notebook by the way. But what is even more weird is if I run the exact code above I do NOT get any error:

weirderror764×368 54.6 KB

Is this a problem with my CUDA version? On my system I have CUDA 11.2 but I installed cuda toolkit=10.2 with Pytorch and I thought that Pytorch would use the cuda toolkit installed with Pytorch within Conda?

Context:
Matthew_Chen37:

This was all done in Jupyter Notebook by the way.

Setting CUDA_LAUNCH_BLOCKING=1 after the CUDA context was already created won’t have any effect, which seems to be the case here. Thus I do not recommend to set this env variable inside the script and be especially careful when running in Jupyter notebooks (restart the Kernel after each failure to avoid confusing sticky failures).
Use CUDA_LAUNCH_BLOCKING=1 python script.py args (or export the env variable) and rerun your code.


Finding the cause of RuntimeError: Expected to mark a variable ready only once

Question:
I’m extending a complex model (already with DistributedDataParallel with find_unused_parameters set to True) in PyTorch on detectron2.
I’ve added a new layer generating some additional output to the original network - initially, that layer was frozen (requires_grad = False) and everything was working fine. I later decided to unfreeze this layer. Unfortunately, this results in this error on multiple GPUs:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
...
  File "/xxx/trainer.py", line 585, in some_method:
    losses.backward()
  File "/xxx/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/xxx/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag

RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons:
1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes
2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases yet.

Clearly, the error is related to that unfrozen layer, since everything was working before unfreezing it. This means that I need to check every change I made.
I wonder if there’s a way/trick to determine which particular tensor/operation causes this behaviour? In other words - how can I speed up the debugging process?
Asked the same question on SO here 18.

Context:
Thanks - it helped me to move forward. Turns out this feature is available in PyTorch 1.9.0 (was using 1.7.0), so no need to use nightly builds.
Now I have this error:
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
Parameter at index 73 has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration. You can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print parameter names for further debugging.

With setting TORCH_DISTRIBUTED_DEBUG to DETAIL I also have :
Parameter at index 73 with name roi_heads.box_predictor.xxx.bias has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration.

So, technically, there is a problem with roi_heads.box_predictor.xxx. Could not find one. However, with version 1.9.0 the console also outputs this:
[W reducer.cpp:1158] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())

Turned out to be the root cause. Switching find_unused_parameters to False makes the training run normally. Not sure why this does work now, but I don’t mind.


Pytorch elastic and local set of machines

Question:
Hi all!
I have set a machines in my local network with GPU (some have one GPU, some few, and there is 3 types of GPU models)
I would like to turn this machines for somewhat cluster to use multi-node training for my team.
And now I am considering different variants how to do it.
One of this variant is torch.distributed.elastic. But I do not understand how it works.
I found a couple examples but all of them for some cloud provider with kubernetes which is not my case.
So some question (let’s assume I do not want to install kubernetes cluster yet)

What is c10d? I really unable to google it.
Quick start examples just pass train.py to torchrun but what about virtualenv or conda env. Is it possible to run docker image like this?
Is it not clear for me what is ElasticAgent and how to use it?

Context: cc @aivanou @Kiuk_Chung


How to avoid pickling errors when using PyTorch DDP with S3 data?

Question:
Hi,
I’m using Pytorch DDP to data parellize this code GitHub - aws-samples/sagemaker-a2d2-segmentation-pytorch
I’m launching the training with
def main():

    # launches 1 data parallel replica per GPU
    mp.spawn(
        fn=train_replica,
        nprocs=world_size,
        join=True)

if __name__ == "__main__":

    main()

My dataset class uses boto3 to move records from S3 to memory.
To avoid boto3 serialization issue, the dataset and dataloaders are instantiated from within the train_replica function, so that they should all be created on their own and not boto3 things should need to be serialized.
However, my training fails for

_pickle.PicklingError: Can't pickle <class 'boto3.resources.factory.s3.ServiceResource'>: attribute lookup s3.ServiceResource on boto3.resources.factory failed

How to avoid that? Given how popular S3 is, I guess it is possible to use DDP with a dataset getitem that uses S3 right? how to avoid the serialization problem?

Context:
It looks like some multiprocessing code is attempting to pickle some data to send across processes, but boto3 resource can’t be pickled.
Similar to this comment: PicklingError encountered when using multiple GPUs · Issue #67681 · pytorch/pytorch · GitHub, you might want to try:

Passing arguments and construct boto3 client in the subprocess
Use/cache/store results of using boto3 APIs in some format that can be pickled, and pass around the result.


How to use quantized tensors in torch.distributed?

Question:
I am trying to all gather a list of qint8 tensors on Pytorch 1.10.1 with the following code:
import torch
import torch.multiprocessing as mp
import torch.distributed as dist
import os

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '17777'
    # initialize the process group
    dist.init_process_group("nccl", rank=rank, world_size=world_size)

def subprocess_fn(rank):
    setup(rank, 4)
    test_tensor = torch.Tensor([rank]).cuda(rank)
    tensor_list=[torch.ones(1,dtype=torch.qint8).cuda(rank) for _ in range(4)]
    print(f'Before broadcast and quant data on rank {rank} is {test_tensor},type is {type(test_tensor)}.')
    dist.all_gather(tensor_list,torch.quantize_per_tensor(inputx=test_tensor,scale=test_tensor.max(),zero_point=0,dtype=torch.qint8))
    print(f'After broadcast data on rank {rank} is {tensor_list}, dtype is {tensor_list[0].dtype}.')
    tensor_list=[n.float() for n in tensor_list]
    print(f'After broadcast and expand data on rank {rank} is {tensor_list}, dtype is {tensor_list[0].dtype}.')

if __name__ == '__main__':
    mp.spawn(subprocess_fn, args=(), nprocs=4)

and I got the following error:
Traceback (most recent call last):
  File "/data/users/heyangqin/anaconda3/envs/deepspeed_fp8/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/vc_data/users/heyangqin/fp8/fp8_dist_test.py", line 18, in subprocess_fn
    tensor_list=[torch.ones(1,dtype=torch.qint8).cuda(rank) for _ in range(4)]
  File "/vc_data/users/heyangqin/fp8/fp8_dist_test.py", line 18, in <listcomp>
    tensor_list=[torch.ones(1,dtype=torch.qint8).cuda(rank) for _ in range(4)]
NotImplementedError: Could not run 'aten::empty.memory_format' with arguments from the 'QuantizedCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty.memory_format' is only available for these backends: [CPU, CUDA, Meta, MkldnnCPU, SparseCPU, SparseCUDA, BackendSelect, Python, Named, Conjugate, Negative, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, UNKNOWN_TENSOR_TYPE_ID, Autocast, Batched, VmapMode].

Is there any way to work around this or the torch.distributed cannot work with quantized tensors yet? Thank you!

Context:
It looks like this approach results in an error indicating that some ops used in all_gather are not implemented for quantized tensors, although we have built out prototype APIs for some quantized collective communication. See:

      github.com

    pytorch/pytorch/blob/master/test/distributed/algorithms/quantization/test_quantization.py

    # Owner(s): ["oncall: distributed"]

import torch
import os
import torch.cuda
import sys
import torch.distributed as dist
import torch.distributed.algorithms.quantization.quantization as quant
from torch.distributed.algorithms.quantization.quantization import DQuantType
from torch.testing._internal.common_distributed import (
    MultiProcessTestCase,
    init_multigpu_helper,
    requires_gloo,
    skip_if_rocm,
    skip_if_lt_x_gpu,
    requires_nccl,
)
from torch.testing._internal.common_utils import sandcastle_skip_if, run_tests, TEST_WITH_DEV_DBG_ASAN, NO_MULTIPROCESSING_SPAWN

torch.backends.cuda.matmul.allow_tf32 = False

  This file has been truncated. show original

      github.com

    pytorch/pytorch/blob/master/torch/distributed/algorithms/quantization/quantization.py

    import functools
import torch
import torch.distributed as dist

from enum import Enum

TORCH_HALF_MIN = torch.finfo(torch.float16).min
TORCH_HALF_MAX = torch.finfo(torch.float16).max

class DQuantType(Enum):
    """
    Different quantization methods for auto_quantize API are identified here.
    auto_quantize API currently supports fp16 and bfp16 methods.
    """
    FP16 = "fp16",
    BFP16 = "bfp16"

    def __str__(self) -> str:

  This file has been truncated. show original

Although please note that these APIs are prototype and may thus have some issues if you decide to give it a try. cc @wanchaol


Question:
Hi,
I’m trying this example code with PT1.8.1 Distributed Data Parallel — PyTorch 1.10.1 documentation 2
I’m getting a ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable MASTER_ADDR expected, but not set
What is wrong?

Context:
Thanks for bringing this up!
The issue is due to the docs, we need to set env variables MASTER_ADDR and MASTER_PORT to use the default env:// initialization correctly.
Fixing it in [Docs][BE] DDP doc fix by rohan-varma · Pull Request #71363 · pytorch/pytorch · GitHub.


Question: This category is dedicated for iOS and Android issues, new features and general discussion of PyTorch Mobile.

Context: We tried to deploy a object detection model in Android. In PyTorch model, ‘nn.functional.interpolate’ was used in forward path to upsample feature maps . Torch script converted the model without any issues. We are not able to load the model in Android app using ‘Module.load( )’ API. This API is throwing an exception.


Question: Hi, when I inference my qnnpacked models on Android, I see when tracing, that the max. CPU usage is only ~50%. I tried to increase the number of threads via set_num_threads(4), but that didn’t help. Is there any way to increase the inference speed by utilizing more CPU cores ?

Context: @kimishpatel Could you please take a look at this?


Error with Pytorch Mobile on Vulkan backend during prediction

Question:
Hello,
We are trying to use a model on Android on Vulkan backend.
However, we encounter the following error:
PlatformException (PlatformException(DemixingError, Could not run 'aten::reflection_pad1d.out' with arguments from the 'Vulkan' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::reflection_pad1d.out' is only available for these backends: [CPU, QuantizedCPU, BackendSelect, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, AutogradHPU, Functionalize].

CPU: registered at /home/marc/pytorch/build_android_arm64-v8a/aten/src/ATen/RegisterCPU.cpp:20948 [kernel]
QuantizedCPU: registered at /home/marc/pytorch/build_android_arm64-v8a/aten/src/ATen/RegisterQuantizedCPU.cpp:1223 [kernel]
BackendSelect: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
ADInplaceOrView: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:64 [backend fallback]
AutogradOther: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:35 [backend fallback]
AutogradCPU: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:39 [backend fallback]
AutogradCUDA: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:47 [backend fallback]
AutogradXLA: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:51 [backend fallback]
AutogradLazy: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:55 [backend fallback]
AutogradXPU: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:43 [backend fallback]
AutogradMLC: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:59 [backend fallback]
AutogradHPU: fallthrough registered at /home/marc/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:68 [backend fallback]
Functionalize: registered at /home/marc/pytorch/build_android_arm64-v8a/aten/src/ATen/RegisterFunctionalization_1.cpp:5316 [kernel]

  Debug info for handle(s): debug_handles:{-1}, was not found.

Exception raised from reportError at /home/marc/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:434 (most recent call first):
(no backtrace available), null, null))

We followed PyTorch Vulkan Backend User Workflow — PyTorch Tutorials 1.10.1+cu102 documentation 2.
Does this mean our model can’t be used with Vulkan at this time?
You can find our app/build.gradle file here:
def localProperties = new Properties()
def localPropertiesFile = rootProject.file('local.properties')
if (localPropertiesFile.exists()) {
    localPropertiesFile.withReader('UTF-8') { reader ->
        localProperties.load(reader)
    }
}

def flutterRoot = localProperties.getProperty('flutter.sdk')
if (flutterRoot == null) {
    throw new GradleException("Flutter SDK not found. Define location with flutter.sdk in the local.properties file.")
}

def flutterVersionCode = localProperties.getProperty('flutter.versionCode')
if (flutterVersionCode == null) {
    flutterVersionCode = '1'
}

def flutterVersionName = localProperties.getProperty('flutter.versionName')
if (flutterVersionName == null) {
    flutterVersionName = '1.0'
}

apply plugin: 'com.android.application'
apply from: "$flutterRoot/packages/flutter_tools/gradle/flutter.gradle"

android {
    compileSdkVersion flutter.compileSdkVersion

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }

    defaultConfig {
        applicationId "com.demixr.demixr_app"
        minSdkVersion 24
        targetSdkVersion flutter.targetSdkVersion
        versionCode flutterVersionCode.toInteger()
        versionName flutterVersionName
        externalNativeBuild {
            cmake {
                cppFlags ''
            }
        }
    }

    buildTypes {
        release {
            minifyEnabled false
            shrinkResources false
            signingConfig signingConfigs.debug
            applicationVariants.all { variant ->
                variant.outputs.all { output ->
                    project.ext { appName = 'demixr' }
                    outputFileName = "${appName}_${output.name}_${variant.versionName}.apk"
                }
            }
        }
    }
    externalNativeBuild {
        cmake {
            path file('src/main/cpp/CMakeLists.txt')
            version '3.10.2'
        }
    }
}

dependencies {
    implementation 'com.facebook.fbjni:fbjni-java-only:0.2.2'
    implementation 'com.facebook.soloader:nativeloader:0.10.3'

    implementation files('libs/pytorch_android-release.aar')
}

flutter {
    source '../..'
}

We are also not sure why we need to add implementation 'com.facebook.fbjni:fbjni-java-only:0.2.2' and implementation 'com.facebook.soloader:nativeloader:0.10.3' as we built Pytorch Android from source with the lite module.
Thanks in advance!

Context:
Hi Marc, Vulkan GPU backend is still in beta. Unfortunately, we don’t support ReflectionPad1d operator (aten::reflection_pad1d.out) yet. We’re still working on our operator coverage.
You can create a feature request for us to support the missing operator. Or you may want to implement it.
cc: @ssjia


Question:
Hello everyone,
I’ve been trying to load a model that accept 2 images as input source and passed as a list. I got this error:
Process: org.pytorch.demo.imagesegmentation, PID: 10778
    java.lang.ArrayIndexOutOfBoundsException: vector
        at org.pytorch.LiteNativePeer.forward(Native Method)
        at org.pytorch.Module.forward(Module.java:52)
        at org.pytorch.imagesegmentation.MainActivity.run(MainActivity.java:140)

Code to load model:
I tried this one;
Map<String, IValue> outTensors = mModule.forward(
                IValue.tupleFrom(
                        IValue.from(inputTensor),
                        IValue.from(inputTensor2)
                )
        ).toDictStringKey();

and this
IValue img = IValue.listFrom(inputTensor, inputTensor2);
final Tensor outputTensor = mModule.forward(img).toTensor();

Please, someone could help me?

Context: Separate “forward” and “toTensor/toDictStringKey” statements so you can better identify the issue. If still not resolved, please paste the complete inference code here.


How to use module.foward(Ivalue).toTensor()

Question:
My code:
final Tensor img1 = TensorImageUtils.bitmapToFloat32Tensor(bitmap1,
                    TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB);
            final Tensor img2 = TensorImageUtils.bitmapToFloat32Tensor(bitmap2,
                    TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB);

            IValue img = IValue.listFrom(img1, img2);
            final Tensor outputTensor = module.forward(img).toTensor();

error:

image878×251 39.3 KB

In fact, there are two pictures in my inputs. At first, I used torch.cat to splice on dim = 0 in pytorch. Later, I found that cat operation is not supported in Android. Instead, I changed the model to receive a list [tensor], read list [0] and list [1] respectively in the model, and then splice in the model. I noticed that the input type of forward can be an Ivalue, and the listfrom of Ivalue seems to support the conversion of list [tensor] to Ivalue, so I don’t understand what the problem is.My model input is two pictures . How should I implement this in Android, or how can I implement the torch.cat operation in Android？

Context:
from the error message your model expects a single Tensor as input, instead of a list of tensors. Try
module.forward(img1)


Input dims and input type for speed benchmarking on Android

Question:
Hi,
I have a custom PyTorch model, the inputs are of the model as like this:
model(torch.randint(0, 10,(1,1)), torch.rand(1, 51, 2048), torch.randint(0, 10,(1, 1))).
I am trying to speed benchmark this model with Android, but I am unable to understand how I should use the “input_dims” and “input_type” parameters as this model involves a mix of float and integer types.
Thanks!

Context:
An update:
I checked the speed_benchmark_torch.cc file. It specifies “If multiple input needed, use semicolon to separate the dimension of different tensors.”
Therefore, I tried: --input_dims=“1,1;1,51,2048;1,1” --input_type=“int64;float;int64”.
This results in error:
terminating with uncaught exception of type c10::Error: [enforce fail at speed_benchmark_torch.cc:111] input_dims_list.size() == input_type_list.size(). 1 vs 0. Input dims and type should have the same number of items.
(no backtrace available)
Aborted


Perform reshape of tensor in JAVA.pytorch

Question:
Hi, I have python code
and converting the code to android, and I have question of some commands of pythons.
First, my python code is unsqueeze(x.0) where x is pytorch.tensor.
e.g) when unsqueeze(x,0) is performed, shape of x will be [3,2,4] to [1,3,2,4] <-- add just 1 dimension

in JAVA, can I do it how…?? (java command is so unusual for me… please help)

Also, I want to reorder the pytorch.tensor in JAVA
e.g) [3,2,4] to [2,4,3]
in python it is performed by torch.from_numpy(x).permute(2, 0, 1)
In JAVA, how to can I do it? (JAVA command is fine to me… I found some java function code or package that support reshape in python where the input is array… in pytorch.java… what is the type of tensor? it is array?)

thank you.

Context: hi @SungmanHong Bro, did you found any workaround for the above task. I am also stuck on the above?


Question:
I have this sample code in file.m (compiled by Objective-C++ compiler) in Xcode iOS project (LibTorch is loaded via Cocoapods):
auto param = torch::tensor({1.}, torch::requires_grad());
auto result = 1 + param;
result.backward();

Executing gives this error:
libc++abi.dylib: terminating with uncaught exception of type c10::Error: Could not run ‘aten::_backward’ with arguments from the ‘CPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. ‘aten::_backward’ is only available for these backends: [BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].
I searched a lot and still have no answer, please give any suggestions how can to fix it.

Context: I think Autograd is disabled by default in the mobile backend. You might be able to build it from source and reenable it, but I haven’t tried it and am not sure if this would still be possible (it was in the past, if I’m not mistaken).


Is NNAPI conversion an all-or-nothing in terms of operators?

Question:
Hi everybody, first post.
I have a question regarding NNAPI conversion: do all operators need to be supported by NNAPI for the conversion to be successful? I am coming from the Tensorflow Lite world, where NNAPI unsupported operators get pushed back to CPU (with obvious performance implications, but at least the model can be run).
Since torch.backends._nnapi.prepare.convert_model_to_nnapi doesn’t take any additional arguments, it seems it’s all-or-nothing. E.g. I was trying to convert MobileNet V3 to NNAPI, but getting an error that quantized::hardswish is not supported.

Context: Yes @carrythetorch, it’s all the operators need to be supported at this point. They aren’t that hard to implement, we’re definitely open to PRs: pytorch/serializer.py at master · pytorch/pytorch · GitHub 2


Custom Yolov5s TorchScript model does not work on Android Object Detection Demo App

Question:
I have custom model trained on yolov5s v5 and I converted it to torchscript.ptl using ultralytics export.py with code modification as told here.
When I build on android i get the following error:

image892×265 30.7 KB

error that appears in debugging:
E/AndroidRuntime: FATAL EXCEPTION: Thread-2
Process: org.pytorch.demo.objectdetection, PID: 27891
java.lang.ArrayIndexOutOfBoundsException: length=1310400; index=1310449
at org.pytorch.demo.objectdetection.PrePostProcessor.outputsToNMSPredictions(PrePostProcessor.java:123)
at org.pytorch.demo.objectdetection.MainActivity.run(MainActivity.java:248)
I changed my torch version also, I am still facing the same issue.
yolov5s.torchscript.ptl model file provided in the repo. works fine. I’m facing this issue when I use my custom model.

Context: cc: @IvanKobzarev @Linbin


Question:
Getting an error saying the metalprepack to Conv2dprepack is not supported. Seems like it is in the code, so is it maybe the usage of it in the model below? I added the following check for unsupported things:
optimization_blocklist = [
MobileOptimizerType.CONV_BN_FUSION,
MobileOptimizerType.INSERT_FOLD_PREPACK_OPS,
MobileOptimizerType.REMOVE_DROPOUT,
MobileOptimizerType.FUSE_ADD_RELU,
MobileOptimizerType.HOIST_CONV_PACKED_PARAMS,
]
OcclusionAwareGenerator(
(dense_motion_network): DenseMotionNetwork(
(hourglass): Hourglass(
(encoder): Encoder(
(down_blocks): ModuleList(
(0): DownBlock2d(
(conv): Conv2d(44, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0)
)
(1): DownBlock2d(
(conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0)
)
(2): DownBlock2d(
(conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0)
)
(3): DownBlock2d(
(conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0)
)
(4): DownBlock2d(
(conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0)
)
)
)
(decoder): Decoder(
(up_blocks): ModuleList(
(0): UpBlock2d(
(conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): UpBlock2d(
(conv): Conv2d(2048, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(2): UpBlock2d(
(conv): Conv2d(1024, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(3): UpBlock2d(
(conv): Conv2d(512, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(4): UpBlock2d(
(conv): Conv2d(256, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
)
)
(mask): Conv2d(108, 11, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3))
(occlusion): Conv2d(108, 1, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3))
(down): AntiAliasInterpolation2d()
)
(first): SameBlock2d(
(conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3))
(norm): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(down_blocks): ModuleList(
(0): DownBlock2d(
(conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0)
)
(1): DownBlock2d(
(conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0)
)
)
(up_blocks): ModuleList(
(0): UpBlock2d(
(conv): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): UpBlock2d(
(conv): Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(bottleneck): Sequential(
(r0): ResBlock2d(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(norm2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(r1): ResBlock2d(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(norm2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(r2): ResBlock2d(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(norm2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(r3): ResBlock2d(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(norm2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(r4): ResBlock2d(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(norm2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(r5): ResBlock2d(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(norm1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(norm2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(final): Conv2d(64, 3, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3))
)
converting model to optimize for mobile gets the error:
RuntimeError                              Traceback (most recent call last)
 in ()
17 traced_detector = torch.jit.trace(generator, (image_tensor, kp_driving, kp_driving), strict=False)
18 #optimized_traced_detector = optimize_for_mobile(traced_detector, backend=‘metal’)
—> 19 optimized_traced_detector = optimize_for_mobile(traced_detector, optimization_blocklist, backend=‘metal’)
20 #print(torch.jit.export_opnames(optimized_traced_detector))
21 optimized_traced_detector._save_for_lite_interpreter("/content/generator_metal.pt")
/usr/local/lib/python3.7/dist-packages/torch/utils/mobile_optimizer.py in optimize_for_mobile(script_module, optimization_blocklist, preserved_methods, backend)
67         optimized_cpp_module = torch._C._jit_pass_vulkan_optimize_for_mobile(script_module._c, preserved_methods_str)
68     elif backend == ‘metal’:
—> 69         optimized_cpp_module = torch._C._jit_pass_metal_optimize_for_mobile(script_module._c, preserved_methods_str)
70     else:
71         raise TypeError(“Unknown backend, must be one of ‘CPU’, ‘Vulkan’ or ‘Metal’”)
RuntimeError: 0 INTERNAL ASSERT FAILED at “/pytorch/torch/csrc/jit/ir/alias_analysis.cpp”:532, please report a bug to PyTorch. We don’t have an op for metal_prepack::conv2d_prepack but it isn’t a special case.  Argument types: Tensor, Tensor, int[], int[], int[], int, None, None,
CPU Model Works perfectly, Model Performs great on GPU. Just not able to get it onto phone quite yet.

Context: It seems you are hitting this issue 4. Could you comment on this bug with your use case (and if possible a code snippet to reproduce it)?


Error running jit model on medal, any possible solutions?

Question:
Open to ideas on if this is an actual missing issue or something I can solve.
RuntimeError                              Traceback (most recent call last)
 in ()
1 kp_driving = kp_detector(image_tensor)
2 traced_generator = torch.jit.trace(generator, (image_tensor, kp_driving, kp_driving), strict=False)
----> 3 optimized_traced_generator = optimize_for_mobile(traced_generator, backend=‘metal’)
4 optimized_traced_generator._save_for_lite_interpreter("/content/generator_metal.pt")
/usr/local/lib/python3.7/dist-packages/torch/utils/mobile_optimizer.py in optimize_for_mobile(script_module, optimization_blocklist, preserved_methods, backend)
67         optimized_cpp_module = torch._C._jit_pass_vulkan_optimize_for_mobile(script_module._c, preserved_methods_str)
68     elif backend == ‘metal’:
—> 69         optimized_cpp_module = torch._C._jit_pass_metal_optimize_for_mobile(script_module._c, preserved_methods_str)
70     else:
71         raise TypeError(“Unknown backend, must be one of ‘CPU’, ‘Vulkan’ or ‘Metal’”)
RuntimeError: 0INTERNAL ASSERT FAILED at “…/torch/csrc/jit/ir/alias_analysis.cpp”:584, please report a bug to PyTorch. We don’t have an op for metal_prepack::conv2d_prepack but it isn’t a special case.  Argument types: Tensor, Tensor, int[], int[], int[], int, NoneType, NoneType,

Context: Based on the error message it sounds like an unexpected bug, so could you create a GitHub issue 2 so that we can track and fix it, please?


Question:
I went through the speed benchmark for Android page to test if our hardware could potentially be used with PyTorch models for accelerating something we’re working on.
Actual results are that the NNAPI models are much much slower:

./speed_benchmark_torch --pthreadpool_size=1 --model=mobilenetv2-quant_core-cpu.pt --use_bundled_input=0 --warmup=5 --iter=50
Starting benchmark.
Running warmup runs.
Main runs.
Main run finished. Microseconds per iter: 97858.2. Iters per second: 10.2189
./speed_benchmark_torch --pthreadpool_size=1 --model=mobilenetv2-quant_core-nnapi.pt  --use_bundled_input=0 --warmup=5
Starting benchmark.
Running warmup runs.
Main runs.
Main run finished. Microseconds per iter: 560976. Iters per second: 1.78261

Quick sanity check… did I run this test right to get expected faster results with the NNapi model?
I compiled PyTorch from sources and just used the latest origin/master (relatively recent).  My platform is Android 11 and the hardware is NXP i.mx8m which claims to support hardware acceleration for NNapi (It has a built in NPU).
NXP however only provides demo acceleration for TensorFlow.  If I run that demo, the results are a significant speedup for NPU acceleration.
So my goal is to find out which module is failing.  Can someone suggest a known working phone or android device that will show PyTorch successfully hardware accelerated?  If I can get desired results there, I can then move on to find out if our hardware is not behaving properly and get support from NXP.
Any other debugging advice appreciated.
Thanks for work on a cool project!

Context:
Hi @dennism Try turning on verbose adb logs and checking if the ops are running on the accelerator?
https://developer.android.com/ndk/guides/neuralnetworks#nnapi_logs 2
Sounds like some part of the graph is running on the CPU which goes to the NNAPI reference implementation which might be slower than regular Pytorch code


RuntimeError: Module ‘NnapiModule’ has no attribute ‘weights’ (This attribute exists on the Python module, but we failed to convert Python type: ‘list’ to a TorchScript type.)

Question:
hello.
I am trying to quantize my custom model and apply nnapi.
nnapi prototype “(Prototype) Convert MobileNetV2 to NNAPI — PyTorch Tutorials 1.7.1 documentation  1”
I followed the example provided here.
First, as in mobilenet v2, a quant pair was created, inherited as quantizable_my_model, and the pretrained weight was loaded, and the example was followed.
But nnapi_model = torch.backends._nnapi.prepare.convert_model_to_nnapi(MY_MODEL,MY_TENSOR)
The following error occurred in.
RuntimeError:
Module'NnapiModule' has no attribute'weights' (This attribute exists on the Python module, but we failed to convert Python type:'list' to a TorchScript type.):
  File "/home/ubuntu/anaconda3/envs/nnapi_2/lib/python3.6/site-packages/torch/backends/_nnapi/prepare.py", line 35
    def init(self):
        assert self.comp is None
        self.weights = [w.contiguous() for w in self.weights]
                                                ~~~~~~~~~~~~ <--- HERE
        comp = torch.classes._nnapi.Compilation()
        comp.init(self.ser_model, self.weights)

def make_custom_nnapi(output_dir_path):

    example_img = torch.rand(1, 3, 256, 256, device="cpu")
    example_val = torch.rand(1, 10, 2, device="cpu")
    example_jac = torch.rand(1, 10, 2, 2, device="cpu")

    MY_MODEL = quantizable_MY_MODEL() # This function just loads a model that inherits a pair of quantstub and dequantstub. and this function load pretrained weight

    MY_MODEL.fuse_model() # fuse_model() is equal with mobilenetv2

    MY_MODEL.qconfig = torch.quantization.get_default_qconfig('fbgemm')

    MY_MODEL_fp32_prepared = torch.quantization.prepare(MY_MODEL)
    MY_MODEL_fp32_prepared(example_img)
    MY_MODEL_int8 = torch.quantization.convert(MY_MODEL_fp32_prepared)
    example_img.nnapi_nhwc = True

    with torch.no_grad():
        MY_MODEL_traced = torch.jit.trace(MY_MODEL_int8, example_img)
    nnapi_model = torch.backends._nnapi.prepare.convert_model_to_nnapi(MY_MODEL_traced,example_img) #error occured this line

class QuantizableMYMODEL(MYMODEL2Q):
    def __init__(self, *args, **kwargs):
        """
        MobileNet V2 main class
        Args:
           Inherits args from floating point MobileNetV2
        """
        super(QuantizableMYMODEL, self).__init__(*args, **kwargs)
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        # x = self._forward_impl(x)
        x = self.dequant(x)
        return x

def quantizable_MY_MODEL(pretrained=False, progress=True, quantize=False,**kwargs):
    """
    Constructs a MobileNetV2 architecture from
    `"MobileNetV2: Inverted Residuals and Linear Bottlenecks"
    <https://arxiv.org/abs/1801.04381>`_.
    Note that quantize = True returns a quantized model with 8 bit
    weights. Quantized models only support inference and run on CPUs.
    GPU inference is not yet supported
    Args:
     pretrained (bool): If True, returns a model pre-trained on ImageNet.
     progress (bool): If True, displays a progress bar of the download to stderr
     quantize(bool): If True, returns a quantized model, else returns a float model
    """

    config_path=PATH TO CONFIG
    checkpoint_path=PATH TO CHEKCPOINT
    checkpoint = torch.load(checkpoint_path, map_location=torch.device("cpu"))

    with open(config_path) as f:
        config = yaml.load(f)

    model = Quantizable_MY_MODEL(
        **config["model_params"]["MYMODEL_params"],
        **config["model_params"]["common_params"]
    )
    model.load_state_dict(checkpoint["MYMODEL"])
    model.eval()
    # _replace_relu(model)

    return model

This is my source code, Hope this helps solve this problem. Thank you.
Don’t mind about the syntax of the uppercase letters
Please help me what to do.

Context: cc @David_Reiss @axitkhurana @Martin_Yuan thoughts?


Question:
Are there any best practices for using torchscript with pytorch-mobile? Im currently trying to implement a conv-net with pytorch-mobile on android. The model is very slow in comparison to an identical tensorflow trained tflite model. Perhaps this might have something to do with the torchscript tracing.
Any thoughts would be appreciated

Context: I tries freeze the graph using torch.utils.mobile_optimizer.optimize_for_mobile which is new for pytorch 1.6.0. It seems to improve performance a little but not significantly.


How to reduce processing time of TensorImageUtils.bitmapToFloat32Tensor function

Question:
After converting the bitmap generated by Android into a Tensor through the TensorImageUtils.bitmapToFloat32Tensor function, we have been receiving the result value through inference.
However, the processing time of that function is about 2 seconds, so it takes a lot of time to get the full result.
In bitmap, an image is drawn in real time, and each time a bitmap is obtained and inferred.
The result is delayed compared to the real-time video, so matching is not possible.

Context: @IvanKobzarev Wondering if you may have some thoughts on this. Thank you!