‘step_size_min’ referenced before assignment

Question:
I am using the Rprop optimizer and MSE for my loss function but am finding zero_grad isn’t setting the values for grad to zero for the network and the MSE.backward() doesn’t update the values either. I must be doing something very silly.
model = Net(feature_number=2, hidden_number=6, output_number=1)
search_method = torch.optim.Rprop(model.parameters(), lr=0.02)
## ommitted
def local_search(weights, network, loss_function, search_method , iterations, x_train, y_train, approach=LEARNING_METHOD):
    """ Perform local search for a given number of iterations """
    weights_into_network(network, weights)
    for i in range(iterations):
        output = network(x_train)
        loss = loss_function(output, y_train)
        search_method .zero_grad()
        loss.backward()
        search_method .step() #<--- here thrown
        learned_loss = loss.item()
## ommitted

  File "C:\Users\user\nn_ga\main.py", line 381, in local_search
    search_method.step()
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\optim\optimizer.py", line 88, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\autograd\grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\optim\rprop.py", line 109, in step
    step_size_min=step_size_min,
UnboundLocalError: local variable 'step_size_min' referenced before assignment

Versions of relevant libraries:
[pip3] numpy==1.21.1
[pip3] torch==1.10.1+cu102
[pip3] torchaudio==0.10.1+cu102
[pip3] torchvision==0.11.2+cu102

      github.com/pytorch/pytorch

      Rprop Optimizer: UnboundLocalError: local variable 'step_size_min' referenced before assignment 1

        opened

        Dec 23, 2021

          priceluke

          module: optimizer

          triaged

    ### 🐛 Describe the bug

When using the optimiser's step method, I get an error r…egarding step minimums. I believe this value should default to 1e-6 if not explicitly set via params.

```
model = Net(feature_number=2, hidden_number=6, output_number=1)
search_method = torch.optim.Rprop(model.parameters(), lr=0.02)
## ommitted
def local_search(weights, network, loss_function, search_method , iterations, x_train, y_train, approach=LEARNING_METHOD):
    """ Perform local search for a given number of iterations """
    weights_into_network(network, weights)
    for i in range(iterations):
        output = network(x_train)
        loss = loss_function(output, y_train)
        search_method .zero_grad()
        loss.backward()
        search_method .step() #<--- here thrown
        learned_loss = loss.item()
## ommitted
```

```
  File "C:\Users\user\nn_ga\main.py", line 381, in local_search
    search_method.step()
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\optim\optimizer.py", line 88, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\autograd\grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\optim\rprop.py", line 109, in step
    step_size_min=step_size_min,
UnboundLocalError: local variable 'step_size_min' referenced before assignment
```

### Versions

Versions of relevant libraries:
[pip3] numpy==1.21.1
[pip3] torch==1.10.1+cu102
[pip3] torchaudio==0.10.1+cu102
[pip3] torchvision==0.11.2+cu102

cc @vincentqb @jbschlosser @albanD

Context:
Found the issue…
layer.weight = torch.nn.Parameter(torch.from_numpy(temp))

should have been
layer.weight.data = torch.nn.Parameter(torch.from_numpy(temp))

Hope this helps someone in the future


Discussion about the paper of name CenterNet

Question:
Hello everyone!
Currently I’ve started reading the paper of name “CenterNet: Objects as Points”. The general idea is to train a keypoint estimator using heat-map and then extend those detected keypoint to other task such as object detection, human-pose estimation, etc. But the thing that confused me is how to splat the ground truth keypoint onto a heat-map by using Gaussian kernel. Annotation 2020-05-11 162433768×297 93.9 KB  What is object size-adaptive standard deviation? How to calculate the sigma for that formula.
Best regards!

Context:
From [30 - Hei Law, Jia Deng: CornerNet: Detecting Objects as Paired Keypoints]:
image816×773 252 KB
So the reference without comment how they do something different would seem to indicate that they do something very similar, i.e. I expect them to see how much you can move the bounding box (keeping the size the same, if they only consider the center, I guess) while keeping IoU >= 0.7. Without some thought, I wouldn’t know whether axial or some diagonal translation is crucial, but you could experiment with that. This would be the radius. Then you could take 1/3 like in the reference or so.
Whether that is exactly what they do, I don’t know. However, I would expect to get radii that look somewhat similar to the ones pictured.
Best regards
Thomas


Question:
I am trying to run end-to-end stereo code.
I am getting the following error.
python_anomaly_mode.cpp:104] Warning: Error detected in MaxPool2DWithIndicesBackward. Traceback of forward call that caused the error:
  File "train.py", line 85, in <module>
    trainer.fit(stereo_model, data_module)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in fit
    self._run(model)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in _run
    self._dispatch()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _dispatch
    self.accelerator.start_training(self)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 996, in run_stage
    return self._run_train()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1045, in _run_train
    self.fit_loop.run()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in advance
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 130, in advance
    batch_output = self.batch_loop.run(batch, self.iteration_count, self._dataloader_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 101, in run
    super().run(batch, batch_idx, dataloader_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 148, in advance
    result = self._run_optimization(batch_idx, split_batch, opt_idx, optimizer)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 202, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 396, in _optimizer_step
    model_ref.optimizer_step(
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1618, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 209, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 129, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 292, in optimizer_step
    make_optimizer_step = self.precision_plugin.pre_optimizer_step(
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/native_amp.py", line 59, in pre_optimizer_step
    result = lambda_closure()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 236, in _training_step_and_backward_closure
    result = self.training_step_and_backward(split_batch, batch_idx, opt_idx, optimizer, hiddens)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 537, in training_step_and_backward
    result = self._training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 307, in _training_step
    training_step_output = self.trainer.accelerator.training_step(step_kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 193, in training_step
    return self.training_type_plugin.training_step(*step_kwargs.values())
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 172, in training_step
    return self.model.training_step(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/model.py", line 63, in training_step
    output, all_outputs = self(batch)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/model.py", line 52, in forward
    output, all_outputs = self.model(left_image, right_image)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "models/hdrn_alpha_stereo.py", line 394, in forward
    disparity = self.disparity_refinement(left_image, disparity_small, matchability, cost_volume)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "models/hdrn_alpha_stereo.py", line 278, in forward
    base_downsample = self.base_downsample(disparity)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/pooling.py", line 162, in forward
    return F.max_pool2d(input, self.kernel_size, self.stride,
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/_jit_internal.py", line 405, in fn
    return if_false(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 718, in _max_pool2d
    return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
 (function _print_stack)
Traceback (most recent call last):
  File "train.py", line 85, in <module>
    trainer.fit(stereo_model, data_module)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in fit
    self._run(model)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in _run
    self._dispatch()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _dispatch
    self.accelerator.start_training(self)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 996, in run_stage
    return self._run_train()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1045, in _run_train
    self.fit_loop.run()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in advance
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 130, in advance
    batch_output = self.batch_loop.run(batch, self.iteration_count, self._dataloader_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 101, in run
    super().run(batch, batch_idx, dataloader_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 148, in advance
    result = self._run_optimization(batch_idx, split_batch, opt_idx, optimizer)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 202, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 396, in _optimizer_step
    model_ref.optimizer_step(
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1618, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 209, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 129, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 292, in optimizer_step
    make_optimizer_step = self.precision_plugin.pre_optimizer_step(
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/native_amp.py", line 59, in pre_optimizer_step
    result = lambda_closure()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 236, in _training_step_and_backward_closure
    result = self.training_step_and_backward(split_batch, batch_idx, opt_idx, optimizer, hiddens)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 549, in training_step_and_backward
    self.backward(result, optimizer, opt_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 590, in backward
    result.closure_loss = self.trainer.accelerator.backward(result.closure_loss, optimizer, *args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 276, in backward
    self.precision_plugin.backward(self.lightning_module, closure_loss, *args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 78, in backward
    model.backward(closure_loss, optimizer, *args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1481, in backward
    loss.backward(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8, 1, 120, 224]], which is output 0 of SumBackward1, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Context:
It’s a bit hard to narrow down the issue from the stacktrace, but you should check if

you are explicitly using inplace operations and replace them with their out-of-place equivalents
you are trying to use stale forward activations as described in this post 5


Reproducibility with all the bells and whistles

Question:
Had to dig up a lot of docs and discussions to finally figure out the right set of code to make a pytorch pipeline completely reproducible.
There are two things here, firstly you would want to use a seed that will be used to seed pytroch, numpy and python right at the start of main process. Be it training, be it inference.
You would want to use this function:
def seed_all(seed):
    if not seed:
        seed = 10

    print("[ Using Seed : ", seed, " ]")

    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    numpy.random.seed(seed)
    random.seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

Just call it right at the start and you are good to go.
Now comes the part where you use worker processes in DataLoader to speed up training. Each worker process is a separate realm and has no relation to the seed that you have used in your main process except for the fact that internally PyTorch seeds itself for you using “base_seed + worker_id” and base_seed is generated based on the seed you used in the main process. So we are still predictable until this part. Keep in mind that whether u define a worker_init_fn or not, PyTorch will get seeded.
The reason why you would want to implement the worker_init_fn is so that you can seed the other libraries that PyTorch does not seed by default. For example numpy and python random module. As you may know, Albumentations library uses python random module and thus making sure that each worker has a unique yet predictable seed is important for randomness. So you would want to use this (Use the worker_seed to seed everything else except PyTorch which has been seeded for u):
def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    numpy.random.seed(worker_seed)
    random.seed(worker_seed)

you would define this function in the global scope and use it like so:
DataLoader(
  train_dataset,
  batch_size=batch_size,
  num_workers=4,
  worker_init_fn=seed_worker,
  shuffle=True
)

I am training on a single GPU and this ensures AtoZ reproducibility. The only catch here is you will get different results if you change the number of workers for obvious reasons.
I hope this post reaches people on time so that they don’t waste precious GPU compute trying to figure out where on earth the randomness comes from.

Context:
Thanks for sharing this post!
Would you be interested on adding some of this information to the Randomness docs 120?


Selecting a particular layer from the pytorch pre-trained model

Question:
I have pre-trained model which has 227 layers. The last but one layer is the MLP. The pre-trained model is a state_dict which has key value pair. The values of all the keys are tensor objects. Now I have access the the MLP layer and extract the embeddings on the test images.How to do it?
‘mlp_head.0.weight’, tensor([0.9932, 0.9924, 1.0286, 0.9775, 1.0295, 1.0285, 1.0340, 0.9975, 1.0089,
1.0182, 1.0191, 1.0727, 1.0375, 1.0204, 1.0035, 1.0275, 1.0305, 1.0383,
0.9900, 1.0200, 1.0360, 1.0018, 1.0014, 0.9629, 0.9696, 1.0488, 0.9846,
1.0166, 1.0004, 1.0193, 1.0052, 1.0267, 1.0243, 1.0107, 0.9456, 1.0478,
0.9933, 1.0224, 1.0337, 0.8880, 1.0013, 0.9733, 1.0558, 0.9860, 1.0284,
1.0364, 1.0264, 1.0486, 1.0706, 1.0193, 0.9990, 1.0349, 1.0028, 0.9599,
1.0390, 1.0043, 1.0125, 1.0550, 1.0478, 1.0416, 0.9980, 0.9994, 0.9806,
0.9902, 1.0020, 1.0574, 0.9765, 1.0064, 0.9898, 0.9972, 1.0216, 1.0402,
0.9730, 1.0502, 1.0091, 0.9761, 1.0121, 1.0241, 1.0432, 0.9697, 1.0423,
1.0502, 1.0249, 1.0326, 1.0516, 1.0145, 1.0687, 0.9958, 1.0580, 0.9947,
0.8908, 0.9941, 1.0354, 1.0422, 1.0353, 1.0140, 1.0395, 1.0555, 1.0154,
1.0325, 1.0212, 0.9976, 1.0089, 0.9949, 1.0112, 0.8455, 1.0415, 1.0202,
1.0137, 0.9632, 1.0298, 1.0080, 1.0302, 0.9589, 1.0115, 1.0142, 1.0384,
1.0214, 1.0406, 1.0281, 1.0158, 1.0356, 1.0363, 0.9789, 1.0268, 0.9997,
0.9798, 1.0295, 1.0490, 1.0406, 1.0319, 1.0491, 0.9804, 1.0375, 1.0092,
1.0166, 0.9765, 1.0302, 1.0472, 1.0011, 0.9737, 1.0238, 0.9457, 1.0232,
1.0547, 1.0108, 1.0015, 1.0235, 1.0073, 0.9836, 1.0202, 1.0412, 0.9726,
1.0105, 1.0332, 1.0335, 1.0399, 0.9981, 0.9722, 0.9925, 1.0199, 1.0253,
0.9931, 1.0254, 1.0412, 1.0363, 0.9674, 1.0248, 0.9878, 1.0503, 1.0360,
0.9900, 1.0171, 1.0059, 1.0310, 1.0047, 1.0354, 1.0173, 1.0377, 1.0234,
1.0059, 1.0264, 1.0382, 0.9575, 1.0607, 1.0096, 1.0119, 0.9823, 0.9702,
1.0038, 1.0403, 0.9807, 1.0222, 0.9837, 0.9580, 1.0418, 1.0393, 1.0006,
1.0109, 0.9838, 1.0098, 1.0321, 1.0173, 1.0424, 0.9828, 1.0320, 1.0090,
0.7263, 1.0356, 1.0334, 1.0278, 1.0256, 1.0093, 1.0276, 1.0390, 1.0187,
1.0042, 1.0446, 1.0096, 1.0331, 1.0213, 1.0019, 1.0301, 1.0093, 1.0420,
1.0412, 0.9844, 0.9913, 1.0505, 1.0311, 0.9108, 1.0195, 1.0426, 1.0111,
1.0421, 1.0140, 1.0066, 1.0258, 0.9818, 1.0153, 0.9956, 1.0241, 1.0067,
1.0110, 1.0109, 1.0443, 1.0040, 1.0444, 1.0420, 1.0453, 1.0007, 1.0426,
1.0081, 0.9448, 1.0177, 1.0033, 1.0049, 0.9890, 1.0073, 1.0198, 1.0485,
1.0486, 1.0514, 1.0442, 1.0128, 1.0035, 0.9939, 1.0338, 1.0136, 1.0490,
1.0483, 1.0213, 0.9902, 1.0412, 0.9877, 0.9700, 1.0446, 1.0773, 1.0425,
0.9962, 0.9884, 0.9983, 1.0213, 0.9991, 0.9959, 1.0060, 1.0242, 1.0087,
0.9715, 1.0195, 1.0285, 1.0259, 1.0072, 0.9626, 1.0100, 1.0173, 0.9628,
1.0260, 0.9918, 1.0367, 0.9541, 0.9944, 1.0355, 1.0081, 0.9988, 0.9805,
1.0187, 1.0623, 0.9889, 1.0154, 1.0200, 1.0226, 1.0471, 1.0404, 1.0180,
1.0497, 1.0200, 1.0021, 1.0176, 1.0192, 0.9869, 1.0124, 1.0370, 1.0522,
0.9919, 1.0465, 0.9398, 0.9798, 1.0182, 1.0265, 1.0638, 1.0409, 1.0433,
1.0165, 1.0251, 1.0108, 1.0179, 1.0334, 1.0235, 0.9636, 0.9982, 0.9862,
1.0315, 1.0203, 1.0328, 1.0409, 1.0232, 1.0200, 1.0183, 1.0322, 1.0324,
0.9795, 0.9581, 1.0115, 0.9940, 1.0160, 1.0612, 0.0848, 0.9953, 0.9940,
0.9840, 1.0136, 1.0366, 1.0321, 1.0162, 1.0296, 1.0038, 0.9978, 1.0298,
0.9862, 1.0045, 1.0084, 1.0136, 0.9857, 0.9729, 1.0590, 1.0305, 1.0069,
1.0403, 0.9882, 1.0400, 1.0210, 1.0071, 1.0199, 1.0361, 1.0103, 1.0321,
0.9971, 1.0242, 1.0043, 1.0083, 1.0358, 1.0249, 1.0248, 1.0253, 1.0145,
1.0308, 1.0350, 1.0026, 1.0243, 1.0622, 1.0130, 1.0450, 1.0352, 1.0044,
1.0198, 0.9947, 1.0206, 1.0624, 1.0011, 1.0171, 1.0067, 0.9833, 1.0265,
1.0177, 1.0020, 1.0103, 0.9927, 1.0338, 1.0246, 1.0028, 1.0184, 1.0532,
1.0212, 1.0513, 0.9979, 1.0056, 0.9834, 1.0311, 1.0111, 0.9987, 0.9418,
1.0497, 0.9920, 1.0564, 1.0277, 1.0527, 1.0265, 1.0009, 1.0173, 1.0130,
1.0417, 1.0247, 0.9964, 1.0203, 0.9906, 1.0352, 0.9994, 1.0124, 1.0124,
1.0030, 1.0344, 0.9895, 1.0337, 0.9933, 1.0251, 1.0052, 1.0388, 1.0371,
1.0128, 0.9976, 0.9893, 1.0142, 1.0612, 1.0193, 1.0249, 0.9843, 1.0064,
1.0315, 1.0404, 1.0377, 1.0080, 1.0284, 1.0408, 1.0599, 1.0578, 0.9919,
1.0388, 1.0239, 1.0189, 1.0336, 0.9965, 0.9937, 1.0287, 1.0425, 0.9853,
1.0257, 1.0320, 1.0446, 1.0142, 1.0003, 1.0219, 1.0120, 1.0285, 0.9889,
1.0396, 1.0238, 0.9810, 1.0061, 0.9414, 1.0253, 1.0192, 1.0393, 1.0319,
1.0012, 0.9929, 1.0029, 1.0078, 1.0248, 1.0323, 1.0434, 0.9189],
device=‘cuda:0’))

Context:
Could you explain a bit more what “extract the embeddings on the test images” would mean?
If you would like to get the forward activations for some inputs (i.e. your test images) you could use forward hooks as described here 3.


Why weights/bias are initialized like this for RPN?

Question:
In the RPN() function why are the weights initialized like this? Am I missing something from the paper? and why this why not use kalming initilization and all.
I don’t even understand why bias is set as 0.
Can someone help?
for layer in self.children():
            torch.nn.init.normal_(layer.weight, std=0.01)  # type: ignore[arg-type]
            torch.nn.init.constant_(layer.bias, 0)  # type: ignore[arg-type]

The RPN function 1
Am I missing something obvious?

Context:
Another doubt inside RPN() here
self.bbox_pred = nn.Conv2d(in_channels, num_anchors * 4, kernel_size=1, stride=1)

Here why are we doing ‘num_anchors * 4’?
from where is this 4 coming?
whereas in detectron2, in the RPN it is defined as
self.anchor_deltas = nn.Conv2d(
            in_channels, num_cell_anchors * box_dim, kernel_size=1, stride=1
        )

here its “num_cell_anchors * box_dim”
(detectron2/rpn.py at 5e2a1ecccd228227c5a605c0a98d58e1b2db3640 · facebookresearch/detectron2 · GitHub)
I don’t know whats happening here.


Reproduce same network architecture (with dropout active) in multiple steps: zeroes multiple times the same elements with dropout

Question:
Hi everyone,
I have a network architecture with Dropout 1 layers.
From the above official reference:

During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution.

I need to fix the same configuration for multiples consecutive N steps; when I say fix the same configuration I mean:

at step 0, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution.
In the following N steps, zeroes the same input’s elements chosen in step 0 (so the network seen by the system for each of these steps is exactly the same)
At step N+1 we restart from the beginning (randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution.)
repeat

How can I do this?
[EDIT]
There are some example of problem similar to the above (example_1, Example_2, Example_3).
Anyhow I still have some doubt about how to proceed:

In my network I have something like this:

cfg = {

    'VGG16': [64, 'Dp', 64, 'M', 128, 'Dp', 128, 'M', 256, 'Dp', 256, 'Dp', 256, 'M', 512,'Dp', 512,'Dp', 512, 'M', 512,'Dp', 512,'Dp', 512, 'A', 'Dp'], #dropouts dependent from a single parameter (useful for hyper-par optim.)

}

class VGG(nn.Module, NetVariables, OrthoInit):
    def __init__(self, params):

        self.params = params.copy()

        nn.Module.__init__(self)
        NetVariables.__init__(self, self.params)
        OrthoInit.__init__(self)

        self.features = self._make_layers(cfg['VGG16'])
        self.classifier = nn.Linear(512, self.num_classes)

        self.weights_init() #call the orthogonal initial condition

    def forward(self, x):
        outs = {}
        L2 = self.features(x)
        outs['l2'] = L2
        Out = L2.view(L2.size(0), -1)
        Out = self.classifier(Out)
        outs['out'] = Out
        return outs

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]

            elif x=='A':
                layers += [nn.AvgPool2d(kernel_size=2, stride=2)]

            elif x == 'D3':
                layers += [nn.Dropout(0.3)]

            elif x == 'D4':
                layers += [nn.Dropout(0.4)]

            elif x == 'D5':
                layers += [nn.Dropout(0.5)]

            elif x == 'Dp':
                layers += [nn.Dropout(self.params['dropout_p'])]

            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),

                           nn.Tanh()
                           ,nn.GroupNorm(int(x/self.params['group_factor']), x)
                           ]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)

How can I adapt the code in such a way:

Substitute Dropout in nn.Sequential by a layer that is nn.Identity when model.training()==False (evaluation mode) and is a mask layer when model.training()==False (training mode)

the mask mentioned in the point 1. keeps constant and changes only when triggered by a given external input flag

Context:
Either use the proposed approach from your cross post 1 or initialize the layers first and use a switch in the custom forward method.

I would write a custom dropout layer, which accepts an additional flag to resample the mask.


TF to PT weight tranfer doesnt get same eval results

Question:
Hey there,
I’m trying to transfer weights between pytorch and tensorflow and viceversa. The problem that I’m getting is that even having the same weights for all layers the results are different for evaluation of the model and i can’t understand why.
this is my tensorflow model:
num_classes = 10
input_shape = (28, 28, 1)
tf.random.set_seed(2)
model = keras.Sequential(
[
keras.Input(shape=input_shape),
layers.Conv2D(4, kernel_size=(3, 3), name = “conv2d”),
layers.MaxPooling2D(pool_size=(2, 2)),
layers.Flatten(name=“flatten”),
layers.Dense(num_classes, activation=“softmax”,name=“dense”),
]
)
and this is my pytorch model
torch.manual_seed(2)
class Network(nn.Module):
def init(self):
super().init()
self.conv2d = nn.Conv2d(1,4,kernel_size=(3,3))
self.hidden = nn.MaxPool2d((2,2))
self.dense = nn.Linear(676, 10)
self.softmax = nn.Softmax(dim=1)
def forward(self, x):
    in_size = x.size(0)
    x = self.hidden(self.conv2d(x))
    x = x.view(in_size, -1)
    x = self.dense(x)
    return self.softmax(x)

I also test it after training tensorflow model but even after the weight the tranfer the results are completly different:
Tensorflow acc:  Test accuracy: 0.9228000044822693
Pytorch acc : Test set: Average loss: 0.0013, Accuracy: 972/10000 (10%)
These are the weights also after the training:
Tensorflow :
[array([[[[ 0.46365955, -0.12814473,  0.14183734, -0.5937301 ]],
     [[ 0.35576758,  0.5139991 ,  0.6756656 , -0.65329105]],

     [[ 0.48599064,  0.57487786, -0.11121935, -0.6914515 ]]],

    [[[ 0.27835783,  0.43169427,  0.33743912, -0.7600264 ]],

     [[ 0.547942  ,  1.0308589 ,  0.489442  , -0.67891765]],

     [[ 0.5344402 ,  0.6332875 ,  0.6738143 , -0.43144146]]],

    [[[-0.05558264,  0.7325994 ,  0.28004876, -0.17341942]],

     [[-0.1670951 ,  0.7585311 ,  0.54995114, -0.36898494]],

     [[ 0.3589517 ,  0.14311437,  0.41515586, -0.5163795 ]]]],
   dtype=float32),

array([ 0.27913818, -0.39000487, -0.23639436, -0.2614709 ], dtype=float32)]
Pytorch :
Parameter containing:
tensor([[[[ 0.4637,  0.3558,  0.4860],
[ 0.2784,  0.5479,  0.5344],
[-0.0556, -0.1671,  0.3590]]],
    [[[-0.1281,  0.5140,  0.5749],
      [ 0.4317,  1.0309,  0.6333],
      [ 0.7326,  0.7585,  0.1431]]],

    [[[ 0.1418,  0.6757, -0.1112],
      [ 0.3374,  0.4894,  0.6738],
      [ 0.2800,  0.5500,  0.4152]]],

    [[[-0.5937, -0.6533, -0.6915],
      [-0.7600, -0.6789, -0.4314],
      [-0.1734, -0.3690, -0.5164]]]], requires_grad=True)

Parameter containing:
tensor([ 0.2791, -0.3900, -0.2364, -0.2615], requires_grad=True)
Can someone give me some light on what is creating this big difference?

Context:
I don’t know how you’ve loaded the parameters from TF to PyTorch, but would recommend to check the layers in isolation and make sure e.g. the memory layout, flipping etc. are right.
In your PyTorch model you re using nn.Softmax as the last activation, which is usually wrong. Guessing you are working on a multi-class classification using nn.CrossEntropyLoss, remove the softmax as the criterion expect raw logits.


ZeroDivisionError when trying to train MaskRCNN

Question:
I am trying to fine tune MaskRCNN instance segmentation following this tutorial. My custom dataset and model build seem to pass, but I run into this ZeroDivisionError upon training. Any hints on how to debug are appreciated. Happy to include more code, did not want to overwhelm with irrelevant …

ZeroDivisionError                         Traceback (most recent call last)
 in ()
4 for epoch in range(num_epochs):
5     # train for one epoch, printing every 10 iterations
----> 6     train_one_epoch(model, optimizer, data_loader, device, epoch,print_freq=100)
7     # update the learning rate
8     lr_scheduler.step()
1 frames
/content/utils.py in log_every(self, iterable, print_freq, header)
216         total_time_str = str(datetime.timedelta(seconds=int(total_time)))
217         print(’{} Total time: {} ({:.4f} s / it)’.format(
 → 218             header, total_time_str, total_time / len(iterable)))
219
220
ZeroDivisionError: float division by zero

Context:
I don’t know why shuffle=True should raise the error only, as it seems that your dataset might be empty.
While creating the DataLoader a RandomSampler or SequentialSampler will be created if shuffle is set to True or False, respectively, and no custom sampler is passed as seen here 1.
In both cases the length of the data_source will be used (in your case the dataset) here 1 or here 1 so I would assume that both use cases fail if the dataset is empty.
Could you check print(len(dataset)) before passing it to the DataLoader and make sure it has a valid length?


Question:
So i just started learning programming and i wanted to try out a really simple NN which i then made with help of a  YT video.Which means that this code is a 100 percent copied… for what ever reason i get that error message and i dont know how to fix it
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.optim as optim
class MeinNetz(nn.Module):
def __init__(self):
    super(MeinNetz, self).__init__()
    self.lin1 = nn.Linear(10, 10)
    self.lin2 = nn.Linear(10, 10)

def forward(self, x):
    x = F.relu(self.lin1(x))
    x = self.lin2(x)
    return x

def num_flat_features(self,x):
    size = x.size()[1:]
    num = 1
    for i in size:
        num *= i
        return num

netz = MeinNetz()
for i in range (100):
x = [1,0,0,0,1,0,0,0,1,1]
target = Variable(torch.Tensor([x for _ in range(10)]))
out = netz(input)

x= [0,1,1,1,0,1,1,1,0,0]
target = Variable(torch.Tensor([x for _ in range(10)]))
criterion = nn.MSELoss()
loss = criterion(out,target)
print(loss)

netz.zero_grad()
loss.backward()
optimizer = optim.SGD(netz.parameters(), lr=0.01)
optimizer.step()

This is my Error:
Traceback (most recent call last):
File “”, line 1, in
runfile(‘G:/Erstes NN(6.11.19).py’, wdir=‘G:’)
File “E:\Programmieren\Anaconda\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 827, in runfile
execfile(filename, namespace)
File “E:\Programmieren\Anaconda\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 110, in execfile
exec(compile(f.read(), filename, ‘exec’), namespace)
File “G:/Erstes NN(6.11.19).py”, line 43, in
out = netz(input)
File “E:\Programmieren\Anaconda\lib\site-packages\torch\nn\modules\module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “G:/Erstes NN(6.11.19).py”, line 24, in forward
x = F.relu(self.lin1(x)) #letzes mus hier immer ein return sein#
File “E:\Programmieren\Anaconda\lib\site-packages\torch\nn\modules\module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “E:\Programmieren\Anaconda\lib\site-packages\torch\nn\modules\linear.py”, line 67, in forward
return F.linear(input, self.weight, self.bias)
File “E:\Programmieren\Anaconda\lib\site-packages\torch\nn\functional.py”, line 1350, in linear
if input.dim() == 2 and bias is not None:
AttributeError: ‘function’ object has no attribute ‘dim’

Context:
Hi Malte!

 Malte5768:

This is my Error:
Traceback (most recent call last):
…
File “G:/Erstes NN(6.11.19).py”, line 43, in
out = netz(input)
…
File “E:\Programmieren\Anaconda\lib\site-packages\torch\nn\functional.py”, line 1350, in linear
if input.dim() == 2 and bias is not None:
AttributeError: ‘function’ object has no attribute ‘dim’

Your problem is that (in your code) input is not defined – or, more
precisely, it is defined by python to be a function.  (This kind of thing is
a common source of errors in weakly-type languages such as python.)
Python’s input function doesn’t have a dim attribute, hence the
error.
Here are some results from a python session that illustrate this:
>>> type (input)
<class 'builtin_function_or_method'>
>>> input.dim()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'builtin_function_or_method' object has no attribute 'dim'
>>> import torch
>>> input = torch.randn (3, 4)
>>> type (input)
<class 'torch.FloatTensor'>
>>> input.dim()
2

When the python session first starts, input exists and is defined by
python to be a ‘builtin_function_or_method’.
Then I import torch and assign a tensor to input.  (Python is very
permissive – you are allowed to assign things to variable names that
are used for python builtins.)  Now input is a ‘torch.FloatTensor’, and
you can happily call its dim() method.
Note, input in:
File “G:/Erstes NN(6.11.19).py”, line 43, in
out = netz(input)

and in:
File “E:\Programmieren\Anaconda\lib\site-packages\torch\nn\functional.py”, line 1350, in linear
if input.dim() == 2 and bias is not None:

are not, in general, the same thing.  They just happen to be labelled
with the same character string.
In the first case, input is whatever it is when you call netz(input),
namely the python builtin.  In the second case it is the name of an
argument to a function inside of functional.py.  In your situation, it
is the same as the first input because it was passed as an
argument through a series of functions until it gets to that line of
code in functional.py that actually tries to do something with it, at
which point python discovers that it isn’t what you thought it was.
(In pytorch, “input” is often used as the name of the tensor that
is passed to a model.  You could argue that pytorch shouldn’t be
using names of python builtins for these sorts of things, but that
ship has long since sailed.)
How to fix it?  You need to pass some kind of sensible data to your
model.  If you copied your code from a tutorial, either the tutorial
is buggy, or you missed copying the part of the code where “input”
is defined to be something meaningful for training the model.
Good luck.
K. Frank


Question:
I have  been trying to download pytorch and this is error message i get: Could not remove or rename C:\ProgramData\Anaconda3\pkgs\pytorch-1.5.1-py3.7_cuda102_cudnn7_0\Lib\site-packages\torch\lib\cudnn64_7.dll.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(140): Could not remove or rename C:\ProgramData\Anaconda3\pkgs\pytorch-1.5.1-py3.7_cuda102_cudnn7_0\Lib\site-packages\torch\lib\cudnn64_7.dll.  Please remove this file manually (you may need to reboot to free file handles)

Context:
It seems conda is trying to remove the mentioned file, but isn’t able to do so.
Did you try to remove it manually (or reboot and remove it) as suggested in the error message?


Question:
I think words in build-version in  libtorch-win-shared-with-deps-debug-1.10+cu113  seems wrong.
It should be 1.10.0+cu113  but it reads "1.10.1+cu113. "

Context: I check it again. Build-version in libtorch-win-shared-with-deps-debug-1.10.0+cu113 is right.    You are right.  Maybe I got some mistakes.   Thank you for your reply.


Input size for EfficientNet versions from torchvision.models

Question:
Hi guys!
I’m doing some experiments with the EfficientNet as a backbone.
I’m using the pre-trained EfficientNet models from torchvision.models.
As I found from the paper and the docs of Keras, the EfficientNet variants have different input sizes as below.

image734×355 11.2 KB

Is it true for the models in Pytorch?
If I want to keep the same input size for all the EfficientNet variants, will it affect the performance of the models?

Context:
Yes, the torchvision EfficienNet models also expect a different input shape as described in the docs 4:

The sizes of the EfficientNet models depend on the variant. For the exact input sizes check here 4

where the link points to:
    elif args.model.startswith('efficientnet_'):
        sizes = {
            'b0': (256, 224), 'b1': (256, 240), 'b2': (288, 288), 'b3': (320, 300),
            'b4': (384, 380), 'b5': (489, 456), 'b6': (561, 528), 'b7': (633, 600),
        }


Image warping for backward flow using forward flow matrix- optical flow

Question:
Hi
The code I’m using to warp the image is

B, C, H, W = image_B.size()

xx = torch.arange(0, W).view(1,-1).repeat(H,1)

yy = torch.arange(0, H).view(-1,1).repeat(1,W)

xx = xx.view(1,1,H,W).repeat(B,1,1,1)

yy = yy.view(1,1,H,W).repeat(B,1,1,1)

grid = torch.cat((xx,yy),1).float()

vgrid = grid + flow_AB

vgrid[:,0,:,:] = 2.0*vgrid[:,0,:,:].clone() / max(W-1,1)-1.0

vgrid[:,1,:,:] = 2.0*vgrid[:,1,:,:].clone() / max(H-1,1)-1.0

warped_image = torch.nn.functional.grid_sample(image_B, vgrid.permute(0,2,3,1), 'nearest')

I’ve also tried creating a flow explicitly using image processing techniques
img2 = cv2.imread('frame_0011.png',0)

img1 = cv2.imread('frame_0010.png',0)

flow = cv2.calcOpticalFlowFarneback(img2, img1,  None, 0.5, 3, 15, 3, 5, 1.2, 0)

image2 = cv2.imread('frame_0011.png',1)

plt.imshow(cv2.remap(image2, flow, None, cv2.INTER_LINEAR))

But it doesn’t hold up
This is the difference with the image I want to warp to

I don’t get why it’s not working. Could anyone please help me?

Context:
This is the image I’m warping to (imageA) and the image I’m warping from (imageB)
image1046×229 78.6 KB


Issues with Dataloader for imagenet (should I use datasets.ImageFolder or datasets.ImageNet)?

Question:
I want to use the pretrained models from pytorch and evaluate them on imagenet val data. That should be fairly straightforward, but I am getting stuck on the dataloader.
I downloaded the imagenet and the folder structure that I have is like this:
/media/SSD2/ILSVRC/
                |----Annotation
                |----ImageSets
                |----Data
                      |----CLS-LOC
                               |----test
                               |----train
                               |----val
                                      |----ILSVRC2012_val_00000009.JPEG
                                      |----ILSVRC2012_val_00000010.JPEG
                                      |----...

First I tried datasets.ImageNet, based on pytorch 1  where it says to use the following
imagenet_data = torchvision.datasets.ImageNet('path/to/imagenet_root/')
data_loader = torch.utils.data.DataLoader(imagenet_data,
                                          batch_size=4,
                                          shuffle=True,
                                          num_workers=args.nThreads)

I changed the path_to_imagenet_to /media/SSD2/ILSVRC/ like this
torchvision.datasets.ImageNet('/media/SSD2/ILSVRC/',split='val',download=False)
but I get this error:
RuntimeError: The archive ILSVRC2012_devkit_t12.tar.gz is not present in the root directory or is corrupted. You need to download it externally and place it in /media/SSD2/ILSVRC/.

Then I tried using datasets.ImageFolder in the following way:
imagenet_data = torchvision.datasets.ImageFolder('/media/SSD2/ILSVRC/Data/CLS-LOC/val' , transform=transform)
data_loader = torch.utils.data.DataLoader(imagenet_data,
                                          batch_size=4,
                                          shuffle=True,
                                          num_workers=1)

it kinda works but the labels that generate are garbage.
I was wondering if anyone knows which one of the aforementioned methods can work and how should I do it?

Context:
I don’t know how and where you’ve downloaded the ImageNet dataset, but on my system each class uses a separate subfolder, which is needed in order to find all classes.
E.g. the val folder contains:
n01440764  n01824575  n02093754  n02115641  n02457408  n02877765  n03272010  n03710193  n03958227  n04296562  n04597913
n01443537  n01828970  n02093859  n02115913  n02480495  n02879718  n03272562  n03710637  n03961711  n04310018  n04599235
n01484850 ...

where each folder contains the images for this particular class.


Question:
Hi, I have trained MLP Mixer. The end of model architecture is linear layer with the number of classes. I am calculating the accuracy with the code below
def test(args, testloader, model, criterion):
    model.eval()
    testloss = 0
    correct = 0
    i = -1
    with torch.no_grad():
        for data, target in tqdm(testloader):
            data, target = data.to(args.device), target.to(args.device)
            output = model(data)
            loss = criterion(output, target)
            testloss += loss.item()
            #_, predicted = torch.max(output, 1)
            _,predicted = torch.max(nn.Softmax()(preds),dim = 1)
            correct += (predicted == target).sum().item()
            i += 1
        print("80.78554")
    return correct/(i+1)*args.batch_size

What is the problem with this code beacuse I got 1.38 accuracy. The dataset has 10 classes. If I randomly pick one of them the result becomes %10 instead of %1.38. Dataset and dataloaders embedded pytorch classes. Optimizer is adam and the criterion is CrossEntropy.

Context:
This line of code seems to be missing parentheses such that args.batch_size is in the divisor:
correct/(i+1)*args.batch_size


Question: I’m not very experienced in using CNNs but I want to use one in a reinforcement learning environment where I have 4 stacked gray scale images. How do I begin to write the parameters for the Conv layer? I don’t know how to find out the dimensions of the images (which are just arrays) or how to make a conv layer for stacked frames.

Context:
Bobingstern:

I don’t know how to find out the dimensions of the images (which are just arrays)

Once you’ve created the tensor you can check its shape via print(tensor.shape). Depending how you’ve created this particular tensor (from a dataset or DataLoader) it would have 3 or 4 dimensions. A DataLoader will create a batch of samples, so it would have 4 dimensions in the shape [batch_size, channels, height, width].

 Bobingstern:

how to make a conv layer for stacked frames.

Based on your description it seems you are dealing with images with 4 channels, so the first conv layer should be created via in_channels=4.


Heatmap as an output of a model in pytorch

Question: Can I use heatmap as an output of my model along with the main output in pytorch ? I want to generate a loss from heatmap

Context: You could treat the additional heatmap output in the same way as the original output, i.e. return it in the forward method of your model, and calculate the loss(es). I’m not sure if you are stuck at the implementation of this approach or if you are looking for a general advice regarding loss functions etc.


RuntimeError: Found dtype Double but expected Float during (loss.backward())

Question:
Hi Team, I’m performing object detection task. I’m getting below error,
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_8392/1841461244.py in <module>
      2     _n = len(train_loader)
      3     for ix, inputs in enumerate(train_loader):
----> 4         loss, losses = train_batch(inputs, model, optimizer)
      5         loc_loss, regr_loss, loss_objectness, loss_rpn_box_reg = \
      6             [losses[k] for k in ['loss_classifier','loss_box_reg','loss_objectness','loss_rpn_box_reg']]

~\AppData\Local\Temp/ipykernel_8392/1903493105.py in train_batch(inputs, model, optimizer)
      8     losses = model(input_, targets)
      9     loss = sum(loss for loss in losses.values())
---> 10     loss.backward()
     11     optimizer.step()
     12     return loss, losses

~\miniconda3\envs\pytorch_gpu\lib\site-packages\torch\_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    253                 create_graph=create_graph,
    254                 inputs=inputs)
--> 255         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    256
    257     def register_hook(self, hook):

~\miniconda3\envs\pytorch_gpu\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    147     Variable._execution_engine.run_backward(
    148         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 149         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    150
    151

RuntimeError: Found dtype Double but expected Float

please find the code which I used,
from torch_snippets import *
from PIL import Image
import glob, numpy as np, cv2, warnings,random, albumentations as A
from albumentations.pytorch.transforms import ToTensorV2
warnings.filterwarnings('ignore')

def seed_everything(seed):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = True

seed_everything(42)

IMAGE_ROOT = 'images'
DF_RAW = pd.read_csv('train_labels.csv')
DF_RAW['image_id'] = DF_RAW['filename'].apply(lambda x: x.split('.')[0])
DF_RAW['labels'] = DF_RAW['class'].apply(lambda x: 1 if x=='car' else 0)

label2target = {l:t+1 for t,l in enumerate(DF_RAW['class'].unique())}
label2target['background'] = 0
target2label = {t:l for l,t in label2target.items()}
background_class = label2target['background']
num_classes = len(label2target)

def preprocess_image(img):
    img = torch.tensor(img).permute(2,0,1)
    return img.to(device).float()

class OpenDataset(torch.utils.data.Dataset):
    def __init__(self, df, image_folder=IMAGE_ROOT,transforms = None):
        self.root = image_folder
        self.df = df
        self.unique_images = df['image_id'].unique()
        self.transforms = transforms
    def __len__(self): return len(self.unique_images)
    def __getitem__(self, ix):
        image_id = self.unique_images[ix]
        #image_path = f'{self.root}/{image_id}.jpg'
        image = cv2.imread(f'{self.root}/{image_id}.jpg', cv2.IMREAD_COLOR)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)
        image = np.array(image)/255
        df = self.df.copy()
        df = df[df['image_id'] == image_id]
        boxes = df[['xmin','ymin','xmax','ymax']].values
        classes = df['class'].values
        target = {}
        target["boxes"] = boxes
        #target["labels"] = torch.Tensor([label2target[i] for i in classes]).long()
        target["labels"] = torch.Tensor([label2target[i] for i in classes]).long()

        if self.transforms:
            sample = {
                'image': image,
                'bboxes': target['boxes'],
                'labels': torch.Tensor([label2target[i] for i in classes]).long()
            }
            sample = self.transforms(**sample)

            target["boxes"] = torch.as_tensor(sample['bboxes'])
            #target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)
        image = preprocess_image(image)

        return image, target
    def collate_fn(self, batch):
        return tuple(zip(*batch))

def get_train_transform():
    return A.Compose([
        A.HorizontalFlip(p=0.5),
        ToTensorV2(p=1.0)
    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})

def get_valid_transform():
    return A.Compose([
        ToTensorV2(p=1.0)
    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})

from sklearn.model_selection import train_test_split
trn_ids, val_ids = train_test_split(DF_RAW['image_id'].unique(), test_size=0.1, random_state=99)
trn_df, val_df = DF_RAW[DF_RAW['image_id'].isin(trn_ids)], DF_RAW[DF_RAW['image_id'].isin(val_ids)]
print(len(trn_df), len(val_df))

train_ds = OpenDataset(trn_df,transforms = get_train_transform())
test_ds = OpenDataset(val_df,transforms = get_valid_transform())

train_loader = DataLoader(train_ds, batch_size=2, collate_fn=train_ds.collate_fn, drop_last=True,shuffle=True)
test_loader = DataLoader(test_ds, batch_size=2, collate_fn=test_ds.collate_fn, drop_last=True,shuffle=False)

import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

device = 'cuda' if torch.cuda.is_available() else 'cpu'

def get_model():
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
    return model

# Defining training and validation functions for a single batch
def train_batch(inputs, model, optimizer):
    model.train()
    input_, targets = inputs
    input_ = list(image.to(device) for image in input_)
    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
    optimizer.zero_grad()
    losses = model(input_, targets)
    loss = sum(loss for loss in losses.values())
    loss.backward()
    optimizer.step()
    return loss, losses

@torch.no_grad() # this will disable gradient computation in the function below
def validate_batch(inputs, model):
    model.train() # to obtain the losses, model needs to be in train mode only. # #Note that here we are not defining the model's forward method
#and hence need to work per the way the model class is defined
    input_, targets = inputs
    input_ = list(image.to(device) for image in input_)
    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

    optimizer.zero_grad()
    losses = model(input_, targets)
    loss = sum(loss for loss in losses.values())
    return loss, losses

model = get_model().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=0.005,
                            momentum=0.9, weight_decay=0.0005)
n_epochs = 5
log = Report(n_epochs)

for epoch in range(n_epochs):
    _n = len(train_loader)
    for ix, inputs in enumerate(train_loader):
        loss, losses = train_batch(inputs, model, optimizer)
        loc_loss, regr_loss, loss_objectness, loss_rpn_box_reg = \
            [losses[k] for k in ['loss_classifier','loss_box_reg','loss_objectness','loss_rpn_box_reg']]
        pos = (epoch + (ix+1)/_n)
        log.record(pos, trn_loss=loss.item(), trn_loc_loss=loc_loss.item(),
                   trn_regr_loss=regr_loss.item(), trn_objectness_loss=loss_objectness.item(),
                   trn_rpn_box_reg_loss=loss_rpn_box_reg.item(), end='\r')

    _n = len(test_loader)
    for ix,inputs in enumerate(test_loader):
        loss, losses = validate_batch(inputs, model)
        loc_loss, regr_loss, loss_objectness, loss_rpn_box_reg = \
          [losses[k] for k in ['loss_classifier','loss_box_reg','loss_objectness','loss_rpn_box_reg']]
        pos = (epoch + (ix+1)/_n)
        log.record(pos, val_loss=loss.item(), val_loc_loss=loc_loss.item(),
                  val_regr_loss=regr_loss.item(), val_objectness_loss=loss_objectness.item(),
                  val_rpn_box_reg_loss=loss_rpn_box_reg.item(), end='\r')
    if (epoch+1)%(n_epochs//5)==0: log.report_avgs(epoch+1)

I’m trying to fix this, it would be help if anyone can help me here.
Thanks in advance.

Context: Could you replace the data loading with some random tensors and check if the error would still be reproducible? If so, could you then post it so that we could debug? If not, I would guess that your data loading is creating some inputs in float64, which might be creating the issue so you could try to narrow down which of these tensors is could be.


Why is Faster RCNN bounding box always in the range [0,300]?

Question:
I am using the a Faster RCNN model to learn an object detection task with one class of objects (plus the background).

The size of images is 1000x1000x3.
After training, the bounding boxes are always in the range [0,300]

Even after resizing the images to any other size, the predicted bounding boxes are always in this range

I need help with the following questions:

Why is this happening?
Is there a way to avoid this? Nobody else seems to be having this problem, so is there anything that I am supposed to do while constructing the model, or at any other step?

Code for loading and training the model:
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
num_classes = 2  # 1 class (insulator) + background
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
model.transform = torchvision.models.detection.transform.GeneralizedRCNNTransform(min_size=1000,
                                                                                max_size=1000,
                                                                                image_mean=[0.485, 0.456, 0.406],
                                                                                image_std=[0.229, 0.224, 0.225])

params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=1e-2, momentum=0.9, weight_decay=0.005)

num_epochs = 10
for epoch in range(num_epochs):
        train_one_epoch(model, optimizer, data_loader_train, torch.device('cpu'), epoch, print_freq=10)

As mentioned, I have changed the size to different combinations, like [500x500], [800x800], [500x800], [800x500]. The result is still always the same - the predicted bounding boxes are in the range [0,300].
A sample set of predicted bounding boxes:
tensor([[108,   8, 140, 128],
        [ 39, 213,  61,  62],
        [ 36, 172,  56,  11],
        [ 97, 113, 115, 193],
        [247, 148,   8, 234],
        [229, 220, 254,  74],
        [ 28,  86,  44, 141],
        [213, 122, 233, 177],
        [138, 100, 209, 128],
        [239, 219,   1,  61],
        [ 44, 172,  58, 253],
        [ 29, 176,  63,  56],
        [227, 220, 244,  62],
        [  4, 240,  26,  12],
        [ 37, 177,  56, 213],
        [ 40, 187,  60, 225],
        [ 90, 114, 107, 188],
        [ 75, 112, 118, 186]], dtype=torch.uint8)

The ground truth for this sample:
tensor([[367., 782., 394., 899.],
        [552., 716., 573., 829.],
        [488., 466., 508., 583.],
        [807., 427., 823., 524.],
        [760., 661., 777., 750.],
        [866., 624., 879., 706.],
        [797., 858., 812., 910.]])

Context:
I don’t think the model is clipping the bounding box predictions, as they should be created in this linear layer.
If I run the detection tutorial 2 on Colab I also get predictions outside the 300x300 range by adding this at the end of the notebook:
 import torch.nn.functional as F
img = img.unsqueeze(0)
img = F.interpolate(img, (500, 500))
img = img.squeeze(0)
with torch.no_grad():
    prediction = model([img.to(device)])
prediction[0]['boxes']
> tensor([[104.4511,  59.9262, 339.6740, 466.1226],
          [474.9314,  37.3134, 497.4988, 105.6687]], device='cuda:0')

img = img.unsqueeze(0)
img = F.interpolate(img, (1000, 1000))
img = img.squeeze(0)
with torch.no_grad():
    prediction = model([img.to(device)])
print(prediction[0]['boxes'])
> tensor([[207.8041, 118.8091, 676.5738, 933.9277]], device='cuda:0')


Question:
i need to extract features before classifying it … i think i should ignore those layers (avgpool and head)… right? how can i do this

Screenshot from 2021-12-19 15-51-021054×551 67.3 KB

Context: is there any help please ?


How to change the shape inside DataLoader?

Question:
When training on a classification problem with multiclass and using CrossEntropyLoss, the targets should integer, and hence targets.long() is needed. But when training on multilabel or binary, we may invoke BCEWithLogitsLoss, which expects your targets to be in float.
I made changes in Dataset as follows:
if CRITERION_PARAMS.train_criterion_name == "BCEWithLogitsLoss":
    # Make changes to reshape rather than in Trainer.
    y = torch.as_tensor(y, dtype=torch.float32).view(-1, 1)
else:
    y = torch.as_tensor(y, dtype=torch.long)

However, I soon realized that this does not work since DataLoader will (collate?) the shape wrongly. What I expect is the following:
original_y_shape -> (batch_size, ) or (batch_size, 1)
modified_y_shape -> (batch_size, 1) # to match shapes with BCEWithLogitsLoss

But if I change the shape inside Dataset, the DataLoader gives me (batch_size, 1, 1) instead.
One easy way is to change the shape in the training function directly. But I would like to keep the changes in Dataset unless there is a strong reason not to (speed issues).

Context: Or you can do flatten() instead, then both (x,1) and (x) will result in (x)


Can I use torchvision Dataset and Dataloader with AWS S3?

Question: So basically I would like to use ImageFolder dataset with path data_path = s3://bucketname/image_folder and a DataLoader. Should be examples of something like that lying around, but just can’t find anything very useful…

Context: Were you able to solve this?


Memory Error with UNET implementation, on both CPU and GPU

Question:
Hi, I am a newbie to pytorch, I made the following pytorch implementation of pytorch, it seems correct but it always kills memory, both cuda and cpu.
class DoubleConv(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.duo = nn.Sequential(
                    nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False),
                    nn.ReLU(inplace=True),
                    nn.BatchNorm2d(out_channels),
                    nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False),
                    nn.ReLU(inplace=True),
                    nn.BatchNorm2d(out_channels),
                )

    def forward(self, x):
        return self.duo(x)

class UNET1(nn.Module):
    def __init__(self, in_channel=3, out_channel=1, features=[64, 128, 256, 512]):
        super().__init__()
        self.downs = nn.ModuleList()
        self.ups = nn.ModuleList()

        self.copy = []
        self.compilation = []
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.features = features

        in_feature = in_channel
        for feature in features:
            model = DoubleConv(in_feature, feature)
            in_feature = feature
            self.downs.append(model)
        self.baseconv = DoubleConv(feature, feature*2)

        for feature in reversed(features):
          conv = nn.ConvTranspose2d(feature*2, feature, kernel_size=2, stride=2, bias=False)
          self.ups.append(conv)
          self.ups.append(DoubleConv(feature*2, feature))
        self.last_conv = nn.Conv2d(features[0], out_channel, kernel_size=1, stride=1)

    def encoder(self, x):
        for down in self.downs:
            x = down(x)
            self.copy.append(x)
            x = self.pool(x)
            self.compilation.append(x)

        self.copy = self.copy[::-1]
        return self.compilation

    def base(self):
      last = self.compilation[-1]
      basic = self.baseconv(last)
      self.compilation.append(basic)
      return basic

    def decoder(self):
        layer = self.compilation[-1]
        for enum in range(0, len(self.ups), 2):
          up_conv = self.ups[enum](layer)
          concatena = TF.center_crop(self.copy[enum//2], up_conv.shape[2:])
          layer = torch.cat((concatena, up_conv), dim=1)
          layer = self.ups[enum+1](layer)
        return self.last_conv(layer)

    def forward(self, x):
      self.encoder(x)
      self.base()
      res  = self.decoder()
      return res

Now, I copied this code online, using torchinfo.summary, they had similar memory usage, but when actually running code, this does okay in terms of memory while my implementation always crash out with a memory error after an epoch.
class DoubleConv(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(DoubleConv, self).__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(in_channels, out_channels, 3, 1, 1, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_channels, out_channels, 3, 1, 1, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True),
        )

    def forward(self, x):
        return self.conv(x)

class UNET2(nn.Module):
    def __init__(
            self, in_channels=3, out_channels=1, features=[64, 128, 256, 512],
    ):
        super(UNET, self).__init__()
        self.ups = nn.ModuleList()
        self.downs = nn.ModuleList()
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)

        # Down part of UNET
        for feature in features:
            self.downs.append(DoubleConv(in_channels, feature))
            in_channels = feature

        # Up part of UNET
        for feature in reversed(features):
            self.ups.append(
                nn.ConvTranspose2d(
                    feature*2, feature, kernel_size=2, stride=2,
                )
            )
            self.ups.append(DoubleConv(feature*2, feature))

        self.bottleneck = DoubleConv(features[-1], features[-1]*2)
        self.final_conv = nn.Conv2d(features[0], out_channels, kernel_size=1)

    def forward(self, x):
        skip_connections = []

        for down in self.downs:
            x = down(x)
            skip_connections.append(x)
            x = self.pool(x)

        x = self.bottleneck(x)
        skip_connections = skip_connections[::-1]

        for idx in range(0, len(self.ups), 2):
            x = self.ups[idx](x)
            skip_connection = skip_connections[idx//2]

            if x.shape != skip_connection.shape:
                x = TF.resize(x, size=skip_connection.shape[2:])

            concat_skip = torch.cat((skip_connection, x), dim=1)
            x = self.ups[idx+1](concat_skip)

        return self.final_conv(x)

def test():
    x = torch.randn((3, 1, 161, 161))
    model = UNET(in_channels=1, out_channels=1)
    preds = model(x)
    assert preds.shape == x.shape

What am I missing please?

Context: What kind of “memory error” do you get? Could you post the error message with the stack trace, please?


RuntimeError: set_sizes_and_strides is not allowed on a Tensor created from .data or .detach()

Question:
I have a GAN style code, like below:
self.optimizer_generator.zero_grad()
fake_high_resolution = self.generator(low_resolution)

score_real = self.discriminator(high_resolution)
score_fake = self.discriminator(fake_high_resolution)

# calculate generator_loss

generator_loss.backward()
self.optimizer_generator.step()

self.optimizer_discriminator.zero_grad()

score_real = self.discriminator(high_resolution)
score_fake = self.discriminator(fake_high_resolution.detach())

# calculate discriminator_loss

discriminator_loss.backward()
self.optimizer_discriminator.step()

How this error come from and why? since I written my code based on code can correctly run.
env:
PyTorch version: 1.8.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 3090

Nvidia driver version: 460.67
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] pytorch-lightning==1.2.5
[pip3] torch==1.8.0
[pip3] torchelastic==0.2.2
[pip3] torchmetrics==0.2.0
[pip3] torchtext==0.9.0
[pip3] torchvision==0.9.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py38he904b0f_0
[conda] mkl_fft                   1.3.0            py38h54f3939_0
[conda] mkl_random                1.1.1            py38h0573a6f_0
[conda] numpy                     1.19.2           py38h54aff64_0
[conda] numpy-base                1.19.2           py38hfa32c7d_0
[conda] pytorch                   1.8.0           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
[conda] pytorch-lightning         1.2.5                    pypi_0    pypi
[conda] torchelastic              0.2.2                    pypi_0    pypi
[conda] torchmetrics              0.2.0                    pypi_0    pypi
[conda] torchtext                 0.9.0                      py38    pytorch
[conda] torchvision               0.9.0                py38_cu111    pytorch

Context: Could you post the complete stack trace for this error, which would also point to the line of code, which raises the issue, please?


Cuda Runtime Assertion Error:Index out of bound

Question:
Any help on the below error,reason of it and point of error
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [0,0,0], thread: [14,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [0,0,0], thread: [15,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [0,0,0], thread: [16,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [0,0,0], thread: [17,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [0,0,0], thread: [18,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [0,0,0], thread: [19,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/p

Error Trace
  231         mask_feats = mask_roi_extractor(x[:mask_roi_extractor.num_inputs],
--> 232                                 enlarged_rois)
    233         if self.with_semantic:
    234             mask_semantic_feat = self.semantic_roi_extractor([semantic_feat],

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py in new_func(*args, **kwargs)
    212                     digit_version(TORCH_VERSION) >= digit_version('1.6.0')):
    213                 with autocast(enabled=False):
--> 214                     output = old_func(*new_args, **new_kwargs)
    215             else:
    216                 output = old_func(*new_args, **new_kwargs)

/kaggle/working/mmdetection/mmdet/models/roi_heads/roi_extractors/single_level_roi_extractor.py in forward(self, feats, rois, roi_scale_factor)
     98                 roi_feats += roi_feats_t
     99                 continue
--> 100             inds = mask.nonzero(as_tuple=False).squeeze(1)
    101             if inds.numel() > 0:
    102                 rois_ = rois[inds]

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1

Context: Rerun the code via CUDA_LAUNCH_BLOCKING=1 python script.py args and check which operation is causing the out-of-bounds index error. Alternatively, run the code on the CPU which should also show the failing op.


Question:
Hello,
I’m new to PyTorch and I don’t know too well Tensorflow. I am currently re-writting a code in Tensorflow to Pytorch and they use tf.nn.dilation2d. I didn’t find any function close to the tensorflow’s one. Can anyone help me ?
Thanks !

Context:
I don’t think a dilation is implemented yet.
You could unfold the image tensor and apply the max operation on each patch.
Have a look at this example 193 to see how to create the patches.


Multiple GPUs training is different than Single GPU training

Question:
Hello everyone,
I use the Residual Flows network repository to train the network on the CIFAR-10 dataset. But, when I use 4 GPUs to speed up the training, generated fake images are very noisy. It seems the network isn’t trained in this way, however using 1 GPU the generated images are realistic.
One GPU:
Found 1 CUDA devices.
GeForce RTX 2080 Ti      Memory: 10.76GB
Current LR 0.001
/home/hkeshvarik/residual-flows/lib/optimizers.py:88: UserWarning: This overload of add_ is deprecated:
add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
add_(Tensor other, *, Number alpha) (Triggered internally at  …/torch/csrc/utils/python_arg_parser.cpp:1050.)
exp_avg.mul_(beta1).add_(1 - beta1, grad)
Epoch: [0][0/3125] | Time 3.300 | GradNorm 0.78 | Bits/dim 7.8337(7.8337) | Logpz -4263 | -DeltaLogp 4617 | EstMoment (1,20)
Epoch: [0][20/3125] | Time 1.873 | GradNorm 1.13 | Bits/dim 7.7749(7.8731) | Logpz -4344 | -DeltaLogp 4615 | EstMoment (1,19)
Epoch: [0][40/3125] | Time 1.814 | GradNorm 1.54 | Bits/dim 7.4018(7.7304) | Logpz -4063 | -DeltaLogp 4637 | EstMoment (17,46)
Epoch: [0][60/3125] | Time 1.856 | GradNorm 2.05 | Bits/dim 7.1400(7.5244) | Logpz -3775 | -DeltaLogp 4787 | EstMoment (84,475)
Epoch: [0][80/3125] | Time 1.886 | GradNorm 2.37 | Bits/dim 6.7452(7.2833) | Logpz -3603 | -DeltaLogp 5129 | EstMoment (255,3357)
Epoch: [0][100/3125] | Time 1.878 | GradNorm 2.57 | Bits/dim 6.3381(6.9521) | Logpz -3506 | -DeltaLogp 5738 | EstMoment (640,18912)
Epoch: [0][120/3125] | Time 1.948 | GradNorm 2.50 | Bits/dim 5.6473(6.4880) | Logpz -3546 | -DeltaLogp 6766 | EstMoment (1397,81412)
Epoch: [0][140/3125] | Time 2.003 | GradNorm 3.00 | Bits/dim 5.2198(5.9421) | Logpz -3730 | -DeltaLogp 8112 | EstMoment (2454,220792)
Epoch: [0][160/3125] | Time 1.907 | GradNorm 3.30 | Bits/dim 4.7968(5.5149) | Logpz -3933 | -DeltaLogp 9224 | EstMoment (3331,365044)
Epoch: [0][180/3125] | Time 1.906 | GradNorm 2.90 | Bits/dim 4.6803(5.2312) | Logpz -4101 | -DeltaLogp 9997 | EstMoment (3929,477682)
Epoch: [0][200/3125] | Time 1.966 | GradNorm 2.49 | Bits/dim 5.0990(5.0599) | Logpz -4239 | -DeltaLogp 10499 | EstMoment (4277,546810)
Quad GPUs:
Found 4 CUDA devices.
GeForce RTX 2080 Ti      Memory: 10.76GB
GeForce RTX 2080 Ti      Memory: 10.76GB
GeForce RTX 2080 Ti      Memory: 10.76GB
GeForce RTX 2080 Ti      Memory: 10.76GB
Current LR 0.001
/home/hkeshvarik/residual-flows/lib/optimizers.py:88: UserWarning: This overload of add_ is deprecated:
add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
add_(Tensor other, *, Number alpha) (Triggered internally at  …/torch/csrc/utils/python_arg_parser.cpp:1050.)
exp_avg.mul_(beta1).add_(1 - beta1, grad)
Epoch: [0][0/782] | Time 17.099 | GradNorm 0.25 | Bits/dim 7.8317(7.8317) | Logpz -4263 | -DeltaLogp 4621 | EstMoment (-0,14)
Epoch: [0][20/782] | Time 2.413 | GradNorm 0.59 | Bits/dim 8.0066(7.9317) | Logpz -4458 | -DeltaLogp 4603 | EstMoment (-1,15)
Epoch: [0][40/782] | Time 2.473 | GradNorm 0.70 | Bits/dim 8.0849(7.9832) | Logpz -4525 | -DeltaLogp 4561 | EstMoment (-1,15)
Epoch: [0][60/782] | Time 2.451 | GradNorm 0.63 | Bits/dim 7.9804(8.0029) | Logpz -4509 | -DeltaLogp 4503 | EstMoment (-1,15)
Epoch: [0][80/782] | Time 2.384 | GradNorm 0.57 | Bits/dim 8.0048(8.0065) | Logpz -4472 | -DeltaLogp 4458 | EstMoment (-1,15)
Epoch: [0][100/782] | Time 2.334 | GradNorm 0.58 | Bits/dim 8.0372(8.0153) | Logpz -4486 | -DeltaLogp 4454 | EstMoment (-1,15)
Any suggestion?

Context:
It’s a bit unclear what setup you are using so some guesses.
Are you using nn.DataParallel or DistributedDataParallel? In the latter case, are you using SyncBatchNorm layers? Is the batch size the same on each device using the single GPU run vs. the multi-GPU one? If so, this would mean that you are basically scaling up the global batch size by 4x and might not expect to see the same results.


Question: I don’t know how grid_sample() compute the output base on the grid and input?Does any one give me an example？Thank you.

Context:
Hi,
The doc on grid_sample 7.6k contains details on how the output is computed. Let us know if you need more precisions.


RuntimeError: CUDA out of memory ONLY for validation but NOT for training

Question:
Hello, I’m trying to train a pre-trained Faster R-CNN in Google Collab Pro. My function for the training is OK and everything works very well.
But I tried to launch a validation step using the exact same function except that i don’t use the backpropagation when I computed the loss.
But when I skip the backpropagation step, I get an error of CUDA MEMORY, whilst when I apply the backpropagation (during the training) I don’t have any error of CUDA MEMORY even without using the   torch.cuda.empty_cache(). This is my code :
def train_one_epoch(model, optimizer, data_loader, device, validation):
    model.train()
    for i, values in enumerate(data_loader):
        images, targets = values
        images = list(image.to(device) for image in images)
        targets = [{'boxes' : t['boxes'] , 'labels' : t['labels'].to(device)} for t in targets ]
        loss_dict = model(images, targets)
        losses = sum(loss for loss in loss_dict.values())
        optimizer.zero_grad()

        if validation == False :
            losses.backward()
            optimizer.step()

        torch.cuda.empty_cache()

The only thing that changes between training and validation is that during training I set validation = False, so I enter in the if condition. And during validation i set the validation variable = True, so I don’t ente rin the if condition.
So, setting validation to False gave me this error :
RuntimeError: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 15.90 GiB total capacity; 14.53 GiB already allocated; 25.75 MiB free; 14.86 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Does someone have an idea to fix it please ?

Context: I think if you wanted to disable the gradients for validation you might want to do torch.no_grad() for one of the steps rather than just conditionally skipping the calls to losses.backwards() and optimizer.step.  Or, alternatively, set model.eval() before running the evaluation step and then set model.train() when you’re done.


Question:
Hi there, I’m testing with fp16 features of pytorch with a benchmark script provided here 77, getting these result(all with CUDA8 and cuDNN6):
➜  ~ python test_pytorch_vgg19_fp16.py
Titan X Pascal(Dell T630, anaconda2, pytorch 0.3.0):
FP32 Iterations per second:  1.7890313980917067
FP16 Iterations per second:  1.8345766566297141

Tesla P100(DGX-1, anaconda3, pytorch 0.3.0):
('FP32 Iterations per second: ', 2.001722807676548)
('FP16 Iterations per second: ', 1.9109340821222125)

Tesla P100(DGX-1, pytorch docker image provided by NVIDIA, pytorch 0.2.0):
FP32 Iterations per second:  1.9826932313239591
FP16 Iterations per second:  1.8854441494961636

For the detailed information of the docker image refer to here 7, it is theoretically optimized for DGX-1.
It can be seen that on P100 fp16 is even slower than fp32. Am I doing anything wrong? or PyTorch fp16 support is still under heavily developing? If so is there any doc for fp16 support status?
I would like to do more testing if needed, thanks a lot!
And another thanks for this great framework!

Context: on P100 we dont expect FP16 to be any faster, because we disabled FP16 math on P100 (it is numerically unstable). We use simulated FP16, where storage is FP16, but compute is in FP32 (so it upconverts to FP32 before doing operations).


GPU utilisation low but memory usage high

Question:
GPU utilisation is low but memory is usage high. This is my code:
from tqdm import tqdm
import torchvision.transforms as transforms
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision.datasets import CIFAR100
import torch.nn as nn
from torch.functional import split
import torch
import ssl
ssl._create_default_https_context = ssl._create_unverified_context

class VGG(nn.Module):
    def __init__(self, in_channels=3, num_classes=1000):
        super(VGG, self).__init__()
        self.in_channels = in_channels
        self.conv_layers = self.create_conv_layers(VGG16)

        self.fcs = nn.Sequential(
            nn.Linear(512*7*7, 4096),
            nn.ReLU(),
            nn.Dropout(p=0.5),
            nn.Linear(4096, 4096),
            nn.ReLU(), nn.Dropout(p=0.6), nn.Linear(4096, num_classes)
        )

    def forward(self, x):
        x = self.conv_layers(x)
        x = x.reshape(x.shape[0], -1)
        x = self.fcs(x)
        return x

    def create_conv_layers(self, architecture):
        layers = []
        in_channels = self.in_channels

        for x in architecture:
            if type(x) == int:
                out_channels = x

                layers += [nn.Conv2d(in_channels=in_channels, out_channels=out_channels,
                                     kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
                           nn.BatchNorm2d(x),
                           nn.ReLU()
                           ]

                in_channels = out_channels
            else:
                layers += [nn.MaxPool2d(kernel_size=(2, 2), stride=(2, 2))]

        return nn.Sequential(*layers)

if __name__ == '__main__':
    VGG16 = [64, 64, 'M', 128, 128, 'M', 256, 256, 256,
             'M', 512, 512, 512, 'M', 512, 512, 512, 'M']

    batch_size = 32
    lr = 0.0001
    device = "cuda"
    num_epochs = 1
    transform = transforms.Compose(
        [transforms.ToTensor(), transforms.Resize((224, 224))])

    train_dataset = CIFAR100(
        root='dataset/', train=True, transform=transform, download=True)
    train_loader = DataLoader(dataset=train_dataset,
                              batch_size=batch_size, shuffle=True, num_workers=6, pin_memory=True)

    model = VGG().to(device)
    optimizer = optim.Adam(model.parameters(), lr=lr)
    criterion = nn.CrossEntropyLoss()

    for epoch in range(num_epochs):
        losses = []

        with tqdm(train_loader, unit="Batch", desc=f"Epoch {epoch + 1}") as tepoch:
            num_batches = len(tepoch)

            for batch_idx, (data, targets) in enumerate(tepoch):
                data = data.to(device=device)
                targets = targets.to(device=device)

                preds = model(data)
                loss = criterion(preds, targets)
                losses.append(loss.item())

                optimizer.zero_grad()
                loss.backward()

                optimizer.step()

I used the pytorch profiler. This is what is got:
--------------------------------------------------------------------------------
  Environment Summary
--------------------------------------------------------------------------------
PyTorch 1.10.0 DEBUG compiled w/ CUDA 11.3
Running with Python 3.8 and CUDA 11.2.67

`pip3 list` truncated output:
mypy-extensions==0.4.3
numpy==1.20.1
numpydoc==1.1.0
torch==1.10.0
torch-tb-profiler==0.3.1
torchaudio==0.10.0
torchvision==0.11.1
--------------------------------------------------------------------------------
  cProfile output
--------------------------------------------------------------------------------
         6012080 function calls (5926529 primitive calls) in 631.801 seconds

   Ordered by: internal time
   List reduced from 1550 to 15 due to restriction <15>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     3223  379.244    0.118  379.244    0.118 {method 'to' of 'torch._C._TensorBase' objects}
     1565  204.858    0.131  204.858    0.131 {method 'item' of 'torch._C._TensorBase' objects}
       12   12.053    1.004   12.054    1.005 {method 'dump' of '_pickle.Pickler' objects}
    20319    8.883    0.000    8.883    0.000 {built-in method conv2d}
     1563    8.339    0.005    8.339    0.005 {method 'run_backward' of 'torch._C._EngineBase' objects}
     4689    2.900    0.001    2.900    0.001 {built-in method torch._C._nn.linear}
     1563    1.661    0.001    6.370    0.004 C:\Users\techi\anaconda3\lib\site-packages\torch\optim\_functional.py:54(adam)
    20319    1.608    0.000    1.608    0.000 {built-in method batch_norm}
   181308    1.338    0.000    1.338    0.000 {method 'mul_' of 'torch._C._TensorBase' objects}
   181308    1.272    0.000    1.272    0.000 {method 'add_' of 'torch._C._TensorBase' objects}
    90654    0.751    0.000    0.751    0.000 {method 'sqrt' of 'torch._C._TensorBase' objects}
    90654    0.689    0.000    0.689    0.000 {method 'addcdiv_' of 'torch._C._TensorBase' objects}
    23445    0.648    0.000    0.648    0.000 {built-in method relu}
    90654    0.595    0.000    0.595    0.000 {method 'addcmul_' of 'torch._C._TensorBase' objects}
       32    0.569    0.018    0.569    0.018 {method 'uniform_' of 'torch._C._TensorBase' objects}

--------------------------------------------------------------------------------
  autograd profiler output (CUDA mode)
--------------------------------------------------------------------------------
        top 15 events sorted by cpu_time_total

        Because the autograd profiler uses the CUDA event API,
        the CUDA time column reports approximately max(cuda_time, cpu_time).
        Please ignore this output if your code does not use CUDA.

-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                               Optimizer.step#Adam.step        20.72%     498.367ms        20.91%     503.092ms     503.092ms     611.000us         9.00%     328.025ms     328.025ms             1
                                         aten::uniform_        18.01%     433.238ms        18.01%     433.238ms     433.238ms       1.000us         0.01%       1.000us       1.000us             1
enumerate(DataLoader)#_MultiProcessingDataLoaderIter...        14.69%     353.386ms        14.69%     353.390ms     353.390ms       2.000us         0.03%       3.000us       3.000us             1
                     Optimizer.zero_grad#Adam.zero_grad        13.09%     314.864ms        13.12%     315.547ms     315.547ms      77.000us         1.13%     317.186ms     317.186ms             1
                                               aten::to         0.00%      10.000us        11.90%     286.182ms     286.182ms       2.000us         0.03%       2.141ms       2.141ms             1
                                         aten::_to_copy         0.00%      19.000us        11.90%     286.172ms     286.172ms       4.000us         0.06%       2.139ms       2.139ms             1
                                            aten::copy_        11.90%     286.142ms        11.90%     286.142ms     286.142ms       2.134ms        31.43%       2.134ms       2.134ms             1
                                               aten::to         0.00%       7.000us        10.90%     262.290ms     262.290ms       2.000us         0.03%       2.135ms       2.135ms             1
                                         aten::_to_copy         0.00%      20.000us        10.90%     262.283ms     262.283ms       3.000us         0.04%       2.133ms       2.133ms             1
                                            aten::copy_        10.90%     262.254ms        10.90%     262.254ms     262.254ms       2.129ms        31.36%       2.129ms       2.129ms             1
                                               aten::to         0.00%       8.000us        10.69%     257.116ms     257.116ms       2.000us         0.03%       1.820ms       1.820ms             1
                                         aten::_to_copy         0.00%      20.000us        10.69%     257.108ms     257.108ms       4.000us         0.06%       1.818ms       1.818ms             1
                                            aten::copy_        10.69%     257.078ms        10.69%     257.078ms     257.078ms       1.813ms        26.70%       1.813ms       1.813ms             1
                                               aten::to         0.00%       7.000us        10.64%     255.998ms     255.998ms       2.000us         0.03%       1.963ms       1.963ms             1
                                         aten::_to_copy         0.00%      18.000us        10.64%     255.991ms     255.991ms       3.000us         0.04%       1.961ms       1.961ms             1
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 2.405s
Self CUDA time total: 6.789ms

Please help me find the issue. Thanks!

Context: I would recommend to check out the performance guide 4, which would point to towards using e.g. torch.backends.cudnn.benchmark = True and to e.g. avoid synchronizations via item() calls.


Question:
Hi everyone,
I want to modify the value of my learning rate at each step instead of doing it at the end of each epoch.
In particular at each step of my training I compute a generic measure x and I want to modify the learning rate for the next step as a function of x.
Just to give a more explicit idea of the pipeline, given the constant C what I want to do is something like:
for epoch in range (0, n_epochs):
    for data, label in train_loader:
        #training
        #computation of x
        #modify of the learning rate as a function of x (for example lr = C*(1+x))

Till now I’ve only used lr_schedule = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda = lr_rule)  to modify the learning rate value.
Is there a more flexible method in pytorch to modify it at each step and with a generic function of a generic computed measure?

Context:
I’m not sure if there is a more flexible class for your use case, but I guess you might skip using the scheduler and update the learning rate manually as you wish via:
for param_group in self.optimizer.param_groups:
    param_group['lr'] = lr


Question:
With a resnet50 or larger, I am trying to change the fc layer into a conv layer followed by a fc. I need to do this because of hardware constraint (2048 feature vector too large) and resnet34 may be too small to learn.
from torchvision.models import resnet50
model = resnet50(pretrained=True)

....
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)v
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=2048, out_features=1000, bias=True)

that last fc layer should be this
  (heads): EmbeddingHead(
    (pool_layer): GlobalAvgPool(output_size=1)
    (bottleneck): Sequential(
       (0): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
       (1): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
     )
     (fc): Linear(in_features=512, out_features=125, bias=True)
   )

From what I read this should work
self.bottleneck = nn.Sequential(
            nn.Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False),
            nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        )
self.logits = nn.linear(512, 125)
self.heads = nn.Sequential(*[self.bottleneck, self.logits])
model.fc = self.heads

but i keep getting
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [512, 2048, 1, 1], but got 2-dimensional input of size [8, 2048] instead

The goal is after training I can remove the fc layer at the end and have the 512 feature vectors

Context: The error is raised since the activation is flattened in the original resnet before being passed to self.fc as seen here 1. Since you expect a 4D tensor you could try to replace the self.avgpool with the original avgpool layer in combination with your additional modules, and replace self.fc with nn.Identity. Since this approach sounds quite brittle, the proper way would be to write a custom model and override the forward with your desired modules.


PyTorch Is not installing PIP - latest version

Question:
Here is the Pip command I used. I got it from the official website (Start Locally | PyTorch)
pip3 install torch==1.10.1+cu102 torchvision==0.11.2+cu102 torchaudio===0.10.1+cu102 -f https://download.pytorch.org/whl/cu102/torch_stable.html

The Error
ERROR: Could not find a version that satisfies the requirement torch==1.10.1+cu102 (from versions: none)
ERROR: No matching distribution found for torch==1.10.1+cu102

I think I know why the installation is not working.
The Pip command is trying to install a Module from the website listings (https://download.pytorch.org/whl/cu102/torch_stable.html 4)
The Modules pip is trying to install are
torch==1.10.1+cu102
torchvision==0.11.2+cu102
torchaudio===0.10.1+cu102

the Modules are Non-existing, not on the list that is why the errors happened.
the Owners need to update this problem

Context:
The pip wheels exist at the linked location (e.g. here 5 is the Python3.9 pip wheel for Linux) and your posted command also works fine for me:
pip install torch==1.10.1+cu102 torchvision==0.11.2+cu102 torchaudio===0.10.1+cu102 -f https://download.pytorch.org/whl/cu102/torch_stable.html
Looking in links: https://download.pytorch.org/whl/cu102/torch_stable.html
Collecting torch==1.10.1+cu102
  Downloading https://download.pytorch.org/whl/cu102/torch-1.10.1%2Bcu102-cp38-cp38-linux_x86_64.whl (881.9 MB)
     |████████████████████████████████| 881.9 MB 34 kB/s
Collecting torchvision==0.11.2+cu102
  Downloading https://download.pytorch.org/whl/cu102/torchvision-0.11.2%2Bcu102-cp38-cp38-linux_x86_64.whl (23.2 MB)
     |████████████████████████████████| 23.2 MB 12.5 MB/s
Collecting torchaudio===0.10.1+cu102
  Downloading https://download.pytorch.org/whl/cu102/torchaudio-0.10.1%2Bcu102-cp38-cp38-linux_x86_64.whl (2.9 MB)
     |████████████████████████████████| 2.9 MB 14.4 MB/s
Requirement already satisfied: typing-extensions in ./miniforge3/envs/tmp/lib/python3.8/site-packages (from torch==1.10.1+cu102) (4.0.0)
Collecting pillow!=8.3.0,>=5.3.0
  Using cached Pillow-8.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting numpy
  Using cached numpy-1.21.5-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Installing collected packages: torch, pillow, numpy, torchvision, torchaudio
  Attempting uninstall: torch
    Found existing installation: torch 1.10.0
    Uninstalling torch-1.10.0:
      Successfully uninstalled torch-1.10.0
  Attempting uninstall: torchaudio
    Found existing installation: torchaudio 0.10.0
    Uninstalling torchaudio-0.10.0:
      Successfully uninstalled torchaudio-0.10.0
Successfully installed numpy-1.21.5 pillow-8.4.0 torch-1.10.1+cu102 torchaudio-0.10.1+cu102 torchvision-0.11.2+cu102

I don’t know if you might be facing a Python version mismatch (supported versions are 3.6-3.9), are using a 32bit Python installation, or outdated pip versions, which can’t find the wheels.


Does torch.utils.data.random_split split every class equally?

Question:
So I have a directory with subdirectories, each subdirectory is a class.
+Directory
--+class1
--+class2
... etc

If I load them using torchvision.datasets.ImageFolder, and then split into training and testing like,
train_size = int(0.8*len(dataset_total))
test_size = len(dataset_total) - train_size
train_dataset,test_dataset = torch.utils.data.random_split(dataset_total,[train_size,test_size])

and then make a train_loader and test_loader for the train_dataset and test_dataset.
Could I expect an equal division of data from all classes? What is the best practice here to send data to network?
Thanks!

Context:
It is less likely to evenly split the dataset into two. It simply creates a list of permuted indices to sample the dataset into subsets, which doesn’t care about the classes. Say if random falls back to sequential, other classes are doom lost.
Check link 77 for original code.