akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Get the grad in the next epoch

rui_zhou: <p>Hi, how to use the grad of the first epoch in the next epoch</p>
<pre><code class="lang-auto">optimizer.zero_grad()
        feat_s,train_pred = model(train_data)
        feat_s[0].retain_grad() 
        grad = feat_s[0].grad
        if grad is not None:
            print (grad)
loss.backward()
optimizer.step()
</code></pre>
Questions about c++17.h in pytorch source code

zephyrrrr: <p>In the source code, it describes that gcc 4.8 doesn’t support <code>std::to_string</code> . And you guys implemented it.</p>
<pre><code class="lang-auto">namespace detail {
class DummyClassForToString final {};
} // namespace detail
} // namespace guts
} // namespace c10
namespace std {
// We use SFINAE to detect if std::to_string exists for a type, but that only
// works if the function name is defined. So let's define a std::to_string for a
// dummy type. If you're getting an error here saying that this overload doesn't
// match your std::to_string() call, then you're calling std::to_string() but
// should be calling c10::guts::to_string().
inline std::string to_string(c10::guts::detail::DummyClassForToString) {
  return "";
}

} // namespace std
namespace c10 {
namespace guts {
namespace detail {

template &lt;class T, class Enable = void&gt;
struct to_string_ final {
  static std::string call(T value) {
    std::ostringstream str;
    str &lt;&lt; value;
    return str.str();
  }
};
// If a std::to_string exists, use that instead
template &lt;class T&gt;
struct to_string_&lt;T, void_t&lt;decltype(std::to_string(std::declval&lt;T&gt;()))&gt;&gt;
    final {
  static std::string call(T value) {
    return std::to_string(value);
  }
};
} // namespace detail
template &lt;class T&gt;
inline std::string to_string(T value) {
  return detail::to_string_&lt;T&gt;::call(value);
}
</code></pre>
<p>You guys implemented the C++17 feature <code>void_t</code> but parameter packs was introduced in C++11.</p>
<p>So is there a problem?</p>
Weighted Class-Wise Product

Sayyed_Ali_Mousavi: <p>Hello everyone.<br>
Let A be a tensor of shape 64*10. Each row corresponds to the output of our model on a single image and we know the ground label of all images. I want to multiply each row by weight according to its label. For example, I want to multiply all rows with labels 1 by 1.5, all rows with labels 2 by 0.25, and so on. Finally, I want a tensor with the same shape 64*10.</p>
<p>What is vectorized efficient way to do this in PyTorch. Thanks a lot.</p>

ParGG: <p>Generate your weighted vector of shape 1x10, and then multiply it element-wise with your label tensor. Multiply then the result by the output of your model.</p>
<pre><code class="lang-python">wighted_mask = torch.tensor(weights)
weighted_output = output * (lables * weighted_mask).sum(dim= 1)
</code></pre>
<p>Just be careful about the dimensionality of the tensors, depending on that you might need to deal with broadcasting or not.</p>

Sayyed_Ali_Mousavi: <p>Maybe I didn’t express my question well; I think there is something wrong. When I print “labels * weighted_mask”, the weights are multiplied by the label number. This is not what I wanted. Suppose the first row was [1, 0.5, 2, 1.5] and its label was 2. I want to multiply all rows with labels 2 by 5. So this row should be [5, 2.5, 10, 7.5].</p>

ParGG: <p>Hi, I was expecting the labels to be one-hot encoded. Let’s say:</p>
<pre><code class="lang-python">labels = [
          [0, 1, 0, 0]
         ]

weighted_mask = [
                 [1.5, 5, 3.2, 0.3]
                ]

# labels * weighted_mask
row_weights = (labels * weighted_mask).sum(dim= 1, keepdim= True) # [[5]]

wighted_output = output * row_weights # [[1, 0.5, 2, 1.5]] * [[5]] = [[5, 2.5, 10, 7.5]]
</code></pre>

Sayyed_Ali_Mousavi: <p>Hi, Thank you very much. It’s great and it works. Just a small typo in the last line it should be row_weights instead of weight.</p>

ParGG: <p>Edited <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>I am happy I could help you</p>
LSTM vs LSTMCell different results with same hidden states?

shivammehta007: <p>Let’s say I want to use nn.LSTM as an nn.LSTMCell I initialize these two modules</p>
<pre><code class="lang-python">lstm = nn.LSTM(5, 10, batch_first=True)
cell = nn.LSTMCell(5, 10)
</code></pre>
<p>I set their hidden and cell states as equal</p>
<pre><code class="lang-python">cell.weight_ih.data = lstm.weight_ih_l0.data
cell.weight_hh.data = lstm.weight_hh_l0.data
cell.bias_ih.data = lstm.bias_ih_l0.data
cell.bias_hh.data = lstm.bias_hh_l0.data
</code></pre>
<p>Initialize the hidden states and an input</p>
<pre><code class="lang-python">h, c = torch.zeros(1, 10), torch.zeros(1, 10)
inp = torch.randn(1, 2, 5)
</code></pre>
<p>Run this through lstmcell and lstm</p>
<pre><code class="lang-python"># Cell
h1, c1 = cell(inp[:, 0], (h, c))
h2, c2 = cell(inp[:, 1], (h1, c1))

# LSTM
o_l1 , (h_l1, c_l1) = lstm(inp, (h.unsqueeze(1), c.unsqueeze(1)))

# This all makes sense
assert torch.isclose(h_l1, h2).all()
assert torch.isclose(o_l1[:, 0], h1).all()
</code></pre>
<p>But now if I use the LSTM and pass the input with hidden states generated by the LSTMCell.</p>
<pre><code class="lang-python">o_l2, (h_l2, c_l2) = lstm(inp[:, 1].unsqueeze(1), (h1.unsqueeze(1), h1.unsqueeze(1)))

print(h_l2)
# tensor([[[ 0.0331, -0.0655, -0.0764,  0.0269, -0.1706, -0.0454,  0.0769,
#           0.0008, -0.0629, -0.1064]]], grad_fn=&lt;StackBackward0&gt;)
print(h2)
# tensor([[ 0.0454, -0.0672, -0.0577,  0.0335, -0.1670, -0.0498,  0.0721,  0.0143,
#         -0.0400, -0.1339]], grad_fn=&lt;MulBackward0&gt;)
</code></pre>
<p>h_l2 and h2 appear to be different.  Is there something wrong that I am doing? I was expecting it generates the same values.</p>
<p><a href="https://colab.research.google.com/drive/1FXQ7JwqFP98oXU78T_OZ9hqbbzXoDbf1?usp=sharing" rel="noopener nofollow ugc">Here is the colab link to this scenario</a>.</p>
<p>Thank you for the help.</p>

InnovArul: <aside class="quote no-group" data-username="shivammehta007" data-post="1" data-topic="157157">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shivammehta007/48/31230_2.png" class="avatar"> shivammehta007:</div>
<blockquote>
<pre><code class="lang-auto">o_l2, (h_l2, c_l2) = lstm(inp[:, 1].unsqueeze(1), (h1.unsqueeze(1), h1.unsqueeze(1)))
</code></pre>
</blockquote>
</aside>
<p>I think there is a typo. Instead of <code>c1</code>, you seem to pass <code>h1</code>?</p>

shivammehta007: <p>Ah! I am so sorry, for this mistake and thank you for pointing it out!</p>

InnovArul: <p>It happens <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> And thanks for giving reproducible code as well.</p>
Why changing the value of numpy affects original tensor?

ruka: <p>How does this work? Why would this assignment affect the original tensor?</p>
<pre><code class="lang-auto">aa = torch.tensor([1.,2.,3.])
bb = aa.numpy()
bb[0] = 99.
print(aa[0])   # &lt;--- 99
</code></pre>

thecho7: <p>Here’s a reference<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.numpy.html#torch.Tensor.numpy" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.numpy.html#torch.Tensor.numpy</a></p>
<p>aa and bb share the memory.</p>

AlphaBetaGamma96: <p>Python uses shallow copying, what you want is a hard copy so changing <code>bb</code> doesn’t affect <code>aa</code>. You’ll want to use <code>.clone()</code> before moving to numpy.</p>
<pre><code class="lang-auto">aa = torch.tensor([1.,2.,3.])
bb = aa.clone().numpy() #clone before numpy
bb[0] = 99.
print(aa[0])   # &lt;--- 1
</code></pre>

ruka: <p>That is very clear!  Thank you so much!!</p>
Multi-gpu training with custom dataset in pytorch-geometric

Ipap94: <p>Hello,</p>
<p>I created a custom dataset list exactly using your example at the end of this link: (<a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html" rel="nofollow noopener">https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html</a>)</p>
<p>It looks like this:<br>
data_list = [Data(…), …, Data(…)]</p>
<p>I then try to use DatalistLoader instead of Dataloader to perform multi-gpu training but they dont seem to be compatible. What is the way to do it based on the data I provide?</p>

seanco92: <p>Hi,<br>
I have an related issue and I didn’t find a solution.<br>
I have a custom dataset which inherits from torch_geometric Data.<br>
In the single-GPU version I use torch_geometric DataLoader with <strong>follow_batch</strong> argument - because the collate of my dataset is not trivial.</p>
<p>I want to convert my code to multi-GPU - so I tried to yse torch_geometric DataParallel with <strong>DataListLoader</strong>, but this loader doesn’t support follow_batch.<br>
What can I do?</p>
<p>Thanks.</p>
Why Pytorch(Python) doesn't bind LibTorch(C++)

111392: <p>I’m just curious why Pytorch doesn’t bind LibTorch.</p>
<p>I could find <a href="https://discuss.pytorch.org/t/how-are-python-bindings-created/46453">this post</a> which mentions about historical issues and compilation time.</p>
<p>Are there any other reasons to work purely Python instead of binding c++. I think binding reduce duplicated implementation and makes the process faster.</p>
How to create a graph data structure in PyTorch?

walid_sa: <p>I have a txt file has feature and coordinates for multiple rectangles (F, X<em>center</em> , Y<em>center</em> , width, height). I would like to create an undirected graph network where the nodes of which stand for the rectangle and the edges of which represent distance between the rectangle centers. I used KNN but I couldn’t represent a node and assign a feature for each node.</p>
<p><strong>txt file:</strong></p>
<pre><code class="lang-auto">
F(np array)     Xcenter       Ycenter    width     height
[23.4....]     0.568396      0.394130   0.176887  0.345912
[20.4....]     0.391509      0.393082   0.172170  0.327044
[18.1....]     0.717571      0.377358   0.119104  0.320755
[0.33....]     0.254717      0.373166   0.103774  0.299790
</code></pre>
<p><strong>the code:</strong></p>
<pre><code class="lang-auto">dataset = CustomDataset(transform= transforms.ToTensor())
for data in dataset:
  centroid, feature_vector, class_list = data
  data = Data(pos= centroid, x=feature_vector, num_classes=class_list) 
  transform =T.Compose([T.KNNGraph(k=3), T.Distance(norm=False)])
  data = transform(data)
  print(data)
</code></pre>
<p><strong>the output:</strong></p>
<blockquote>
<p>Data(x=<span class="chcklst-box fa fa-square-o fa-fw"></span>, y=[17], pos=[17, 2], edge_index=[2, 51], edge_attr=[51, 1])</p>
</blockquote>
Unsure about placement of non-maximum suppression (NMS) in code

SU801T: <p>I’m doing inference on a faster r-cnn model and have the following set up where I apply non-maximum suppression on some outputs (batch size is always 1):</p>
<pre><code class="lang-auto">model_test.eval()
with torch.no_grad():
     
   for images, targets in data_loader:
            images = list(img.to(device) for img in images)
            outputs = model_test(images)
            outputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]
            predictions = apply_nms(outputs[0], iou_thresh=0.1)
</code></pre>
<p>However, I just wanted to be sure what this line of code is doing and why it is necessary during evaluation:</p>
<pre><code class="lang-auto">outputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]
</code></pre>
<p>Can anyone provide me with a good explanation?</p>
<p>And finally is it ok for me to apply non-maximum suppression like how I have defined?</p>

InnovArul: <aside class="quote no-group" data-username="SU801T" data-post="1" data-topic="157115">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/48.png" class="avatar"> SU801T:</div>
<blockquote>
<p><code>outputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]</code></p>
</blockquote>
</aside>
<p>This seems to be moving data from <code>cuda</code> to <code>cpu</code>, which might be necessary to apply NMS.<br>
Probably NMS routines are typically run on the CPU.</p>

SU801T: <p>tbh I applied NMS there and was wondering if it is the correct place to put it. It appears so from my results but I want to make sure it’s not a wrong placement</p>

InnovArul: <p>As far as I could understand, the placement of NMS seems correct.</p>
I want to know about the opacus version, why there is a problem with using the official example, For example the link https://blog.openmined.org/differentially-private-deep-learning-using-opacus-in-20-lines-of-code/ 

jiuyu_yang: <p>Let’s say this is an example of a runtime error<a>Uploading: 1.png…</a>I would like to know, what is the correct version of Opacus, or is it for other reasons, thank you</p>

ffuuugor: <p>Hi <a class="mention" href="/u/jiuyu_yang">@jiuyu_yang</a><br>
I think your screenshot didn’t upload properly - can you please double check?</p>

ffuuugor: <p>FYI the post is from August 2020, so Opacus v0.9 should be the most appropriate</p>

jiuyu_yang: <p>Thank you, just like you said.</p>
Semantic Segmentation of input image (black and white) generated image target mask RGB 

Alex66: <p>Input images shape (512, 512). Target masks shape (512, 512, 3).</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, datasets, models

class SimDataset(Dataset):
  def __init__(self, input_images_list, target_masks_list, transform=None):
    self.input_images = input_images_list
    self.target_masks = target_masks_list
    self.transform = transform

  def __len__(self):
    return len(self.input_images)

  def __getitem__(self, idx):
    image = self.input_images[idx]
    mask = self.target_masks[idx]
    if self.transform:
      image = self.transform(image)

    return [image, mask]

trans = transforms.Compose([
  transforms.ToTensor(),
  transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) # imagenet
])


train_set = SimDataset(train_image_data_list, train_target_image_list, transform = trans)
val_set = SimDataset(val_image_data_list, val_target_image_list, transform = trans)

image_datasets = {'train': train_set, 'val': val_set}

batch_size = 3

dataloaders = {
  'train': DataLoader(train_set, batch_size=batch_size, num_workers=0),
  'val': DataLoader(val_set, batch_size=batch_size, num_workers=0)
}

inputs, masks = next(iter(dataloaders['train']))
</code></pre>
<p>I got this error</p>
<pre><code class="lang-auto">RuntimeError: output with shape [1, 512, 512] doesn't match the broadcast shape [3, 512, 512]
</code></pre>
<p>How to fix it?</p>

ptrblck: <p>Your <code>Normalize</code> transformation uses stats for 3 channels while your input tensors have a single channel only. Either repeat the channels or use a single value for the <code>Normalize</code> stats.</p>
Gloo timeout unboubd buffer

Antonia_Calvi: <p>Hello,<br>
I am running in a gloo timeout when I try to run gather object from the other processes:</p>
<pre><code class="lang-auto">mask_right = [None for _ in range(world_size)]
torch.distributed.all_gather_object(mask_right, all_predictions_tmp[rxn]['mask_right'])
    
</code></pre>
<p>The size of each <code>all_predictions_tmp[rxn]['mask_right']</code> is quite big - of 440076 instances so I am not sure whether this might be the cause of the timeout. The above lines work when I am working with smaller objects - so I am assuming the size is an issue. Although I somehow tried batching the size of the tensor that I wanted to gather but that still did not seem to work. Below is the full error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/ubuntu/NinaNotebooks/forward-synthesis/stages/model/all_predictions.py", line 393, in &lt;module&gt;
    mp.spawn(benchmark,
  File "/home/ubuntu/NinaNotebooks/forward-synthesis/.venv/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/ubuntu/NinaNotebooks/forward-synthesis/.venv/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/home/ubuntu/NinaNotebooks/forward-synthesis/.venv/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/ubuntu/NinaNotebooks/forward-synthesis/.venv/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/home/ubuntu/NinaNotebooks/forward-synthesis/stages/model/all_predictions.py", line 268, in benchmark
    torch.distributed.all_gather_object(mask_right, all_predictions_tmp[rxn]['mask_right'][i*batch_len:(i+1)*batch_len])
  File "/home/ubuntu/NinaNotebooks/forward-synthesis/.venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1657, in all_gather_object
    all_gather(object_size_list, local_size, group=group)
  File "/home/ubuntu/NinaNotebooks/forward-synthesis/.venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 2075, in all_gather
    work.wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:136] Timed out waiting 1800000ms for send operation to complete
</code></pre>
<p>Any help would be great, thank you!</p>
DistributedDataParallel with multiple losses, freezing some parameters at a time

toliveira: <p>Networks G and E are both used to calculate loss_A and loss_B.</p>
<p>I want to minimize loss_A by changing only G.parameters(), and minimize loss_B by changing only E. parameters().</p>
<p>The following works in serial mode and also with DataParallel.</p>
<pre><code class="lang-auto">for param in E.parameters():
    param.requires_grad = False

loss_A = calculate_loss_A_using_G_and_E()
loss_A.backward(retain_graph=True)

for param in E.parameters():
    param.requires_grad = True
for param in G.parameters():
    param.requires_grad = False

loss_B = calculate_loss_B_using_G_and_E()
loss_B.backward()

for param in G.parameters():
    param.requires_grad = True
</code></pre>
<p>However, when using DistributedDataParallel, the code above raises the <em>RuntimeError: Expected to mark a variable ready only once</em>. I suppose that this happens because on the first call to .backward E.parameters() are marked as ready even if requires_grad is False.</p>
<p>What can be done to use DistributedDataParallel in this scenario?</p>
Loading styleGAN2 for inference without GPU

Hamzeh: <p>I have trained a segmentation model (StyleGAN2) on my machine and want to use the inference in another machine that has not any GPU device.<br>
I tried to modify all imports to use CPU, but still I cannot load it on CPU.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/Desktop/SemGAN/inference.py", line 5, in &lt;module&gt;
    from models.encoder_model import ResEncoder
  File "/home/Desktop/SemGAN/models/encoder_model.py", line 27, in &lt;module&gt;
    from models.stylegan2_seg import EqualLinear, ConvLayer, ResBlock
  File "/home/Desktop/SemGAN/models/stylegan2_seg.py", line 11, in &lt;module&gt;
    from models.utils import *
  File "/home/Desktop/SemGAN/models/utils.py", line 12, in &lt;module&gt;
    from models.op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
  File "/home/Desktop/SemGAN/models/op/__init__.py", line 1, in &lt;module&gt;
    from .fused_act import FusedLeakyReLU, fused_leaky_relu
  File "/home/Desktop/SemGAN/models/op/fused_act.py", line 17, in &lt;module&gt;
    fused = load(
  File "/home/Desktop/land_monitoring/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1202, in load
    return _jit_compile(
  File "/home/Desktop/land_monitoring/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1425, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/Desktop/land_monitoring/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1514, in _write_ninja_file_and_build_library
    extra_ldflags = _prepare_ldflags(
  File "/home/Desktop/land_monitoring/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1622, in _prepare_ldflags
    extra_ldflags.append(f'-L{_join_cuda_home("lib64")}')
  File "/home/Desktop/land_monitoring/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2125, in _join_cuda_home
    raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

</code></pre>

ptrblck: <p>Based on the error message it seems a custom <code>CUDAExtension</code> is supposed to be built, which is failing in your setup. You could check which layer is using this extension and then check if you could replace it with a CPU-only version.</p>
Sliding window with repeated first element

Entropy: <p>Say I have a tensor like</p>
<pre><code class="lang-auto">test = torch.arange(16).resize(4, 4)
</code></pre>
<p>This should look like</p>
<pre><code class="lang-auto">tensor([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11],
        [12, 13, 14, 15]])
</code></pre>
<p>Is there a way to implement a fixed sliding window along the rows that outputs the same number of tensors as the number of rows (by repeating the first row as many times as necessary). In other words, I want to return a tensor that looks like the following (given a window of length 3):</p>
<pre><code class="lang-auto">tensor([[[ 0,  1,  2,  3], 
         [ 0,  1,  2,  3], 
         [ 0,  1,  2,  3]],
        [[ 0,  1,  2,  3], 
         [ 0,  1,  2,  3], 
         [ 4,  5,  6,  7]],
        [[ 0,  1,  2,  3], 
         [ 4,  5,  6,  7], 
         [ 8,  9, 10, 11]],
        [[ 4,  5,  6,  7], 
         [ 8,  9, 10, 11], 
         [12, 13, 14, 15]]])
</code></pre>
<p>Currently, I just use regular python for loops and it works fine. However, I was wondering if this is solvable using built-in pytorch functions.</p>

ptrblck: <p>You should be able to <code>expand</code> the tensor and <code>unfold</code> it, but it depends on your actual use case if this would be faster than your loop as it also could add an additional memory overhead:</p>
<pre><code class="lang-python">test = torch.arange(16).view(4, 4)
test = torch.cat((test[0].unsqueeze(0).expand(3, -1), test[1:]))

out = test.unfold(0, 3, 1).permute(0, 2, 1)
print(out)
# tensor([[[ 0,  1,  2,  3],
#          [ 0,  1,  2,  3],
#          [ 0,  1,  2,  3]],

#         [[ 0,  1,  2,  3],
#          [ 0,  1,  2,  3],
#          [ 4,  5,  6,  7]],

#         [[ 0,  1,  2,  3],
#          [ 4,  5,  6,  7],
#          [ 8,  9, 10, 11]],

#         [[ 4,  5,  6,  7],
#          [ 8,  9, 10, 11],
#          [12, 13, 14, 15]]])
</code></pre>
Reduction Factor in Ray tune

NearsightedCV: <p>How do I use this parameter? When I run I do not get any halving at all at a setting of 2. My tuning works through the trials sequentially. I thought it would try and only carry forward the most promising combinations of hyper parameters.</p>
<p>Am I missing something, documentation seems thin on this setting IMHO</p>
<p>Thanks in advance,</p>

yqchau: <p>Hey, I was facing this problem as well and still am not really sure what this param was supposed to be exactly due to the very limited docs.</p>
<p>This is what I found from <a href="https://docs.ray.io/en/latest/tune/faq.html#id10" rel="noopener nofollow ugc">ray tune faqs</a>, hope it helps.</p>
<p>‘reduction_factor=4` means that only 25% of all trials are kept each time they are reduced.’</p>

NearsightedCV: <p>It does thanks for that clarification!</p>
Deep Supervision - Combined Loss Value

NearsightedCV: <p>Hi,</p>
<p>I am trying to set up a UNet3+ network and I have been using <a href="https://arxiv.org/abs/2004.08790" rel="noopener nofollow ugc">this paper</a> as my baseline. One thing that confuses me is how to aggregate the loss values from the hidden layer supervision with the output later. All the literature uses some weighted average but it is not clear to me how the weights are calculated.</p>
<p>To add to my confusion I found the following <a href="https://arxiv.org/abs/1611.04849" rel="noopener nofollow ugc">paper</a>. Which also has a git hub and this is what they use for the total loss</p>
<pre><code class="lang-auto"># loss function: seven probability map --- 6 scale + 1 fuse
class Loss(nn.Module):
    def __init__(self, weight=[1.0] * 7):
        super(Loss, self).__init__()
        self.weight = weight

    def forward(self, x_list, label):
        loss = self.weight[0] * F.binary_cross_entropy(x_list[0], label)
        for i, x in enumerate(x_list[1:]):
            loss += self.weight[i + 1] * F.binary_cross_entropy(x, label)
        return loss
</code></pre>
<p>This makes no sense to me, especially since I plan to use Dice loss where I would get values of .9 .88 etc… My DiceLoss needs to stay within [0,1] but with equal weights that probably will not happen.</p>
<p>Can anyone chime in how to do a overall loss value when you are using deep supervision?</p>
<p>Thanks</p>
Can't pass subset into model

rjsdebug: <p>I created a subset of data</p>
<pre><code class="lang-auto">subset = torch.utils.data.Subset(train_dataset,newIndices)
</code></pre>
<p>Here, newIndices is just a list of indices to use in my subset.<br>
I want to analyze the logits I get on a forward pass, so I need to pass all the data of the subset into my model at once.</p>
<pre><code class="lang-auto">images=subset.data.type(torch.FloatTensor)
logits = model(images)
</code></pre>
<p>This gives me the error that Subset has no attribute ‘data’, and I get the same issue with ‘targets’. Is there a simple fix to this, to access all the <em>subset</em> data/targets and not the original set?</p>

ptrblck: <p>You would need to iterate the <code>Subset</code> and store the data/targets in e.g. new tensors or a <code>list</code>.<br>
The internal <code>Dataset</code> could lazily load the samples in its <code>__getitem__</code> so you won’t be necessarily able to access a <code>.data</code> or <code>.target</code> attribute as it might not exist until you actually load the sample.<br>
However, if you know that your <code>Dataset</code> pre-loading the data and indeed uses these internal attributes, you might be able to index them via <code>newIndices</code> assuming both are tensors.</p>

rjsdebug: <p>I ended up creating a dataloader for the subset and putting batch size as the entire subset size… is this bad practice? Seemed to get the job done.</p>

ptrblck: <p>No, your approach sounds fine.</p>
Questions ondynamic_axes of torchl.onnx.export()

hitbuyi: <p>here I have another question, more detailed on dynamic axes , my model is</p>
<pre><code>import torch
import numpy as np
model = torch.nn.Linear(3, 2)

w, b = model.state_dict()['weight'], model.state_dict()['bias']
# w, b = model.weight, model.bias
with torch.no_grad():
    w.copy_(torch.tensor([[1., 2., 3.], [4., 5., 6.]]))
    b.copy_(torch.tensor([-1., -2.]))
model.cuda()

# Export to ONNX
model.eval()
x = torch.randn(1, 3, requires_grad=True, device='cuda')

#Export model2.onnx with dynamic batch_size
print('\nExporting model2.onnx ...')
torch.onnx.export(model,
                  x,
                  'model2.onnx',
                  #opset_version=11,
                  verbose=True,
                  export_params=True,
                  input_names=['input'],
                  output_names=['output'],
                  dynamic_axes={'input': {0: 'batch_size'},  # variable lenght axes
                                'output': {0: 'batch_size'}} # change from 0 to 1
                  )
</code></pre>
<p>’</p>
<p>if I set</p>
<pre><code>dynamic_axes={'input': {0: 'batch_size'}, 
              'output': {0: 'batch_size'}} 
</code></pre>
<p>the input is [bacth_size, 3], the output is [ batch_size,2], it is ok</p>
<p>if I set</p>
<pre><code>dynamic_axes={'input': {1: 'batch_size'}, 
              'output': {1: 'batch_size'}} 
</code></pre>
<p>the input is [1,bacth_size], the output is [1, batch_size], it is not what  I want<br>
input: [3,batch_size], output: [2, batch_size]</p>
<p>if I set</p>
<pre><code>   dynamic_axes={'input': {0: 'batch_size',1:'time'}, 
                 'output': {0: 'batch_size',1:'time'}}      
</code></pre>
<p>the input is [bacth_size, time], the output is [ batch_size,time], it is not what I want<br>
input [batch_size,time, 3] ,output [batch,time, 2],</p>
<p>1, how to write correct dynamic_axes?<br>
2, for</p>
<pre><code>       dynamic_axes={'input': {0: 'batch_size'}, 
                     'output': {0: 'batch_size'}} 
</code></pre>
<p>what’s the meaning of ‘0’? if I set 0 to 1, 2,3, …, what happened?</p>