akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the best way to load niftii images (.nii) using pytorch dataloader?

username: ru6928
cooked: <p>What is the best way to load niftii images (.nii) using pytorch dataloader?</p>

username: yw_tt
cooked: <p>This question has nothing to do with pytorch. You can use nibabel to load .nii file.</p>

username: Aliktk
cooked: <p>if someone using pytorch than it would a problem for beginner that how they load the nii format image into their memory and further processed using pytorch method. Nibable is library which load all kind of 3D and 4D scans but how this will load using pytroch syntax and further more how the whole directory can be read using pytorch</p>

username: Aliktk
cooked: <p>where is the code for loading all the images 3D from the directory&gt;&gt;??</p>

username: Balamurali_M
cooked: <p>Nibabel will give numpy. It can be converted to torch. Same procedure followed for classification can be adapted.</p>

username: Aliktk
cooked: <p>yeah bro nibabel proving this but I want to load all the <code>nii</code> files into my memory in an iterative way but how?<br>
what procedure i should follow?</p>

username: Balamurali_M
cooked: <p>Save the nib file to h5 file. Create a list with [(h5_file_name,slice_no),(h5_file_name,slice_no)]. Iterate this using dataloader, during runtime read the h5 file again and take out the slice. Let me know if you understand this, else i will share the pseudocode.</p>
Two different networks, different inputs, how to train them simultaneously?

username: skyraker
cooked: <p>Hello, I am new to Pytorch, I need to train two different networks, and their inputs are different. I need to multiply their outputs and train them simultaneously, how to do it ?</p>

username: skyraker
cooked: <p>inputs of net1 works, but inputs of net2 failed. The  input data of net2 is a np.array generated by a function I def, how to send it to net2</p>

username: ptrblck
cooked: <p>You could transform the numpy array to a tensor via:</p>
<pre><code class="lang-python">data = torch.from_numpy(array)
</code></pre>
<p>Depending on the <code>dtype</code> of the numpy array you might need to transform the tensor to a <code>FloatTensor</code> using:</p>
<pre><code class="lang-python">data = data.float()
</code></pre>

username: skyraker
cooked: <p>data = torch.from_numpy(array)<br>
data = data.float()<br>
data = data.cuda()<br>
These I have done, but when y = net2(data)<br>
got y is NoneType</p>

username: ptrblck
cooked: <p>Could you post the definition of <code>net2</code>?</p>

username: skyraker
cooked: <p>well a strange net<br>
class Matrix(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Matrix,self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(1024,1024,3)<br>
self.conv2 = nn.Conv2d(1024,512,5)<br>
self.conv3 = nn.Conv2d(512,256,5)<br>
self.conv4 = nn.Conv2d(256,128,3)<br>
self.conv5 = nn.Conv2d(128,64,5)<br>
self.pool = nn.AvgPool2d(2,2)<br>
self.fc1 = nn.Linear(64,32)<br>
self.fc2 = nn.Linear(32,10)</p>
<pre><code>def forward(self,x):
    x = self.pool(F.relu(self.conv1(x)))
    x = F.relu(self.conv2(x))
    x = F.relu(self.conv3(x))
    x = F.relu(self.conv4(x))
    x = F.relu(self.conv5(x))
    x = x.view(-1,64)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
</code></pre>
<p>net2 = Matrix()<br>
and input data:<br>
h = hadamard(1024)<br>
h = h.reshape([1,1024,32,32])<br>
h = torch.from_numpy(h)<br>
h = h.float()<br>
h = h.cuda()<br>
y = net2(h)<br>
And I find I have not def a optimizer of net2, is this the reason?</p>

username: ptrblck
cooked: <p>Your <code>forward</code> method seems to be missing the <code>return</code> statement, so you should add:</p>
<pre><code class="lang-python">return x
</code></pre>
<p>at the end of it.</p>

username: skyraker
cooked: <p>OMG!! Thx so much<br>
so ridiculous i am, i need to update my glasses</p>
Get layer's output from nn.Sequential

username: sulin_nwe
cooked: <p>Hello. Below is my code sample. I would like to ask how can I do to get the output from second dropout of decoder?</p>
<pre><code class="lang-auto">class AutoEncoder(nn.Module):
    
    def __init__(self, encoder_dim, decoder_dim, drops=[0.7, 0.5, 0.5], td_units=100):
        super(AutoEncoder, self).__init__()
        
        self.encoder_dim = encoder_dim
        self.decoder_dim = decoder_dim
        self.td_units = td_units
        self.gru = nn.GRU(encoder_dim, decoder_dim, bidirectional=True)
        self.drop0  = drops[0]
        self.drop1  = drops[1]
        self.r_drop = drops[2]
        #encoder
        self.encoder = nn.Sequential(
            nn.Linear(self.encoder_dim, 3*self.td_units),
            nn.Tanh(),
            nn.Dropout(p = self.drop0),
            nn.Linear(3*self.td_units, 2*self.td_units),
            nn.Tanh(),
            nn.Dropout(p = self.drop0),
            nn.Linear(2*self.td_units, 1*self.td_units),
            nn.Tanh(),
            nn.Dropout(p = self.drop0)
        )
        
        #decoder
        self.decoder = nn.Sequential(
            nn.Linear(self.td_units, 2*self.td_units),
            nn.Tanh(),
            nn.Dropout(p = self.drop0),
            nn.Linear(2*self.td_units, 3*self.td_units),
            nn.Tanh(),
            nn.Dropout(p = self.drop0),
            nn.Linear(3*self.td_units, self.decoder_dim),
            nn.Tanh())

        
    def forward(self,x):
        x = self.encoder(x)
        y = self.decoder(x) 
        return y
</code></pre>
<p>Any help is fine. Thanks.</p>

username: ptrblck
cooked: <p>You could use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>

username: sulin_nwe
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for your reply.</p>
Modify adding gradients in backward()

username: amirhf
cooked: <p>my model has two sets of parameters <code>a</code> and <code>b</code> which require gradients. I have two different loss functions as:</p>
<pre><code class="lang-auto">loss1 = function(a, b)
loss2 = function(b)
</code></pre>
<p>and the total loss is</p>
<pre><code class="lang-auto">total_loss = loss1 + loss2
</code></pre>
<p>Assuming:</p>
<ul>
<li>
<code>loss1</code> calculates <code>b1.grad</code>
</li>
<li>
<code>loss2</code> calculates <code>b2.grad</code>,<br>
then the total <code>b.grad</code> calculated by <code>total_loss.backward()</code> is <code>b.grad = b1.grad + b2.grad</code>.</li>
</ul>
<p>My goal is to modify <code>b1.grad</code> coming from <code>loss1</code> and <code>b2.grad</code> comig from <code>loss2</code> and then add them together as backward gradients for <code>b</code>. Currently when I do <code>total_loss.bakward()</code>, it already gives me the accumulated gradients for <code>b.grad=b1.grad + b2.grad</code>. How can I access and modify each individual <code>b1.grad</code> and <code>b2.grad</code> so the <code>total_loss.backward()</code> returns the modified and added <code>b1.grad</code> and <code>b2.grad</code>?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You will have to create new intermediary Tensors like the following.<br>
Also hooks are a good way to modify these gradients.</p>
<pre><code class="lang-auto">b = # some tensor that requires_grad
b1 = b.view_as(b) # returns a new Tensor that is the same as b
def modif_for_grad1(grad):
  return 3 * grad + 42
b1.register_hook(modif_for_grad1)
b2 = b.view_as(b) # returns a new Tensor that is the same as b
b2.register_hook(modif_for_grad2)

loss1 = function(a, b1)
loss2 = function(b2)
</code></pre>

username: amirhf
cooked: <p>Thank you for this nice workaround. However to get <code>b.grad = b1.grad + b2.grad</code> I managed to do a hacky way as below:</p>
<pre><code class="lang-auto">b_grad = 0
for param in total_loss.parameters():
    if param.grad is not None:
      b_grad += param.grad
  
  for param in total_loss.parameters():
    if param.grad is not None:
      param.grad = b_grad
</code></pre>
<p>Is there a better way to do this? Because, when you make copies of the original parameter (intermediate ones) I should their gradients together and replace them.</p>
Train model parameters and scaler parameters together

username: XW324
cooked: <p>Suppose I have a model to train, and I also do some additional post processing on the output of the model. The post processing is parameterized by a scaler tensor which I also want to train. How should I tell the optimizer to train both the model and the scaler?</p>

username: ptrblck
cooked: <p>If your post processing method uses <code>nn.Parameters</code>, you could simply pass them additionally to the optimizer as described <a href="https://discuss.pytorch.org/t/giving-multiple-parameters-in-optimizer/869/2">here</a>.</p>
How to convert nms to torchscript?

username: dalalaa
cooked: <p>I want to convert a detection model to torchscript, but there are some control/loop in nms code, how can I replace them with operations supported in torchscript?</p>

username: driazati
cooked: <p>You can use normal Python control flow in <a href="https://pytorch.org/docs/stable/jit.html#torch.jit.script" rel="nofollow noopener">script mode</a> (in contrast to tracing which it sounds like you’re talking about). Are you running into any specific issues?</p>

username: dalalaa
cooked: <p>Thank you, script mode solved my problem.</p>
Truck visit prediction engine

username: DaniG2k
cooked: <p>Hello all <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=9" title=":wave:" class="emoji" alt=":wave:"> nice to meet you</p>
<p>I need to develop a tool that, given a customer id, predicts how many truck visits they will receive in the next hour.</p>
<p>I have several years worth of historical data for customers, so I think that a Deep Learning mechanism might be well suited for the task and I’d like to use PyTorch to build it. I have only taken the Udacity course on PyTorch so never attempted a real-world application with it and would really really appreciate some guidance.</p>
<p>I think that for now, I won’t be using many data points. Possibly only customer id and historical truck visits for every hour of the day. I can’t really think of how else I’d make such a prediction (weather maybe? But I don’t have the weather data at hand so probably best to skip for the time being)</p>
<p>I have several questions in this regard:</p>
<ul>
<li>
<p>Am I right to assume an LSTM is best suited for this time-based prediction scenario?</p>
</li>
<li>
<p>If I keep getting new truck visit data, how can I include the new data into the prediction engine? Do I need to retrain every so often or are there techniques to include the incoming data in the next prediction as it comes?</p>
</li>
<li>
<p>What would you use as a general strategy to solve this problem? Are there some tutorials out there that you could link me to that try to do something similar?</p>
</li>
</ul>
<p>Again, any help is <strong>much</strong> appreciated.</p>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: SmithyN
cooked: <p>Learned a lot reading this. Thanks a lot. As someone who works with tow trucks and tracks clients and whatnot this is important to know and I love you for sharing it! To learn what I’m talking about <a href="https://www.hamiltontowtruckservice.com/" rel="nofollow noopener">click here</a>. I love the work I do and hopefully you just made it better! We’ll see!</p>
Worse performance using PyTorch Mobile than on GPU/CPU

username: IsaacBerman
cooked: <p>I have a custom model that is a variation on YOLOv3, to test the results, I have asserted that the inputTensor on device is the same as that I am loading on the computer. The output (which has detections and classifications) is giving near identical object and class confidences. However, the locations (x,y,w,h) are slightly off. Is this expected behaviour? Do you know if there is anything in particular I should investigate in my model or the trace of my model?</p>

username: IvanKobzarev
cooked: <p>Hello <a class="mention" href="/u/isaacberman">@IsaacBerman</a>,</p>
<p>Do you use completely the same model on mobile and desktop, without any quantization on mobile?</p>
<p>If you already debugged it, maybe you have some ideas which operator/layer is producing different results?</p>
<p>Could you please dump all operators that your model use, here are steps how to do it:</p>
<p>First we need <code>dump_operator_names</code> executable which can be built with command:</p>
<pre><code class="lang-auto">BUILD_BINARIES=1 python setup.py build
</code></pre>
<p>After succesfull build you can find it in path like <code>./build/lib.linux-x86_64-3.7/torch/bin/dump_operator_names</code>.</p>
<p>It depends on <code>libcaffe2_observers.so</code>, <code>libtorch.so</code>, <code>libc10.so</code>, so they should be either located in the same folder or be installed on the system.</p>
<pre><code class="lang-auto">
mkdir tmp
cp ./build/lib.linux-x86_64-3.7/torch/bin/dumpop_operator_names tmp/
cp ./build/lib.linux-x86_64-3.7/torch/lib/libcaffe2_observers.so tmp
cp ./build/lib.linux-x86_64-3.7/torch/lib/libtorch.so tmp
cp ./build/lib.linux-x86_64-3.7/torch/lib/libc10.so tmp
</code></pre>
<p>After that you can run it, specifing model and output file.</p>
<pre><code class="lang-auto">./dump_operator_names --model=model.pt --output=model_ops.yaml
</code></pre>

username: IsaacBerman
cooked: <p>Hi Ivan,</p>
<p>Thanks for your response. I ended up solving the error. It turned out it was a problem with the tracing – see here: <a href="https://discuss.pytorch.org/t/torch-jit-trace-only-works-on-example-input/62478" class="inline-onebox">Torch.jit.trace() only works on example input?</a></p>
Equivalence of torch.tensor([[1]]) and torch.tensor(1)

username: marcindulak
cooked: <p>I’ve encountered the following, and looking for an explanation</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)
1.3.1
a = torch.tensor([[1]])
b = torch.tensor(1)
print(a.shape, b.shape)
torch.Size([1, 1]), torch.Size([])
print(a == b)
tensor([[True]])
</code></pre>

username: richard
cooked: <p>If you want to detect equivalence of sizes and shape, <a href="https://pytorch.org/docs/master/torch.html?highlight=equal#torch.equal">torch.equal</a> is the right thing to use.</p>
<p>In your example above, <code>a == b</code>, we are broadcasting <code>a</code> and <code>b</code> and then performing the operation: <a href="https://pytorch.org/docs/stable/notes/broadcasting.html">https://pytorch.org/docs/stable/notes/broadcasting.html</a>.</p>
<p>The steps look roughly like the following:</p>
<ol>
<li>a has size [1, 1], b has size []</li>
<li>If the sizes are broadcastable (they are), we make it so that they have the same size before evaluation. So a_broadcasted has size [1, 1], b_broadcasted has size [1, 1]</li>
<li>We compare a_broadcasted with b_broadcasted. This produces a tensor with size [1, 1] that has one element True.</li>
</ol>

username: marcindulak
cooked: <p>Understood. Is there a proper name for a torch.Size([])?</p>

username: marcindulak
cooked: <p>OK, it looks it’s called a scalar <a href="https://discuss.pytorch.org/t/what-does-torch-size-0-means">https://discuss.pytorch.org/t/what-does-torch-size-0-means</a>.</p>

username: kartikey_shaurya
cooked: <p>thanks for answering</p>
Efficiency repeat_interleave

username: Diego999
cooked: <p>Hi guys.</p>
<p>I’ve seen that the new function repeat_interleave has been released recently <a href="https://pytorch.org/docs/stable/torch.html?highlight=repeat_interleave#torch.repeat_interleave" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html?highlight=repeat_interleave#torch.repeat_interleave</a> .</p>
<p>However, it is unclear how this function works internally:</p>
<ul>
<li>is it more similar to torch.expand or torch.repeat ?</li>
<li>is it preferable to copy data from CPU to GPU instead of using repeat_interleave on GPU ?</li>
</ul>
<p>Thank you for your help !</p>
Slow pytorch inference speed

username: Kunal_Goyal
cooked: <p>Recently I was experimenting with different object detection techniques faster than <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/" rel="nofollow noopener">mobilnet SSD v1 tensorflow api</a>. I came across similar implementation in <a href="https://github.com/qfgaohao/pytorch-ssd" rel="nofollow noopener">pytorch</a>. But while execution I found that time taken for pytorch implementation(0.1 sec) is more than to that with tensorflow OD(0.06 sec).  I think all frameworks should give similar results for same algorithm.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You will need to give more background on the hardware you’re using to get a better idea.<br>
Also this is a third party implementation so it might not be very well optimized. compared to the builtin version provided by tensorflow.</p>

username: Kunal_Goyal
cooked: <p>Thanks for your reply, How can I optimize PyTorch implementation? Can you please share some references I can look at to get the more satisfying results? Also, is it true that a fully optimized code in PyTorch will give the same results to that with Tensorflow’s OD?</p>
<p>PS: Also I think the main purpose of these frameworks is that we should more focus on implementation rather than optimization</p>

username: albanD
cooked: <blockquote>
<p>How can I optimize PyTorch implementation?</p>
</blockquote>
<p>I am not familiar with this particular implementation.<br>
You can look online for “pytorch in production” and you should get all the materials you need.</p>
<blockquote>
<p>Also, is it true that a fully optimized code in PyTorch will give the same results to that with Tensorflow’s OD?</p>
</blockquote>
<p>This is not guaranteed. They will most likely be very similar because all the heavy lifting is done by cudnn and BLAS libraries anyway. But there might be some differences for less traditional architectures.</p>
<blockquote>
<p>Also I think the main purpose of these frameworks is that we should more focus on implementation rather than optimization</p>
</blockquote>
<p>I think this is a difference between pytorch and Tensorflow yes. While you can find many many things internalized and optimized in Tensorflow, it’s not the case for pytorch. We only add the very widely used methods to the core. You can find a bit more in the specialized libraries like torchvision or torchtext.</p>
Libtorch program crashes at startup

username: Vishaal_Selvaraj
cooked: <p>I have been quite eager to try out libtorch C++, but I can’t seem to get it to work properly.<br>
I am using cmake 3.12.0-rc1 and MSVC-15 compiler to build the program but the application seems to crash at start.<br>
I am using libtorch-cpu-only release version.<br>
Although the build is successful, there are lots of warnings during the build process.</p>
<p>You can find the build log <a href="https://pastebin.com/7ijvFPaY" rel="nofollow noopener">here</a></p>
<p><strong>main.cpp</strong> :</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
    #include &lt;torch/torch.h&gt;
    
    int main() {
        std::cout &lt;&lt; "Hello libtorch c++!" &lt;&lt; std::endl;
        torch::Tensor tensor = torch::rand({2, 3});
        std::cout &lt;&lt; tensor &lt;&lt; std::endl;
        system("pause");
        return 0;
    }
</code></pre>
<p><strong>CMakelists.txt</strong> :</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.12)
    project(sampleC)
    # set(CMAKE_CXX_STANDARD 17)
    set(Torch_DIR "E:/libtorch/share/cmake/Torch")
    find_package(Torch REQUIRED)
    
    add_executable(sampleC main.cpp)
    target_link_libraries(sampleC "${TORCH_LIBRARIES}")
    set_property(TARGET sampleC PROPERTY CXX_STANDARD 11)
</code></pre>
<p><strong>Program output:</strong></p>
<pre><code class="lang-auto"> F:\sampleC\cmake-build-debug\sampleC.exe
    
    Process finished with exit code -1073741515 (0xC0000135)
</code></pre>
<p>I have been trying to solve this for two days now. But I had no luck.<br>
Any help is appreciated.</p>

username: tom
cooked: <aside class="quote no-group" data-username="Vishaal_Selvaraj" data-post="1" data-topic="50502">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vishaal_selvaraj/48/14054_2.png" class="avatar"> Vishaal_Selvaraj:</div>
<blockquote>
<p>Process finished with exit code -1073741515 (0xC0000135)</p>
</blockquote>
</aside>
<p>Googling the exit code turns up <code>STATUS_DLL_NOT_FOUND</code> as the translation. You probably need to put the torch dll(s) in the search path (PATH?) or in your program folder or so.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Vishaal_Selvaraj
cooked: <p>Yeah… that was the problem…now it’s working …thanks…</p>

username: sfwindowtint
cooked: <p>Thanks, this was very helpful.  My gosh this industry is so much more complicated compared to when I <a href="http://www.sfwindowtint.com" rel="nofollow noopener">tinted windows</a>.</p>
Creating bundles error

username: burning2
cooked: <p>I’m trying to follow the instructions in <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md" class="inline-onebox" rel="noopener nofollow ugc">glow/docs/AOT.md at master · pytorch/glow · GitHub</a></p>
<p>When I reached this step, something went wrong，</p>
<ul>
<li><code>cmake -G ninja &lt;other cmake flags&gt; -DGLOW_WITH_BUNDLES=ON -DGLOW_WITH_CPU=ON</code></li>
<li><code>ninja RunResNet50Bundle</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/366605ff842f0e5e31c0c5c5280d68944a5bfaa9.png" data-download-href="https://discuss.pytorch.org/uploads/default/366605ff842f0e5e31c0c5c5280d68944a5bfaa9" title="%E6%8D%95%E8%8E%B7"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/366605ff842f0e5e31c0c5c5280d68944a5bfaa9_2_690x105.png" alt="%E6%8D%95%E8%8E%B7" data-base62-sha1="7LemN2dUBrYCq4iy9stw5tYvtQR" width="690" height="105" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/366605ff842f0e5e31c0c5c5280d68944a5bfaa9_2_690x105.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/366605ff842f0e5e31c0c5c5280d68944a5bfaa9_2_1035x157.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/6/366605ff842f0e5e31c0c5c5280d68944a5bfaa9.png 2x" data-dominant-color="4A2A40"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%8D%95%E8%8E%B7</span><span class="informations">1046×160 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Error creating directory “/bundles/resnet50”</li>
</ul>
<p>Is this because I am running the command in the wrong directory?</p>

username: jfix
cooked: <p>Sorry for the delayed response! It looks like <code>GLOW_BINARY_DIR</code> is not being set correctly. I believe it needs to be set in your environment if you’re not building the whole project via <code>-DGLOW_WITH_BUNDLES=ON</code>. In your case it’s probably <code>/home/liss/projects/glow/</code>.</p>
[Pytorch]Hello, I'd like to ask you about data preprocessing of cifar-10

username: cclover
cooked: <p>Hello, I’d like to ask you about data preprocessing of cifar-10<br>
In the pre-processing of data, we need to convert the data, converting the 0-255 RGB image to [0,1], and then to [-1,1],<br>
Code is</p>
<pre><code class="lang-auto">transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) 
])
</code></pre>
<p>I can understand.ToTensor() converts data to [0,1], but I don’t know how Normalize converts data to [-1,1]. Although I know that input[channel] = (input[channel] - mean[channel])/STD [channel]</p>
<p>But I still don’t know how to get to [-1,1]<br>
Thanks.</p>

username: cclover
cooked: <p>I know something about Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) .<br>
The first (0.5, 0.5, 0.5) is the mean of R G B.<br>
The second is the std of R G B .<br>
0.5 is a roughly digit.<br>
<strong>The general statistics given on pytorch</strong> ：</p>
<p><strong>mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])</strong></p>

username: cclover
cooked: <p>ToTensor  -&gt;[-1,1]<br>
Normalize -&gt; [0,1]</p>

username: SimonW
cooked: <p>ToTensor gives you [0,1], then (x - 0.5) / 0.5 gives you [-1,1]</p>
Cnn with multiple outputs

username: mattinjersey
cooked: <p>Lets say that I have a picture, with zero turkeys, or a turkey at some position (x1,y1), or two turkeys at position (x0,y0),(x1,y1).<br>
I want the neural network to tell me the turkeys and their locations.<br>
I’m not sure how to train the nn for multiple outputs.<br>
For example it seems to me that there is now some order to the list. Would it declare a wrong answer if it outputs two turkeys in a different order, and how would it know the correct order.</p>
What is the scale of the sizes used for AnchorGenerator()?

username: Shisho_Sama
cooked: <p>Hello everyone,<br>
I recently stumbled upon this snippet of code given below, but I couldnt find any information concerning the scales of these numbers. what are they? are they in pixels? or they are sth entirely different?</p>
<pre><code class="lang-auto">from torchvision.models.detection.faster_rcnn import FasterRCNN
from torchvision.models.detection.rpn import AnchorGenerator

anchors = AnchorGenerator(sizes=(32,64,128,256,512), 
                          aspect_ratios=(0.5,1.0,2.0))
</code></pre>
<p>Thanks a lot in advance</p>
[solved] CNN+LSTM structured RNN

username: dongshuhao12
cooked: <p>Hi Folks,</p>
<p>I am trying to implement this paper “<a href="https://arxiv.org/pdf/1701.01909.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1701.01909.pdf</a>”. In order to do that, I need to train a sequence-to-one network with following structure:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/4d95403c9fc6a1769e0b11239c25f7cb8a327751.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d95403c9fc6a1769e0b11239c25f7cb8a327751" title="cnn_lstm.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/4d95403c9fc6a1769e0b11239c25f7cb8a327751_2_690x453.png" width="690" height="453" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/4d95403c9fc6a1769e0b11239c25f7cb8a327751_2_690x453.png, https://discuss.pytorch.org/uploads/default/original/1X/4d95403c9fc6a1769e0b11239c25f7cb8a327751.png 1.5x, https://discuss.pytorch.org/uploads/default/original/1X/4d95403c9fc6a1769e0b11239c25f7cb8a327751.png 2x" data-dominant-color="EBECED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cnn_lstm.png</span><span class="informations">766×504 59.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I need to train both the FC (i.e., \phi_{t}^A) and LSTM. I am  new to PyTorch, and I am not sure how to build the network by using PyTorch. I would appreciate any suggestion.</p>

username: dongshuhao12
cooked: <p>I think my understanding was wrong. I do not need to try the FC layer (i.e., \phi_{t}^A) with the LSTM. So, no problem now.</p>

username: akashs
cooked: <p>Could you please share how you handle the batch and sample timesteps in your training process? specifically hows your model forward function looks like? did you use loop to handle each frame or did you squeeze the batches and timesteps, extract the spatial feature using CNN, then unsqueeze batches and timesteps, finally fed into the LSTM. Thanks in advance.</p>
Confusion with CNN-LSTM implementation

username: ilyes
cooked: <p>Hello,</p>
<p>Let’s say I have a video of 10 frames and I want to train a CNN-LSTM model with a Resnet backbone for example. During training, do I use one identical Resnet to train on the 10 seperate frames, <strong>OR</strong> 10 different Resnet one for every frame ??<br>
Notes:</p>
<ul>
<li>let’s suppose that I will use weights of a Resnet trained on Imagenet as Initialization.</li>
<li>The reason im asking this question is because I implemented a paper on time-series where I saw that the model’s parameters were T times the feature extractor parameters. But then, I said if we want to use the, let’s say, Resnet as a fixed feature extractor we use a single one instead of T ones !</li>
</ul>
<p>Thank you</p>

username: 111137
cooked: <p>Hi,</p>
<ul>
<li>
<strong>Identical ResNet</strong><br>
Then you can make instance like this;</li>
</ul>
<pre><code class="lang-python">resnet1 = model()
resnet2 = model()
resnet3 = model()
</code></pre>
<ul>
<li>
<strong>Multi-Model</strong><br>
Then you can make multiple different models;</li>
</ul>
<pre><code class="lang-python">resnet1 = model1()
resnet2 = model2()
resnet3 = model3()
</code></pre>

username: IliasPap
cooked: <p>Hi, I don’t think it’s possible and correct to train multiple models , it won’t even fit in the gpu memory.<br>
You must train a single model where you extract the features for each frame an pass them through the lstm</p>

username: 111137
cooked: <p>Let us see Encoder-Decoder model consisting of encoder model and decoder model in the code.</p>

username: akashs
cooked: <aside class="quote no-group" data-username="ilyes" data-post="1" data-topic="54338">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/f0a364/48.png" class="avatar"> ilyes:</div>
<blockquote>
<ul>
<li>, let’s say, Resnet as a fixed feature extractor we use a single one instead of T ones !</li>
</ul>
<p>Thank you</p>
</blockquote>
</aside>
<p>As you want to extract visual features using the resnet(cnn) model, I think it will be better to keep only one resnet model. Because, using different resnet model for different frame doesn’t help to generalize the training process.</p>
Added some layer initially implemented by tensorflow into a PyTorch neural model

username: sirius
cooked: <p>I’d like to add some layer into a Pytorch based neural model. Basically I am trying to combine to codes together.</p>
<p>But I notice that the layer I want to add is implemented by Tensorflow. I’d like to know if there is an easy way to integrate a TensorFlow layer into a Pytorch neural model… ?</p>
<p>The error is shown as:</p>
<p>module ‘torch.nn’ has no attribute ‘tensorflow_layer’</p>

username: albanD
cooked: <p>I’m afraid there is not tensorflow and pytorch are two very different frameworks. But depending on the layer, it can be fairly easy to reimplement in pytorch.</p>

username: sirius
cooked: <p>Thank you very much for your reply. Yes, I understand that PyTorch can implement the same simple layer. But the layer I am going to add is a complex layer, which is dynamically changing during the training. So it is really complex to rewrite the whole code by using Pytorch…</p>
Change tensor in list inplace

username: dudy_karl
cooked: <p>Hi,<br>
i want to do the same operations on tensors which are not always the exact dimensions and would like to use a list (for instance [4,1,100,100] and [4,3,100,100] as in monochrome vs. RGB images).</p>
<p><em>what i would like to do is something like this:</em></p>
<p>def crop_tensors_list(tensors_list):<br>
for current_tensor in tensors_list:<br>
current_tensor = current_tensor[:,0:50,0:50]<br>
return tensors_list</p>
<p><code>if</code> <code>__name__ </code> <code>=</code> <code>=</code> <code>"__main__"</code> <code>:</code><br>
tensors1 = torch.randn(4,3,100,100)<br>
tensors2 = torch.randn(4,1,100,100)<br>
tensors_list = [tensor1,tensor2]<br>
tensors_list = crop_tensors_list(tensors_list)</p>
<p>however! - when i look:<br>
tensors_list[0] is tensor1 --&gt; it returns False.</p>
<p>anybody?</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/dudy_karl">@dudy_karl</a> ,</p>
<p>You declare <code>tensors1</code> and <code>tensors2</code> but you reference  <code>tensor1</code> and <code>tensor2</code> (with no <code>s</code>) when creating <code>tensors_list</code>. Otherwise what you wrote should work, as such:</p>
<pre><code class="lang-auto">def crop_tensors_list(tensors_list):
    for current_tensor in tensors_list:
        current_tensor = current_tensor[:,0:50,0:50]
return tensors_list


tensors1 = torch.randn(4,3,100,100)
tensors2 = torch.randn(4,1,100,100)
tensors_list = [tensors1,tensors2]
tensors_list = crop_tensors_list(tensors_list)
</code></pre>

username: albanD
cooked: <p>You cannot slice a Tensor inplace, so you won’t be able to do exactly this. But you can reassign the slice to the list:</p>
<pre><code class="lang-auto">for i in range(len(tensors_list)):
    tensors_list[i] = tensors_list[i][:,0:50,0:50]
</code></pre>
ValueError: expected 2D or 3D input (got 1D input)

username: sam101
cooked: <p>I am using a BatchNormalization 1d in my code and shows me this error.</p>
<pre><code>def __init__(self, x_dim, h_dim1, h_dim2,h_dim3,z_dim):
    super(VAE, self).__init__()
    self.x_dim = x_dim
    
    # encoder part
    self.fc1 = nn.Linear(x_dim, h_dim1)
    self.fc2 = nn.Linear(h_dim1, h_dim2)
    self.fc3 = nn.Linear(h_dim2,h_dim3)
    self.fc31 = nn.Linear(h_dim3, z_dim)
    self.fc32 = nn.Linear(h_dim3, z_dim)
    self.dropout = nn.Dropout(0.5)
    
    self.Encoder = nn.Sequential(
        self.fc1,
        nn.Dropout(0.5),
        nn.ReLU(),
        nn.BatchNorm1d(h_dim1),
        self.fc2,
        nn.Dropout(0.5),
        #nn.Dropout(0.1),
        nn.ReLU(),
        nn.BatchNorm1d(h_dim2),
        self.fc3,
        nn.Dropout(0.5),
        nn.ReLU(),
        nn.BatchNorm1d(h_dim3)
    )</code></pre>

username: beaupreda
cooked: <p>Hi,</p>
<p>What is the shape of your input tensor? According to the <a href="https://pytorch.org/docs/1.3.0/nn.html#batchnorm1d" rel="nofollow noopener">docs</a>, <code>nn.BatchNorm1d</code> expects at minimum a 2D input tensor (<code>batch_size x num_features</code>). It says: “Applies Batch Normalization over a 2D or 3D input (a mini-batch of 1D inputs with optional additional channel dimension)”. From your error message, I would assume you are missing a batch dimension, but I may be wrong. If you could provide an example of an input tensor, we could help you debug (if the missing batch dimension is not the problem).</p>

username: sam101
cooked: <p>The shape is 1 by 3000</p>

username: sam101
cooked: <p>where can i add that batch dimension</p>

username: beaupreda
cooked: <p>Ok, you need a batch size larger than 1, otherwise you cannot do batch statistics! I tried your code with a batch size larger than 1, and everything was working fine.</p>

username: beaupreda
cooked: <p>You can simply concatenate multiple examples together with <code>torch.cat</code>, like in this small example:</p>
<pre><code class="lang-auto">    x = torch.rand(size=(1, 3000), dtype=torch.float32)
    y = torch.rand(size=(1, 3000), dtype=torch.float32)
    z = torch.cat((x, y), dim=0)
</code></pre>
<p>I do not know how you load your data, but if you are not using PyTorch’s <code>torch.utils.data.Dataset</code> and <code>torch.utils.data.DataLoader</code> classes, you could as it does the concatenation of multiple examples into a batched tensor for you (see <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">tutorial</a>).</p>
<p>.</p>
How to reshape last layer of pytorch CNN model while doing transfer learning

username: talhaanwarch
cooked: <p>Actually i am trying to replicate keras structure to pytorch.(new in pytorch).<br>
Here is keras architecture</p>
<pre><code>base_model = InceptionV3(weights='imagenet', include_top=False)
x = base_model.output
x = Dense(512, activation='relu')(x)
predictions = Dense(49*6,activation='sigmoid')(x)
reshape=Reshape((49,6))(predictions)
model = Model(inputs=base_model.input, outputs=reshape)
for layer in base_model.layers:
    layer.trainable = False
</code></pre>
<p>I want to reshape last layer of my netwrok. I have implemented transfer learning.</p>
<pre><code>model =  models.inception_v3(pretrained=True)

for param in model.parameters():
    param.requires_grad = False
    
num_ftrs = model.fc.in_features
</code></pre>
<p>I believe if i can attach last layer of resnet with my following architecture, problem can be solved. But i dont know how i can attach them</p>
<pre><code>class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.fc1 = nn.Linear(num_ftrs, 512)
        self.fc2 = nn.Linear(512, 49*6)

    def forward(self, x):
        print (x.shape)
        x = x.view(-1,num_ftrs)
        #print (x.shape)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x=torch.sigmoid(x.view(49,6))
        return x
</code></pre>
<p>Any idea, how this problem can be resolved</p>

username: ptrblck
cooked: <p>You could add your custom linear layers to <code>model.fc</code> as shown here:</p>
<pre><code class="lang-python">model =  models.inception_v3()
fc_feat = model.fc.in_features
model.fc = nn.Sequential(
    nn.Linear(fc_feat, 512),
    nn.ReLU(),
    nn.Linear(512, 49*6),
    nn.Sigmoid(),
)
</code></pre>
<p>Let me know, if that would work for your use case.</p>

username: talhaanwarch
cooked: <p>there is no reshaping of last layer output, i want 49,6 instead of 49*6</p>

username: mailcorahul
cooked: <p>You can reshape the output tensor of shape <code>(49*6, )</code> to <code>(49, 6)</code>.<br>
Simply do:</p>
<pre><code class="lang-auto">output = model(input)
output = output.view(-1, 49, 6)
</code></pre>
Input normalization in VAE

username: zelazny
cooked: <p>I am training a special case of a VAE on MNIST and an additional dataset. Whenever I scale the inputs so that they stay in the <code>[0, 1]</code> range, training fails miserably with the decoder generating the same image over and over again. At the same time, standardizing the data with the means and standard deviations of the respective datasets results in relatively rapid progress in training. This is a bit surprising to me as the last layer of my decoder is a sigmoid function which maps to <code>[0, 1]</code> rather than <code>(-inf, +inf)</code>  - the range we theoretically get after standarization. The way I understand VAE, the encoder and decoder should “mirror” each other’s structure, but in my case there is a clear mismatch between the input and what the decoder can actually generate. I am using a binary cross entropy loss function (<code>BCELoss</code>) for the reconstruction loss. Am I missing something fairly obvious here?</p>
How to load my own data?

username: pengpeng
cooked: <p>Hello, I am a new user of pytorch. I want to ask how can I load my own data? Thanks a lot!</p>

username: mmisiur
cooked: <p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Here</a> is an extensive and very well written tutorial on how to make and load custom datasets. I’m sure that it will help you.</p>

username: pengpeng
cooked: <p>Thanks, it helps a lot~</p>
Efficient quantile/k-largest value

username: xzyx
cooked: <p>I saw that there is a median function supported. I am wondering whether there is any operation that can compute the k-largest values without using sorting? (I can implement it with topk() but I assume when k is large, the function is not very efficient?)</p>

username: albanD
cooked: <p>I am not sure to understand what you’re looking for? If you’re looking for the k-largest values, then <code>topk()</code> is the function you should use, this is exactly what this function is doing.</p>

username: tom
cooked: <p>I think it’d be neat to get just the kth largest value efficiently, eg the 0.25 and 0.75 quartile values, similar to np.percentile.</p>
<p>Best regards</p>
<p>Thomas</p>

username: albanD
cooked: <p>So you’re looking for the kth largest value? not all the top k largest values?</p>

username: tom
cooked: <p>I’ve been looking for exactly that a few times. (And that is the similarity with median as far as I understand.)</p>

username: albanD
cooked: <p>No that does not exist as far as I know.</p>

username: aliutkus
cooked: <p>I know it’s not as efficient as it could be, but I implemented a pytorch version for percentile:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/aliutkus/torchpercentile" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/9372516?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/aliutkus/torchpercentile" target="_blank" rel="nofollow noopener">aliutkus/torchpercentile</a></h3>

<p>Percentile computation for pytorch. Contribute to aliutkus/torchpercentile development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: tom
cooked: <p>There now is a kthvalue function for CPU and CUDA in torch 1.1 (or the nightlies).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Zayd
cooked: <p>For those who later read this thread and want to jump to the docs, there are both a <a href="https://pytorch.org/docs/stable/torch.html#torch.kthvalue" rel="nofollow noopener">torch kthvalue</a> method and a similar <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.kthvalue" rel="nofollow noopener">Tensor kthvalue</a> method as well.</p>

username: Dan_Erez
cooked: <p>is there a simple way to have k be a vector rather than an int so that its a different k for each sample in a batch?</p>
Creating tensor with values depending on the indices

username: sapo
cooked: <p>Hello,<br>
I need to create a tensor where entry <code>i,j</code> has value <code>abs(i - j)</code>.<br>
For now, the only solution I’ve found is by cycling each entry with <code>for</code> loop:</p>
<pre><code class="lang-auto">A = torch.zeros(1000, 2000)
for i in range(1000):
    for j in range(2000):
        A[i, j] = abs(i - j)
</code></pre>
<p>Unfortunately this solution is extremely slow.</p>
<p>I need this tensor to use it as mask of another tensor.</p>
<p>Someone has any alternative idea?</p>

username: albanD
cooked: <p>I’m not sure about the exact code but you can use range and expand to have matrices like A[i, j] = i, forall j: range(1000).unsqueeze(1).expand(1000, 2000). And the same for the other one with <code>unsqueeze(0)</code>.<br>
Then you can substract them element-wise and take the absolute value of the result.</p>

username: sapo
cooked: <p>Really thanks, I was using python list multiplication with <code>torch.stack</code>, but this is much faster.</p>
<p>On CPU:</p>
<pre><code class="lang-auto">In [22]: %timeit torch.stack([torch.arange(1000)] * 2000, 1)                                   
42.3 ms ± 125 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [23]: %timeit torch.arange(1000).unsqueeze(1).expand(1000, 2000)                            
13.1 µs ± 21.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

</code></pre>
<p>On GPU:</p>
<pre><code class="lang-auto">In [27]: %timeit torch.stack([torch.arange(1000).to(device)] * 2000, 1).to(device)             
2.15 ms ± 6.66 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [30]: %timeit torch.arange(1000).to(device).unsqueeze(1).expand(1000, 2000).to(device)    
43.8 µs ± 152 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
</code></pre>
Translate a Tensor without copy

username: theevann
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/sunny.png?v=9" title=":sunny:" class="emoji" alt=":sunny:"></p>
<p>In my project, I need to input multiple translations of the same Tensor to a convolution layer.</p>
<p>A simplified version of what I do is :</p>
<pre><code class="lang-python">im = torch.randint(0, 9, (3, 100, 200))
im_tr = torch.zeros_like(im)
im_tr[:, 50:, 50:] = im[:, :-50, :-50]
im_cat = torch.cat((im, im_tr), dim=0)
im_cat
</code></pre>
<p>However, this operation is making a copy of the potentially big Tensor <code>im</code>. More, I don’t want to input one translation but potentially tens. Values of the translation offset can potentially be big.</p>
<p>Is there a clean way to do that in python-pytorch ?<br>
Do you have any suggestion to avoid exploding the memory while always using the same data ?</p>

username: Sunshine352
cooked: <p>In the limited translations, you can regard multiple translations as the original tensor convolved with gaussian kernel. The specific formulas you can formulate yourself.</p>
[PyTorch]Read CIFAR10

username: cclover
cooked: <p>I don’t have CIFAR10 data locally, and then I download one, but it doesn’t download automatically, and the read fails.</p>
<pre><code class="lang-auto">import torchvision as tv
import torchvision.transforms as transforms
from torchvision.transforms import ToPILImage
import torch as t

transform = transforms.Compose([
    transforms.ToTensor()  
    ,transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))  
])
trainset = tv.datasets.CIFAR10(
    root = './data',
    train = True,
    download = True, 
    transform = transform
)
trainloader = t.utils.data.DataLoader(
            trainset,
            batch_size = 4,
            shuffle = True,
            num_workers=2
)
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
&lt;ipython-input-38-355678cb95ef&gt; in &lt;module&gt;
      1 trainloader = t.utils.data.DataLoader(
----&gt; 2             trainset,
      3             batch_size = 4,
      4             shuffle = True,
      5             num_workers=2

NameError: name 'trainset' is not defined
</code></pre>
<p>Best</p>

username: ptrblck
cooked: <p>What do these lines of code return:</p>
<pre><code class="lang-python">trainset = tv.datasets.CIFAR10(
    root = './data',
    train = True,
    download = True, 
    transform = transform
)
</code></pre>
<p>Do you get any error trying to download the data?</p>

username: cclover
cooked: <p>The code doesn’t return anything.<br>
Bset</p>

username: ptrblck
cooked: <p>In that case <code>trainset</code> should be properly initialized.<br>
What does <code>print(trainset)</code> afterwards print out?</p>

username: cclover
cooked: <p>Got nothing and changed nothing</p>
<pre><code class="lang-auto">trainset = tv.datasets.CIFAR10(
    root = './data',
    train = True,
    download = True, 
    transform = transform
)
print(trainset)
</code></pre>

username: ptrblck
cooked: <p><code>print(trainset)</code> prints nothing out? <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_raised_eyebrow.png?v=9" title=":face_with_raised_eyebrow:" class="emoji" alt=":face_with_raised_eyebrow:"><br>
What does <code>print(len(trainset))</code> output?</p>

username: cclover
cooked: <p>Oh, I still didn’t get anything, but I copied the code to another block of code, and it’s done.<br>
Sincerely thank you for helping me again.<br>
Best</p>

username: cclover
cooked: <p>Maybe the jupyter notebook have something wrong?</p>

username: ptrblck
cooked: <p>Might be. I’ve seen all kind of weird restarts etc. with notebooks.<br>
So if something doesn’t seem to right, I would recommend to run the script in a terminal, which should work.</p>

username: cclover
cooked: <p>Thank you again. I’ll take your advice</p>