akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Num_workers > 1 and C++ multithreading

AreTor: <p>Part of my project is written in C++, and I’m using pybind11 gil release/acquire functions to obtain multithreading in a section of my code that preprocesses single images of a dataset.</p>
<p>I wanted to know if it is safe to set <code>num_workers=2</code> or greater in a DataLoader, because I have encountered a nasty random IndexError arising in the C++ code, which may hide a multithreading problem.</p>
<p>EDIT: I am also using a <code>std::mutex</code> in the C++ code.</p>
Cannot kill a process that occupies a port

samin_hamidi: <p>I want to access to the tensorboard’d log files on a remote server. My local machine has windows 10. I use terminal and this command <code>ssh -L 16006:127.0.0.1:6006 ...</code> to connect to the serve and then use <code>tensorboard --logdir Runs --port 6006</code> when I’m one directory higher than Runs. However, I get this error <code>ERROR: TensorBoard could not bind to port 6006, it was already in use</code>. Before I used to solve this issue with using lsof -i:6006 to get access to the occupying PID and then killing that process by using kill -9 PID. Now, using lsof does nothing. Is there a solution to this? I tried terminating the VSCode occupied ports (I sometimes use VSCode to access to the logs) but I still get the message that the port is occupied.</p>
State of art pre-trained image embedding for a custom auto-encoder

Thibaut: <p>Dear,</p>
<p>I would like to try an auto-encoder using a pretrained layer for transfert learning.</p>
<p>The code would be something like that:</p>
<pre><code class="lang-auto">class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()

        self.encoder = pretrained_encoder(...)
        self.decoder = symetric_decoder(...)
        self.linear1 = nn.Linear(dim_encoder_output, dim_intermediary)
        self.linear2 = nn.Linear(dim_intermediary, dim_embedding)
        self.linear3 = nn.Linear(dim_embedding, dim_embedding)
        self.linear4 = nn.Linear(dim_embedding, dim_intermediary)
        self.linear5 = nn.Linear(dim_intermediary,dim_decoder_input)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.encode(x)
        x = self.decode(x)
        return x

    def encode(self, x):
        x = self.encoder(x)
        x = self.linear1(x)
        x = self.relu(x)
        x = self.linear2(x)
        x = self.relu(x)
        x = self.linear3(x)
        x = self.relu(x)
        return x

    def decode(self,x):
        x = self.linear4(x)
        x = self.relu(x)
        x = self.linear5(x)
        x = self.relu(x)
        x = self.decoder(x)
        return x
</code></pre>
<p>The goal of this auto_encoder is to detect small changes in industrial parts. Such as defects, pollutions, rust… Currently it works with a custom auto encoder and I want to see if a pre-trained state of art model improve it.</p>
<p>I wonder what pretrained encoder to use. Is there some state of art pre-trained encoders that are already implemented as a PyTorch module that you would recommend for a test?</p>
Share pytorch model between inference and train process, simple

Frank_Wessels: <p>I would like to share a pytorch model between two processes. I have a trainer and an application (inference) module as separate processes running. I have implemented model.save/load only when the trainer is writing and the inferences module tries to read, it gives an error (obviously).</p>
<p>I am looking at this forum and everywhere and it drives me crazy with complex solutions for training with multiple processes etc which is not what I want. I am really wondering how hard it can be to simple share the state dict between two processes.</p>
<p>One attempt was using SharedMemoryManager, here is an example of what I wrote:</p>
<pre><code class="lang-auto">    with SharedMemoryManager() as shared_memory_manager:

        # Initialize shared memory
        shared_memory = dict()
        for key in shared_memory_settings.keys():
            shape = shared_memory_settings[key]["shape"]
            dtype = shared_memory_settings[key]["dtype"]
            shared_memory_settings[key]["shm"] = shared_memory_manager.SharedMemory(size=np.zeros(**shared_memory_settings[key]).nbytes)
        shared_memory_settings["state_dict"] = shared_memory_manager.SharedMemory(size=sys.getsizeof(model.model.state_dict()))

        # Initialize and start applications
        p1 = Process(target=Trainer, args=(shared_memory_settings, deepcopy(model), model_weights_path))
        p2 = Process(target=Application, args=(dataset, model, model_weights_path, shared_memory_settings))
        p1.start()
        p2.start()
        p1.join()
        p2.join()
</code></pre>
<p>I am trying to use SharedMemory which works fine with numpy arrays but it gives me a difficult time for sharing the state dict.</p>
<p>So my question is, what is the best practise for a very simple problem which is using model weights for inference while a seperate independent process is training and updating these weights on a regular basis.</p>
<p>I have looked into queues but on a first glance it looks very complicated and overcomplicated things, but perhaps that is the way I should go?</p>
Not able to understand this peice of code can anyone help me with it?

DARK-art108: <pre><code>    self.backbone = timm.create_model(backbone, pretrained=pretrained)
    self.in_features = self.backbone.head.in_features
    self.backbone.reset_classifier(0)
    self.fc = nn.Linear(self.in_features, num_classes)
</code></pre>
<p>Also How do we connect a backbone model and embedding model?</p>
<p>backbone model is swin_large_patch4_window12_384_in22k</p>
Feature Extractor

Zafar_Aziz: <p>I want to extract features using a fine tuned vgg16 model based extractor. I want to know, should i extract features in <strong>model.eval()</strong> mode that is I should put my extractor model to eval mode first and then apply it to input image for feature extraction? or should I use my extractor with in <code>with torch.no_grad():</code></p>
Torch.no_grad and "required gradients"

mreister: <p>I am receiving the following error:</p>
<p><em><strong>RuntimeError: Only Tensors of floating point and complex dtype can require gradients.</strong></em></p>
<p>when i run a convolution operation inside a torch.no_grad()context:</p>
<pre><code class="lang-python">import torch.nn as nn
import torch


with torch.no_grad():
    ker_t = torch.tensor([[1, -1] ,[-1, 1]])
    in_t = torch.tensor([ [14, 7, 6, 2,] , [4 ,8 ,11 ,1], [3, 5, 9 ,10], [12, 15, 16, 13] ])
    print(in_t.shape)
    in_t = torch.unsqueeze(in_t,0)
    in_t = torch.unsqueeze(in_t,0)
    print(in_t.shape)

    conv = nn.Conv2d(1, 1, kernel_size=2,stride=2,dtype=torch.long)
    conv.weight[:] = ker_t
    conv(in_t)
</code></pre>
<p>Now i am sure if i turn  my input into floats this message will go away, but i want to work in integers.</p>
<p>But i was under the impression that if i am in a “with torch.no_grad()” context it should turn of the need for gradients.</p>

ptrblck: <p>The error message might be a bit confusing, but since <code>nn.Parameters</code> can require gradients outside of the <code>no_grad()</code> context manager, you won’t be able to assign <code>LongTensor</code>s to them.</p>
Batching custom/layer/module without loops?

mvhv: <p>For reasons I don’t want to delve too deeply into, I’m running inference using the OpenCV DNN runtime, on a PyTorch model that’s been exported as ONNX. I have no issues with the model in PyTorch or ONNX, but when loading into OpenCV I run into an issue where the MatMul node fails to load. As far as I understand this happens is because the OpenCV DNN implementation of the MatMul op doesn’t support batch dimensions.</p>
<p>Obviously the ideal solution here is raising an issue on the OpenCV repo, but in the short term I’m looking for a way around this.</p>
<p>By squeezing/unsqueezing the batch dimension for this one operation I can get the network working for inference, but but only when the input batch size is one. Is there a way I can wrap the 2D matmul in some kind of batching op that applies the matmul to each batch individually?</p>
<p>I know I could do this with a simple for loops, but I think that runs into issues with variable batch sizes once the JIT trace unrolls them.</p>
When to use 1 hot encoding with pythorch

Oussama_Bouldjedri: <p>I have a multi class classification problem so I am going with Cross entropy I have 12 label: initially categorical labels , but I perform an integer encoding,  I was wondering if should I use one hot encoding instead and how that would fit or not my case ? thanks</p>

KFrank: <p>Hi Oussama!</p>
<aside class="quote no-group" data-username="Oussama_Bouldjedri" data-post="1" data-topic="145135" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oussama_bouldjedri/48/64172_2.png" class="avatar"> Oussama_Bouldjedri:</div>
<blockquote>
<p>I was wondering if should I use one hot encoding</p>
</blockquote>
</aside>
<p>I can’t think of a good use case in pytorch where one-hot encoding<br>
makes sense.  (Of course, if you are using some third-party code<br>
that expects one-hot encoding, then you’ll need it.)</p>
<p>In the case of pytorch’s <code>CrossEntropyLoss</code>, unless you are using<br>
<em>probabilistic</em> values for <code>target</code> (the labels), integer (categorical)<br>
class labels are expected.</p>
<p>So, no, don’t use one-hot encoding.  Stick with your “integer encoding.”<br>
That’s what <code>CrossEntropyLoss</code> expects, and it’s a perfectly fine approach.</p>
<p>Best.</p>
<p>K. Frank</p>
Problem with 'num_workers' in my data_loader

adamIsStuck: <p>Hi, there I am trying to train a model and getting a strange error…</p>
<p>If I run the model on a small subsection of my data without the data loader argument ‘num_workers’ the model runs fine…</p>
<p>However, if I attempt to run the model on the whole dataset (or even a significant amount of the dataset) the model hits a bottleneck - still without ‘num_workers’.</p>
<p>I figured the simple solution to this error is to set ‘num_workers’ - I have tried a number of different workers up to 32. This is where I get my error as seen below…</p>
<p>In summary, if I want to train my model on the whole dataset I am stuck as the model either bottlenecks when ‘num_workers’ is removed from the data loader or I get this error if I set it to anything else.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c9f27c36c015b85b3a3b2e0778a772b15aafa37.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0c9f27c36c015b85b3a3b2e0778a772b15aafa37" title="274460471_264838269014206_6455405089353700865_n"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c9f27c36c015b85b3a3b2e0778a772b15aafa37_2_690x146.jpeg" alt="274460471_264838269014206_6455405089353700865_n" data-base62-sha1="1NEIxmOMZLenZH0i6Ct6bE35pxd" width="690" height="146" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c9f27c36c015b85b3a3b2e0778a772b15aafa37_2_690x146.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c9f27c36c015b85b3a3b2e0778a772b15aafa37_2_1035x219.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c9f27c36c015b85b3a3b2e0778a772b15aafa37_2_1380x292.jpeg 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">274460471_264838269014206_6455405089353700865_n</span><span class="informations">2096×446 220 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

thecho7: <p>I guess this is not the problem occurred by <code>num_workers</code>.<br>
I don’t know how you sampled a subset but you’d better unite the format of your dataset.</p>
<p>Check the data type</p>
Computationally Heavy Computer Vision Algorithms on a Remote Server

samin_hamidi: <p>I have a general question regarding running computationally heavy computer vision algorithms on remote servers. So far I have been working with smaller datasets which takes only a few hours to run and yet I have faced many situations where the process broke down due to a multitude of reasons. Maybe it was my internet connection or my VSCode got disconnected, maybe it was due to the VPN. I wonder how people set their computational environment so they can run a process without trouble for days? other than checkpoints, which methods are common practice? I would be glad to hear about your opinion</p>

KFrank: <p>Hi Samin!</p>
<aside class="quote no-group" data-username="samin_hamidi" data-post="1" data-topic="145107" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a698b9/48.png" class="avatar"> samin_hamidi:</div>
<blockquote>
<p>I wonder how people set their computational environment so they can run a process without trouble for days?</p>
</blockquote>
</aside>
<p>Let me assume that your remote server is running a flavor of unix (e.g.,<br>
linux) and that you can run on it a remote shell (e.g., ssh) from your local<br>
computer.</p>
<p>You want to run the remote job so that it is not dependent on the connection<br>
between your local computer and your remote server.</p>
<p>In the unix world, you run your desired remote command using “nohup”<br>
(which comes from “no hang-up”):</p>
<pre><code class="lang-plaintext">nohup python my_remote_job_python_script.py &amp;&gt; my_remote_job_output.txt
</code></pre>
<p>You can then watch the progress of your remote job by running (in a remote<br>
shell):</p>
<pre><code class="lang-plaintext">tail -f my_remote_job_output.txt
</code></pre>
<p>If your connection goes down, you just restart your connection, start a<br>
new remote shell, and go back to watching your job by running <code>tail -f</code><br>
again.  (If your remote python script crashes or your remote server blows<br>
up, you’re in the soup, but, hey, at least your local computer can get struck<br>
by lightning …)</p>
<p>Best.</p>
<p>K. Frank</p>
How to retrieve matching data pairs from different dataloaders?

Alpha_Sight: <p>Hello everyone,</p>
<p>I am doing a binary classification and have four different datasets:</p>
<ol>
<li>Dataset_1: Ground truth of matching pairs (e.g. columns x, y)</li>
<li>Dataset_2: Graph features of x1, x2, …, xn</li>
<li>Dataset_3: Numerical features of x1, x2, …, xn</li>
<li>Dataset_4: Graph features of y1, y2, …, yn</li>
</ol>
<p>Due to the nature of the datatype, it is not feasible to join them into a single dataset.</p>
<p>I intend to train each of these datasets with different models, and concat the results into an ensemble model:</p>
<ol>
<li>Dataloader_1(Dataset_1) → Matching pairs (e.g. x10/y10)</li>
<li>Dataloader_2(Dataset_2) → x10 → Model_2 → Output 2</li>
<li>Dataloader_3(Dataset_3) → x10 → Model_3 → Output 3</li>
<li>Dataloader_4(Dataset_4) → y10 → Model_4 → Output 4</li>
<li>Output 2, 3, 4 → Model_5 → Final prediction</li>
</ol>
<p>However, I am unsure of the following:</p>
<ol>
<li>
<p>Positive Training Examples:<br>
How do I get Dataloader_2, Dataloader_3, and Dataloader_4 to return the correct matching IDs to the model?<br>
For example, if training on x5/y5 pair, Dataloader_2 and 3 should return features on x5, and Dataloader_4 should return features on y5. The purpose is to train the model to learn what pairs will match.</p>
</li>
<li>
<p>Negative Training Examples:<br>
How do I get the Dataloader_2, Dataloader_3, and Dataloader_4 to return wrong matching IDs to the model?<br>
For example, if training on x6/y6 pair, Dataloader_2 and 3 should return features on x6, but Dataloader_4 should return all other ys, except for y6. The purpose is to train the model to learn what pairs won’t match.</p>
</li>
</ol>
<p>Thanks for your help!</p>

ptrblck: <p>I think one potential approach would be to define all <code>Dataset</code>s first and make sure the internal data is stored in the same order. I.e. assuming you are either lazily loading the data or are pre-loading it from a tensor or numpy array, make sure that these samples are sorted accordingly.<br>
Once this is done, you could then create a custom <code>Dataset</code> and call the internal 4 datasets with the same <code>index</code> (so you could also use <code>shuffle=True</code> in the <code>DataLoader</code>).<br>
For 2: I don’t know when his negative sampling should be done, but I assume you could trigger this condition by resampling a new <code>index</code> to sample the negative example inside <code>dataset_4</code>.</p>
LSTM late fusion time series prediction

overclock: <p>Greetings to everyone!</p>
<p>I have a dataset with 4 modalities and I want to feed each modality to a LSTM or RNN and then combining the results of all the 4 networks in order to obtain a global result.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/309855edb92f0b1f7674af34bae468fbc233c984.png" data-download-href="https://discuss.pytorch.org/uploads/default/309855edb92f0b1f7674af34bae468fbc233c984" title="com-features-extraction"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/309855edb92f0b1f7674af34bae468fbc233c984_2_690x345.png" alt="com-features-extraction" data-base62-sha1="6VTimGGPDy1KvkFAp6tG6AcNn3S" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/309855edb92f0b1f7674af34bae468fbc233c984_2_690x345.png, https://discuss.pytorch.org/uploads/default/original/2X/3/309855edb92f0b1f7674af34bae468fbc233c984.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/309855edb92f0b1f7674af34bae468fbc233c984.png 2x" data-dominant-color="DFE2E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">com-features-extraction</span><span class="informations">940×471 97.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I wrote a couple of examples with RNN and softmax at the end, but I don’t know how to deal with 4 networks at once.<br>
Does anyone knows a simple example of combining 2 LSTM’s or RNN’s by means of late fusion in PyTorch?<br>
Should I feed all the 4 networks to the same loss function?<br>
Thanks a lot!</p>

Zoe_Vasileiou: <p>Hello</p>
<p>Did you find how this late fusion is feasible? Since I am facing the same issue.</p>
<p>Thanks<br>
Zoe</p>
Batched 2D index selection

Yifan_Ding: <p>I try to write a model which includes a batched 2D index selection step, want to ask the best/fast way to perform this without any for loops.</p>
<p>Input:<br>
B: batch_size, N: length of a matrix, K1, K2 number of candidates; K1 &lt;&lt; N, K2 &lt;&lt; N.<br>
X: is a 3D binary tensor, shape = [B, N, N], X[b][n][n] \in {0, 1}<br>
S: is a 2D index tensor, shape = [B, K1], S[b][k1] \in {0,1,2,…N-1}<br>
E: is a 2D tensor, shape = [B, K2], E[b][k2] \in {0,1,2,…N-1}</p>
<p>Output:<br>
C: is a 2D tensor, shape = [S, 3] for each c in C satisfy that X[c[0]][c[1]][c[2]] = 1 and c[1] \in S[c[0], :] and c[2] \in E[c[0], :].</p>
Value of Module.state_dict()

Yunxiang_LI: <p>I tried to run the following code and the output confused me.<br>
Code is:</p>
<pre><code class="lang-python">class test_model(nn.Module):
    def __init__(self):
        super(test_model, self).__init__()
        self.fc = nn.Linear(10, 20)
    def forward(self, x):
        x = self.fc(x)
        return x

def model_sum(params):
    # return sum of parameters of a model
    temp = 0
    for value in params.values():
        temp += value.sum().item()
    return temp

model = test_model()
params = model.state_dict()
print(model_sum(params))

new_params = OrderedDict()
for key in params.keys():
    new_params[key] = params[key] + 1.0
model.load_state_dict(new_params)

print(model_sum(params))
</code></pre>
<p>And the output is</p>
<pre><code class="lang-auto">-0.15531682968139648
219.84470748901367
</code></pre>
<p>which means params is like a pointer to the state_dict of model. I am curious about why the output is like this?</p>

ptrblck: <p>I’m not sure I understand your concern completely.<br>
Since you are loading the newly created <code>state_dict</code> (with the addition of 1), I would assume <code>model_sum</code> to return a higher number.</p>

Yunxiang_LI: <p>Thank you for prompt reply.<br>
Yes, <code>model_sum</code> returns a higher number.<br>
My question is why <code>params</code>, which is the OrderedDict of model parameters, also changes when the model parameters change, even without assigning the new <code>model.state_dict()</code> to it.</p>

Raynchow: <p><code>params = model.state_dict()</code> is inplace operation, which means params is always to same as model.state_dict()</p>
LSTM Prediction Flat Line but close shape

Kelly_Tan: <p>I created an LSTM but the prediction is always very close to a straight line. I normalised the train and test set separately (fit on train then transform the whole dataset).</p>
<pre><code class="lang-auto">train_size = int(len(B1_monthly_df) * 0.8)
B1_monthly_df_training = B1_monthly_df.copy()
train_data = B1_monthly_df_training.iloc[:train_size]

sc = MinMaxScaler()
#fit only on train data
scaler = sc.fit(train_data[train_data.columns])
#scale entire dataset
B1_monthly_df_training[B1_monthly_df_training.columns] = scaler.transform(B1_monthly_df_training[B1_monthly_df_training.columns])
</code></pre>
<p>After I inverse transformed the predictions, this is what I got (blue is actual, orange is predicted)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c20a5253236380d2db675a796a2c4999a0a460c.png" alt="image" data-base62-sha1="hI59brgZAc0tenrFSER5Qfja1qY" width="543" height="306"></p>
<p>However when I isolated the actual and predicted and plotted them in their own scales, the shapes seem to be quite similar (and the mean is around the same scale [around 20 for both]), just that the scale of variation in predicted (around 0.8) is so much smaller than in train (around 30).</p>
<p>I also tested out using a more months for test (60% test, 40% train) to see if it was just because the last part is flattish, but I still got a very straight line.</p>
<p>What could be the cause of such an issue? Thanks!</p>
Suitable DNN architecture to map noisy tensors to its clean version

Mohamed_Nabih: <p>I have a dataset represents noisy speech embeddings and clean speech embeddings. The dataset is saved as torch tensors each tensor has a shape of 512 x 350. I built a CNN model based on the U-Net architecture to map the noisy embeddings to its clean embeddings, but the results are not optimistic. Can anyone help or suggest a model that can perform better in this task. Thanks</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x21632 and 1152x120)

DARK-art108: <p>I ammfacing this error from a long time, anyone can tell where is the problem attached the code below<br>
class NetModel(nn.Module):</p>
<pre><code>def __init__(self):

    super().__init__()

    self.conv_1 = nn.Conv2d(3, 32, 3, padding=1)

    self.conv_2 = nn.Conv2d(32, 64, 3)

    self.maxpool_1 = nn.MaxPool2d(2,2)

    self.conv_3 = nn.Conv2d(64, 128, 3)

    self.bn = nn.BatchNorm2d(128)

    self.fc1 = nn.Linear(128*3*3, 120)

    self.fc2 = nn.Linear(120, 84)

    self.fc3 = nn.Linear(84, 10)

def forward(self, x):

    x = F.relu(self.conv_1(x))

    x = F.relu(self.conv_2(x))

    x = self.maxpool_1(x)

    x = F.relu(self.conv_3(x))

    x = self.bn(x)

    x = torch.flatten(x, 1)

    x = F.relu(self.fc1(x))

    x = F.relu(self.fc2(x))

    x = self.fc3(x)

    return x</code></pre>

ptrblck: <p>Based on the error message, I guess the activation input to <code>self.fc1</code> has <code>21632</code> features, while <code>self.fc1</code> is using <code>in_features=1152</code>.<br>
Use:</p>
<pre><code class="lang-python">self.fc1 = nn.Linear(21632, 120)
</code></pre>
<p>and it should work.</p>

aiGeekProgrammer: <p>Did you calculate dimensions of your convnet? Because it’s probably wrong.</p>
<p>General formula with input vector dimensions = HxWxC is floor((W−K+2*P) / S) + 1<br>
P - padding<br>
S- stride<br>
K- kernel size<br>
If W &lt;&gt; H then calculations need to be done for both dimensions: Height and Width.</p>
<p>Plus there is 2x2 maxpool which will reduce the size of each dimension twice.</p>
<p>What is the dimension of your input?</p>

DARK-art108: <p>The dimension of the input image os 32x32x3</p>

aiGeekProgrammer: <p>What you need to do is to flatten your data after the last convolutional layer - look at the script below - and also correct the dimension of your first linear layer</p>
<pre><code class="lang-auto">import torch
from torch.nn import Conv2d, MaxPool2d, BatchNorm2d, Linear

input = torch.rand((1, 3, 32, 32)) # Batch size, number of channels, Heigth, Width

conv_1 = Conv2d(3, 32, 3, padding=1)
t1 = conv_1(input)
t1.size()
&gt;&gt;&gt; torch.Size([1, 32, 32, 32])

conv_2 = Conv2d(32, 64, 3)
t2 = conv_2(t1)
t2.size()
&gt;&gt;&gt; torch.Size([1, 64, 30, 30])

maxpool_1 = MaxPool2d(2,2)
t3 = maxpool_1(t2)
t3.size()
&gt;&gt;&gt; torch.Size([1, 64, 15, 15])

conv_3 = Conv2d(64, 128, 3)
t4 = conv_3(t3)
t4.size()
&gt;&gt;&gt; torch.Size([1, 128, 13, 13])

bn = BatchNorm2d(128) # doesn't change the size
t5 = bn(t4)
t5.size()
&gt;&gt;&gt; torch.Size([1, 128, 13, 13])

t6 = t5.reshape(1, 128*13*13) # flat the output for the input to nn
t6.size()
&gt;&gt;&gt; torch.Size([1, 21632])

fc1 = Linear(128*13*13, 120)
t7 = fc1(t6)
# Looks fine :-)
</code></pre>
Unable to reproduce accuracies on ImageNet validation set

fourleafclover: <p>Hi everyone,</p>
<p>I’m currently trying to reproduce previously published top-1 accuracies on the ImageNet validation set based on pre-trained models whose weights have been made publicly available. I noticed that my results are pretty close to the published ones, but they are still somehow off by a small margin.<br>
To find a potential bug, and as a first sanity check, I first tried to reproduce the <a href="https://pytorch.org/vision/stable/models.html" rel="noopener nofollow ugc">reported top-1 accuracies</a> for ResNet50 and VGG16 w/o batchnorm by simply evaluating the two pytroch models with <code>pretrained=True</code> on the ImageNet validation set. I get top-1 accuracies of 76.02 (vs. 76.130 reported) for ResNet50 and 71.622 (vs. 71.592 reported) for VGG16.</p>
<p>I really wonder, why it is that I cannot exactly reproduce <a href="https://pytorch.org/vision/stable/models.html" rel="noopener nofollow ugc">the results</a>, as the process of simply evaluating the pre-trained nets should be deterministic.</p>
<p>I double checked the following things which I thought could lead to the slight difference:</p>
<ul>
<li>
<p>made sure that data pre-processing is correct (i.e. transformations and normalization)</p>
</li>
<li>
<p>I set a random seed using <code>torch.manual_seed()</code></p>
</li>
<li>
<p>I’m using <code>torch.eval()</code> and <code>troch.no_grad()</code> in the test/eval loop</p>
</li>
</ul>
<p>Any idea what I might be missing?<br>
Thanks in advance.</p>
Cut-paste a segment into a region that satisfy a condition

yegane: <p>Hi,<br>
I have two same-size tensors that are binary masks: A, B<br>
I want to relocate the mask that I have in B, in the region(randomly select any location and use it as an anchor) where A’s mask is located, but I don’t know how to implement it in pytorch.</p>
<p>e.g.<br>
A:<br>
[0 0 0 0 0 0 ]<br>
[0 0 1 1 0 0 ]<br>
[0 0 1 1 0 0 ]<br>
[0 0 0 0 0 0 ]</p>
<p>B:<br>
[0 0 0 0 0 0 ]<br>
[0 0 0 0 0 0 ]<br>
[0 0 0 0 1 0 ]<br>
[0 0 0 0 0 1 ]</p>
<p>C=f(A,B):<br>
[0 0 0 0 0 0 ]<br>
[0 0 1 0 0 0 ]<br>
[0 0 0 1 0 0 ]<br>
[0 0 0 0 0 0 ]</p>
<p>or<br>
[0 0 0 0 0 0 ]<br>
[0 0 0 1 0 0 ]<br>
[0 0 0 0 1 0 ]<br>
[0 0 0 0 0 0 ]</p>

yegane: <p>This is the numpy version of it.</p>
<pre><code class="lang-auto">A_boundary=np.where(A==1)
x_pos=np.random.choice(A_boundary[0])
y_pos=A_boundary[1][np.random.choice(np.where(A_boundary[0]==x_pos)[0])]
candid_mask = np.where(candid_segment_i == 1)
x_seg=np.subtract(candid_mask[0],np.min(candid_mask[0]))
y_seg=np.subtract(candid_mask[1], np.min(candid_mask[1]))
for replace_mask in range(len(candid_mask[0])):
    mask[x_pos+x_seg[replace_mask],y_pos+y_seg[replace_mask]]=1
</code></pre>
Normalization/energy conservation on STFT output

JXuan: <p>Hi,</p>
<p>Torchaudio offers the normalization on STFT output, which I think is to restore the energy lost caused by windowing (Parseval’s theorem).</p>
<pre><code class="lang-auto">def spectrogram():
.....
 if normalized:
        spec_f /= window.pow(2.).sum().sqrt()
(https://github.com/pytorch/audio/blob/0076ab073d1ee6160efbc239e075196b35ed850b/torchaudio/functional.py#L95)
</code></pre>
<p>I could not find more detailed explanation about the formula online. Could anyone briefly explain why do we normalize it this way? Is it generic to all window functions or only to the window functions available on torch API?</p>
<p>Besides, I saw people afterwards performing</p>
<pre><code class="lang-auto">S = stft_ouput.pow(2).sum(-1) 
return S
</code></pre>
<p>on the real tensor of STFT output. The last dim of the real tensor containing the real part and the imaginary part. Do you know what is the operation (<code>pow(2).sum(-1)</code>) for? And do we or do we not need <code>sqrt()</code> after <code>pow(2).sum(-1)</code> and before <code>return S</code>?</p>
<p>Many thanks in advance!</p>
Optimising bmm with triangular mask

Sam_Pr: <p>When building autoregressive transformer models, it’s common to have an <code>attn_mask</code> that’s a simple triangular matrix. The core work comes down to this:</p>
<pre><code class="lang-auto"># Masked values are -inf and we add them on
attn_mask = torch.triu(torch.full((sz, sz), float('-inf')), diagonal=1)
attn = torch.bmm(q, k.transpose(-2, -1)) + attn_mask
</code></pre>
<p>Is there a better way to optimise this masked <code>bmm</code> for the specific case of this triangular mask? Nearly half of the <code>bmm</code> output is unused. Compared to an arbitrary mask, it seems that there should be something more optimal that can be done to exploit the structure, but I don’t know whether any existing kernels are capable of this.</p>
CIFAR-100 training on Resnet-18 downloaded from torchvision.models

paganpasta: <p>I am trying to train a resnet-18 downloaded from torchvision model downloaded using the following command<br>
<code>model=torchvision.models.resnet18(pretrained=False, num_classes=100)</code></p>
<p>I am only able to reach an accuracy of 58%. I am using data-augmentations and hyperparameters followed by a lot of <a href="https://github.com/weiaicunzai/pytorch-cifar100" rel="nofollow noopener">projects</a> at github which locally specify the structure of the network instead of using the one from torchvision. I understand that the difference can arise from a lot of reasons.</p>
<p>What I am looking for is to know if there’s a project/set of hyperparameters which I can follow to perform well on the torchvision models. I have observed a similar trend for vgg16_bn as well.</p>

Shiro: <p>Hi</p>
<p><a class="mention" href="/u/paganpasta">@paganpasta</a> did you solve the issue ? I have exactly the same results than you on cifar100.</p>

paganpasta: <p>Sadly no!</p>
<p>However, the difference arises from the reduced number of parameters in the locally defined models by these repositories.</p>

codestar12: <p>I know this reply is very late. The issue is that torchvision models are not meant for cifar datasets with input size 32x32. Because the model reduces the feature map size with each layer block by the end its just too small.</p>
<p>You can see how to fix this in this colab tutorial</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/lightning_examples/cifar10-baseline.ipynb">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220224-060059-RC00_430667398" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/lightning_examples/cifar10-baseline.ipynb" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/lightning_examples/cifar10-baseline.ipynb" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-python">    model = torchvision.models.resnet18(pretrained=False, num_classes=10)
    model.conv1 = nn.Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    model.maxpool = nn.Identity()
</code></pre>
<p>basically, you need to change the stride and replace the maxpool layer with the identity function. This will result in a larger feature map before pooling.</p>
Pythorch representation of gray scale image

Oussama_Bouldjedri: <p>I have a data with a single channel so the shape is numpy array of  250,164 as I want to use it as a gray scale image what would be the difference of :<br>
method 1:   add a dimension in numpy array</p>
<pre><code>         data_numpy = np.expand_dims(data_numpy, axis=2)
</code></pre>
<p>and once on the training loop flip the dimensions to get the proper shape:</p>
<pre><code>        data=data.permute(0, 3, 1,2)       
</code></pre>
<p>method 2: expand the torch dimension before training :</p>
<pre><code>        tensor_data = tensor_data.unsqueeze(dim=0)</code></pre>

ptrblck: <p>Both approaches should yield the same results.<br>
The difference would be that e.g. <code>tensor_data.unsqueeze(0)</code> would be tracked by Autograd etc., but I assume that’s not your use case.</p>

Oussama_Bouldjedri: <p>tracked by Auto grad would you please give more details it might help ?</p>

ptrblck: <p>Here is a small example showing the <code>unsqueeze</code> is differentiable and won’t break the computation graph:</p>
<pre><code class="lang-python">x = torch.randn(1, 1, requires_grad=True)
y = x.unsqueeze(0)
y.mean().backward()
print(x.grad)
# &gt; tensor([[1.]])
</code></pre>
<p>However, this boils down to the differences between PyTorch and numpy.<br>
Assuming you only care about creating a tensor in the right shape, both approaches would be fine.</p>
How to put tensor from linear to conv layer

Samuel_Bachorik: <p>Hello i have this example, how to reshape output from linear layer if i want to put it inside conv2d ?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


x = torch.rand((32,1,28,28))
real_inputs = x.view(-1, 784)

linear = nn.Linear(784, 784)
conv = nn.Conv2d(784, 64, kernel_size=9, stride=1, padding=9 // 2)

x = linear(real_inputs)
x = conv(x)
</code></pre>
<pre><code class="lang-auto">RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 784, 9, 9], but got 2-dimensional input of size [32, 784] instead
</code></pre>

ptrblck: <p>You would need to add the missing spatial sizes.<br>
The output of <code>linear</code> would have the shape <code>[batch_size=32, features=784]</code>.<br>
The <code>conv</code> layer expects an input with <code>784</code> channels, so if you don’t want to <code>repeat</code> the activation you would only be able to <code>unsqueeze</code> the missing dimensions since reshaping won’t work either without changing the batch dimension:</p>
<pre><code class="lang-python">x = torch.rand((32,1,28,28))
real_inputs = x.view(-1, 784)

linear = nn.Linear(784, 784)
conv = nn.Conv2d(784, 64, kernel_size=9, stride=1, padding=9 // 2)

x = linear(real_inputs)
x = x[:, :, None, None]

out = conv(x)
print(out.shape)
# &gt; torch.Size([32, 64, 1, 1])
</code></pre>
How to compute gradient of 2D tensor with respect to 2D tensor?

biswarupb6: <p>For example, I have a 2D tensor Y of size MxN and another 2D tensor X of size Mx3, while N&gt;3.</p>
<p>I want to compute the gradient of Y with respect to X.</p>
<p>I can compute as follows:</p>
<pre><code class="lang-auto">for i in range(Y.shape[1]):
        dY_dX[i,:,:] = grad(Y[:,i],X)
</code></pre>
<p>I have the following gradient function that I am using:</p>
<pre><code class="lang-auto">def grad(outputs, inputs):
	return torch.autograd.grad(outputs, inputs, grad_outputs=torch.ones_like(outputs), create_graph=True)[0]
</code></pre>
<p>The shape of dY_dX should be NxMx3. With the above-mentioned operation, it is possible to compute the gradients, however, it is computationally expensive. Can anyone recommend a possible faster way to compute the same without the loop?</p>
Each epoch gives the same output

TIANREN_ZHAO: <p>Here is my model and my training process, I don’t think my model is learning since it gives me the same output every epoch. Can someone help me out, please?</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self, num_classes=10):
        super(Net, self).__init__()
        self.conv1 = GCNConv(2, 16)
        self.conv2 = GCNConv(16, 32)
        self.conv3 = GCNConv(32, 48)
        self.conv4 = GCNConv(48, 64)
        self.conv5 = GCNConv(64, 96)
        self.conv6 = GCNConv(96, 128)
        self.linear1 = torch.nn.Linear(128,64)
        self.linear1.weight.data.normal_(0, 0.005)
        self.linear2 = torch.nn.Linear(64,num_classes)
        self.linear2.weight.data.normal_(0, 0.005)
    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x = self.conv1(x, edge_index)
        x = F.relu(x)
        x = self.conv2(x, edge_index)
        x = F.relu(x)
        x = self.conv3(x, edge_index)
        x = F.relu(x)
        x = self.conv4(x, edge_index)
        x = F.relu(x)
        x = self.conv5(x, edge_index)
        x = F.relu(x)
        x = self.conv6(x, edge_index)
        x = F.relu(x)
        x, _ = scatter_max(x, data.batch, dim=0)
        x = self.linear1(x)
        x = F.relu(x)
        x = self.linear2(x)
        return x
</code></pre>
<p>And this is my training process</p>
<pre><code class="lang-auto">use_cuda=torch.cuda.is_available()
device = torch.device("cuda" if use_cuda else "cpu")
model = Net().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE,momentum=MOMENTUM,weight_decay=WEIGHT_DECAY)
criterion = nn.CrossEntropyLoss()

for epoch in range(epoch_num):
    train_loss = 0.0
    model.train()
    for i in range(batch_number):
        batch=train_list[i][1].to(device)
        optimizer.zero_grad()
        outputs = model(batch)
        loss = criterion(output,batch.t)
        loss.backward()
        optimizer.step()


correct = 0
total = 0
batch_num = 0
loss = 0
model.eval()
    with torch.no_grad():
        for data in testloader:
            data = data.to(device)
            outputs = model(data)
            loss += criterion(outputs.embeddings[-1],data.t)
            _, predicted = torch.max(outputs.embeddings[-1], 1)
            total += data.t.size(0)
            batch_num += 1
            correct += (predicted == data.t).sum().cpu().item()
</code></pre>

Dazitu616: <p>You forget to do <code>optimizer.step()</code> to update the model parameters. The training code should be something like:</p>
<pre><code class="lang-auto">        batch=train_list[i][1].to(device)
        optimizer.zero_grad()
        outputs = model(batch)
        loss = criterion(output,batch.t)
        loss.backward()
        # here is the missing code!
        optimizer.step()
</code></pre>

TIANREN_ZHAO: <p>oh sorry, it is I forgot to put this line here, it is actually in the code.</p>

Dazitu616: <p>That’s strange. Maybe you can print the weight of some layers out and see if it really changes or not.</p>
Channel wise fully connected layers without weight sharing

bananenpampe: <p>I have data of shape (N_samples,N_channels,N_features_per_channel)<br>
I would like to implement a layer, where each channel is fully connected to a set of output nodes, and there is no weight sharing between the channels weights.</p>
<p>Can this achieve what I am looking for ?</p>
<pre><code class="lang-auto">nn.Conv1d(in_channels=N_channels, out_channels=N_channels_out, kernel_size=1, groups=N_out_features_per_channel)
</code></pre>
Setup.py not copying config files into d2go/configs directory

GilbertTanner: <p>When running D2Go in Google Colab the <code>setup.py</code> file doesn’t copy the <code>configs</code> directory from the d2go folder into the d2go/d2go folder correctly, which leads to the following error when trying to load the model.</p>
<pre><code class="lang-auto">RuntimeError: &lt;config path&gt; not available in Model Zoo!
</code></pre>
<p>This error comes from the the <code>get_config_file</code> method that attempts to load the config file from the <code>d2go/d2go/configs</code> folder.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/facebookresearch/d2go/blob/5aadaaa4b30d19fe8a86970a64aab007dbad6dab/d2go/model_zoo/model_zoo.py#L44-L58">
  <header class="source">

      <a href="https://github.com/facebookresearch/d2go/blob/5aadaaa4b30d19fe8a86970a64aab007dbad6dab/d2go/model_zoo/model_zoo.py#L44-L58" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/d2go/blob/5aadaaa4b30d19fe8a86970a64aab007dbad6dab/d2go/model_zoo/model_zoo.py#L44-L58" target="_blank" rel="noopener nofollow ugc">facebookresearch/d2go/blob/5aadaaa4b30d19fe8a86970a64aab007dbad6dab/d2go/model_zoo/model_zoo.py#L44-L58</a></h4>



    <pre class="onebox">      <code class="lang-py">
        <ol class="start lines" start="44" style="counter-reset: li-counter 43 ;">
            <li>def get_config_file(config_path):</li>
            <li>    """</li>
            <li>    Returns path to a builtin config file.</li>
            <li>    Args:</li>
            <li>        config_path (str): config file name relative to d2go's "configs/"</li>
            <li>            directory, e.g., "COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml"</li>
            <li>    Returns:</li>
            <li>        str: the real path to the config file.</li>
            <li>    """</li>
            <li>    cfg_file = pkg_resources.resource_filename(</li>
            <li>        "d2go", os.path.join("configs", config_path)</li>
            <li>    )</li>
            <li>    if not os.path.exists(cfg_file):</li>
            <li>        raise RuntimeError("{} not available in Model Zoo!".format(config_path))</li>
            <li>    return cfg_file</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>After adding some debug code the method I found out the the <code>configs</code> folder doesn’t exist after installing d2go but until now I was unable to find the cause of this error. I suspect that it’s caused by the <code>setup.py</code> file  and the changes made in c12469c20a2b3c3d041253ba9903e04eecdc4c89 though.</p>
<p>Also the <code>setup.py</code> file from Detectron2 looks almost identical but there the copying of the configs is working without any issues.</p>
<p>Any suggestions are greatly appreciated.</p>
<p>Github issue: <a href="https://github.com/facebookresearch/d2go/issues/174" rel="noopener nofollow ugc">d2go #174</a></p>
transforms.RandomCrop() has no pad_if_needed attribute

mariosfourn: <p>I have torch 0.4.0 and torchvision 0.2.0<br>
According to the documentation the <code>torchvision.transforms.RandomCrop()</code> class should have a <code>pad_if_needed</code> attribute. However, when I try it I get an error:</p>
<p><code>TypeError: __init__() got an unexpected keyword argument 'pad_if_needed'</code></p>

xxaq66: <p>you may  check your conda env <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"></p>
Gradient clipping for one of two losses when using AMP

eliaz: <p>Hi !<br>
I use two losses during training, and I want to clip the gradient produced by one of the two.<br>
When using mixed-precision, is the following script the best way to handle this problem:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
torch.manual_seed(0)
tens = torch.randn(32, 100, device='cuda')
gt = torch.randn(32, 100, device='cuda')
model = nn.Linear(100, 100).cuda()
l1 = nn.L1Loss().cuda()
l2 = nn.MSELoss().cuda()
scaler = torch.cuda.amp.GradScaler()
opt = torch.optim.SGD(model.parameters(), lr=0.1)

model.zero_grad()
with torch.cuda.amp.autocast():
  out = model(tens)
  loss1 = l1(gt, out)
  loss2 = l2(gt, out)
scaler.scale(loss1).backward(retain_graph=True)
scaler.unscale_(opt)
torch.nn.utils.clip_grad_norm_(model.parameters(), 0.1)
model.weight.grad = scaler.scale(model.weight.grad)
scaler.update()
scaler.scale(loss2).backward()
scaler.unscale_(opt)
scaler.update()
</code></pre>
<p>I am unsure about this script as I have to manually scale the gradient, and perform un update on the scaler before calling <code>.step</code></p>
<p>Thanks for your help !</p>
Does Pytorch have pytorch version of implemention of Hydranet?

hitbuyi: <p>The paper of Hydranet is<br>
Mullapud HydraNets Specialized Dynamic, Ravi Teja Mullapudi,etc<br>
I have found classic pytorch version of VGG,resnet, BUT Hydranet is not found in pytorch lib</p>
<p>the github had ones,but it seems not completed yet</p>
GPU-util is 0% on YOLOv3 training

enmanuelmag: <p>Hi! I’m trying to train a YOLOv3. I have  RTX 3060Ti. I can performance all the training but the use of GPU is ~0%. The source code of my model, training and dataset logic is <a href="https://github.com/enmanuelmag/YOLOv3" rel="noopener nofollow ugc">here</a></p>
<p>I found some reference to change the load of items, but I can’t understand entirely how to make the changes. Thanks for your help <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0acf31cb86958e21b91d5e7bbfca0775d452d00.png" data-download-href="https://discuss.pytorch.org/uploads/default/d0acf31cb86958e21b91d5e7bbfca0775d452d00" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0acf31cb86958e21b91d5e7bbfca0775d452d00.png" alt="image" data-base62-sha1="tM1R7tKK445mSNidtIX8ZkXvbyw" width="382" height="500" data-dominant-color="F5F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">555×725 14.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I don’t think your current outputs show the wanted utilization and you should be able to select a “Compute” tab in task manager (I’m not deeply familiar with Windows). Alternatively, use <code>nvidia-smi</code> to check the GPU util.</p>

enmanuelmag: <p>Yes, I search more options and found that Task Manager cna show CUDA usage<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00f82ff55f42927cb092ea6f636537e2e7608748.png" alt="image" data-base62-sha1="8zJOp4GbUuUN0bLk2goXPTuwoo" width="268" height="202"></p>