akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Use torch.multiprocessing.Queue with torchrun?

hexken: <p>Is it possible to use the multiprocessing Queue to communicate between processes launched with torchrun?</p>
<p>I set up ‘a’ Queue and launch the processes like so:</p>
<p>test.py</p>
<pre><code class="lang-auto">import torch.distributed as dist
from torch.multiprocessing import Queue

if __name__ == "__main__":
    local_rank = int(os.getenv("LOCAL_RANK", 0))
    world_size = int(os.getenv("WORLD_SIZE", 1))
    rank = int(os.getenv("RANK", 0))


    is_distributed = world_size &gt; 1
  
    queue = Queue()

    if is_distributed:
        dist.init_process_group(backend="gloo", rank=rank, world_size=world_size)

   .....
</code></pre>
<p>launch.sh</p>
<pre><code class="lang-auto">#!/bin/bash
#
export OMP_NUM_THREADS=1

torchrun \
    --standalone \
    --nnodes=1 \
    --nproc_per_node=4 \
    --master_addr=$(hostname)\
    --master_port=34567 \
    test.py \
</code></pre>
<p>and find that each rank has an independant Queue… Is it possible to share the queue between processes launched using torchrun?</p>
New open source tool for repeatable experiments

mmulet: <p>I made a new open source tool called JellyML that lets you go back to any of your checkpoints, and reproduce your code exactly as it was when you trained it.<br>
You can find the website here:</p>
<p><a href="https://jellyml.com/" rel="noopener nofollow ugc">https://jellyml.com</a></p>
<p>The GitHub repo:</p>
<p><a href="https://github.com/mmulet/jellyml" rel="noopener nofollow ugc">https://gitHub.com/mmulet/jellyml</a></p>
<p>You can install it with pip:</p>
<p>pip install jellyml</p>
Torch Subset list indexing not working

0piero: <p>I checked the source of the Subset class (<a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#Subset" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data.dataset — PyTorch 1.13 documentation</a>) and saw that passing a list to a subset object like subset.<strong>getitem</strong>([1,2,3]) should work, but actually it’s not:<br>
…/dataset.py", line 272, in <strong>getitem</strong><br>
return self.dataset[self.indices[idx]]<br>
TypeError: list indices must be integers or slices, not list</p>

ptrblck: <p>It seems to work for me:</p>
<pre><code class="lang-python">dataset = TensorDataset(torch.arange(10))
subset = Subset(dataset, indices=list(range(0, 10, 2)))

for data in subset:
    print(data)
# (tensor(0),)
# (tensor(2),)
# (tensor(4),)
# (tensor(6),)
# (tensor(8),)

data = subset.__getitem__([0, 1, 3])
print(data)
# (tensor([0, 2, 6]),)

data = subset[[0, 1, 3]]
print(data)
# (tensor([0, 2, 6]),)

data = subset[3]
print(data)
# (tensor(6),)
</code></pre>
<p>This feature was added in June 2021 in <a href="https://github.com/pytorch/pytorch/commit/7c29ca7f2b49a91dbfc6ee35bf6da55616bd2960">this PR</a> so you might need to update PyTorch in case you are using an older release.</p>

0piero: <p>Sorry, I was indeed using an older Pytorch version (1.7.0), thanks for the reply</p>
Use traced regularization function during training

zak_bel: <p>Hi, I am having doubt about using a traced function during training. So I made this regularization function that is quite computationally expensive and slows down training (despite vectorizing it). I tried optimizing it using JIT so I trace it and use it during training. Is it safe to use that way even if JIT is more for inference?</p>

ptrblck: <p>Scripting the model is not specific to inference use cases and training should work just fine.<br>
Depending on the used backend less aggressive optimizations might be used (since e.g. intermediate activations might be stored or recomputed), but the training should not break.</p>

zak_bel: <p>Thank you for the response! I am not sure I fully understand the internal dynamics of Pytorch during training, but what you said implies that autograd operates on on traced/scripted functions too?</p>

ptrblck: <p>Yes, Autograd will still work and will calculate the same gradients (up to floating point precision).<br>
You can run a quick check as seen in this small example:</p>
<pre><code class="lang-python">device = "cuda"
model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 10)
).to(device)

x = torch.randn(1, 10, device=device)

# reference
out = model(x)
out.mean().backward()

# store a gradient as the reference value
g0 = model[0].weight.grad.clone()
model.zero_grad()

# script
model = torch.jit.script(model)

# run a few iterations to allow JIT optimizations
for _ in range(5):
    model.zero_grad()
    out = model(x)
    out.mean().backward()
    
g1 = model[0].weight.grad.clone()

print((g0 - g1).abs().max())
# tensor(0., device='cuda:0')
</code></pre>
<p>Make sure to execute the scripted model a few times to allow some JIT optimizations to kick in.</p>

zak_bel: <p>Thank you very much for the quick response! It is very helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Models from torch_geomtric.nn vs manually implemented Models in PyTorch

feltface: <p>I am trying to use the built-in models from <code>torch_geometric.nn</code> module. I have also seen many tutorial that most of the models are designed manually by extending the <code>torch.nn.Module</code> class.</p>
<p>What is the difference between the following:</p>
<pre><code class="lang-auto">from torch_geometric.nn import GCN
dataset = TUDataset(root=data_root, name=data_name)
train_dataset = dataset[:800]
test_dataset = dataset[800 :]
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)
model = GCN(dataset.num_node_features, 64, 3, dataset.num_classes)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">from torch_geometric.nn import GCNConv, global_mean_pool, Linear
class MyGCN(torch.nn.Module):
    """GCN"""
    def __init__(self, dim_h):
        super(MyGCN, self).__init__()
        self.conv1 = GCNConv(dataset.num_node_features, dim_h)
        self.conv2 = GCNConv(dim_h, dim_h)
        self.conv3 = GCNConv(dim_h, dim_h)
        self.lin = Linear(dim_h, dataset.num_classes)

    def forward(self, x, edge_index, batch):
        # Node embeddings 
        h = self.conv1(x, edge_index)
        h = h.relu()
        h = self.conv2(h, edge_index)
        h = h.relu()
        h = self.conv3(h, edge_index)

        # Graph-level readout
        hG = global_mean_pool(h, batch)

        # Classifier
        h = F.dropout(hG, p=0.5, training=self.training)
        h = self.lin(h)
        
        return hG, F.log_softmax(h, dim=1)
</code></pre>
<p>Also, what is the relation between <code>hidden_channels</code> and <code>batch_size</code>? I cannot find something specific that describes the difference between the two.</p>

ptrblck: <p>The PyG implementation of <code>GCN</code> seems to define a few additional methods (such as <code>inference</code>) and seems to abstract the actual <code>GCN</code> class by deriving from <code>BasicGNN</code> as seen <a href="https://github.com/pyg-team/pytorch_geometric/blob/8b0fab59351290d1895480a23235aa8eafbdc187/torch_geometric/nn/models/basic_gnn.py#L30-L298">here</a>.<br>
<code>hidden_channels</code> seems to be used to initialize the <code>self.convs</code> while the <code>batch_size</code> is used during the <code>inference</code> call while iterating the <code>loader</code> to slice the outputs.<br>
<a class="mention" href="/u/rusty1s">@rusty1s</a> can correct me if I’m missing something as he is the core developer.</p>
Link to Presentation: PyTorch 2.0 Q&A: Rethinking Data Loading with TorchData

miladm: <p>Would you be able to share the link to this great talk: PyTorch 2.0 Q&amp;A: Rethinking Data Loading with TorchData?</p>

ptrblck: <p>Here it is:</p><div class="youtube-onebox lazy-video-container" data-video-id="65DvI3YrFW8" data-video-title="PyTorch 2.0 Q&amp;A: Rethinking Data Loading with TorchData" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=65DvI3YrFW8" target="_blank" rel="noopener">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/65DvI3YrFW8/hqdefault.jpg" title="PyTorch 2.0 Q&amp;A: Rethinking Data Loading with TorchData" width="480" height="360">
  </a>
</div>
CrossEntropyLoss on Sequence of Class Probabilities

J_Johnson: <p>Wondering what the best practices are. Model gives out [batch_size, sequence_length, class_probabilities]. And I would like to apply CrossEntropyLoss on the class probabilities and sum up that loss over the sequence length before back prop.</p>
<p>So my question is should I loop over the sequence length and sum the losses, or is there a better way?</p>

ptrblck: <p>I’m unsure if I understand the use case correctly, but would this work?</p>
<pre><code class="lang-python">batch_size = 2
sequence_length = 3
class_probabilities = 4

logits = torch.randn([batch_size, sequence_length, class_probabilities], requires_grad=True)
targets = torch.randint(0, class_probabilities, (batch_size, sequence_length))

criterion = nn.CrossEntropyLoss(reduction="none")

logits = logits.permute(0, 2, 1) # permute into [batch_size, nb_classes, seq_len]
loss = criterion(logits, targets)
print(loss.shape)
# torch.Size([2, 3]) # [batch_size, sequence_length]
loss = loss.sum(1) # this will sum in the sequence_length dim

# reduce the loss further e.g. via mean or call backward with the gradient argument
loss.mean().backward()
</code></pre>

J_Johnson: <p>That actually worked. Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
Trouble installing pytorch cuda WinError 126

JJ_Siu: <p>Hi, I used to run torch cuda 11.6 in an anaconda environment and it worked perfectly. I had to uninstall and reinstall ananconda, and after reinstalling pytorch, I just couldn’t get it to work properly and kept running into the issue:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f01420e54c538afba90406eb9fe2ace274bb43d1.png" data-download-href="https://discuss.pytorch.org/uploads/default/f01420e54c538afba90406eb9fe2ace274bb43d1" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f01420e54c538afba90406eb9fe2ace274bb43d1.png" alt="Capture" data-base62-sha1="yfPIw3H7aawYr9S7K95Fmq12Q13" width="690" height="43" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">911×57 2.31 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I used the following command to install pytorch:</p>
<p>conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia</p>
<p>Does anyone know how I could possible fix?</p>
How can I train GPT-2 with ecological signals and generate pseudo-ecological signals?

YuhskeHujisaki: <p>The GPT-2 model first performs 1,000 training steps for each of the 20 channels of EEG data in each class. There is a paper that says to then generate synthetic data for n classes using n GPT-2s. However, I don’t understand how to do this. Is it possible for someone to teach me since the method of generation is not written in the paper?</p><aside class="onebox pdf" data-onebox-src="https://jordanjamesbird.com/publications/Synthetic-Biological-Signals-Machine-Generated-by-GPT-2-Improve-the-Classification-of-EEG-and-EMG-Through-Data-Augmentation.pdf">
  <header class="source">

      <a href="https://jordanjamesbird.com/publications/Synthetic-Biological-Signals-Machine-Generated-by-GPT-2-Improve-the-Classification-of-EEG-and-EMG-Through-Data-Augmentation.pdf" target="_blank" rel="noopener nofollow ugc">jordanjamesbird.com</a>
  </header>

  <article class="onebox-body">
    <a href="https://jordanjamesbird.com/publications/Synthetic-Biological-Signals-Machine-Generated-by-GPT-2-Improve-the-Classification-of-EEG-and-EMG-Through-Data-Augmentation.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://jordanjamesbird.com/publications/Synthetic-Biological-Signals-Machine-Generated-by-GPT-2-Improve-the-Classification-of-EEG-and-EMG-Through-Data-Augmentation.pdf" target="_blank" rel="noopener nofollow ugc">Synthetic-Biological-Signals-Machine-Generated-by-GPT-2-Improve-the-Classification-of-EEG-and-EMG-Through-Data-Augmentation.pdf</a></h3>

  <p class="filesize">2.08 MB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Apply `MultiheadAttention`'s heads to same input

erezinman: <p>Hi,<br>
My question surely has a simple answer, but I couldn’t find it. I<br>
wish to apply <code>MultiheadAttention</code> to the same sequence without copying the sequence. My data is temporal data with dimensions (batch, time, channels). I treat the “channels” dimension as the embedding, and the time dimension as the sequence dimension. For example:</p>
<pre><code class="lang-python">N, C, T = 2, 3, 5
n_heads = 7
X = torch.rand(N, T, C)
</code></pre>
<p>Now, I want to apply 7 different heads as self-attention to the same input <code>X</code>, but as far as I understand, it attrequires me to copy the data 7 times:</p>
<pre><code class="lang-python">attn = torch.nn.MultiheadAttention(C * n_heads, n_heads, batch_first=True)
X_ = X.repeat(1, 1, n_heads)
attn(X_, X_, X_)
</code></pre>
<p>Is there any way to do this without copying the data 7 times?<br>
Thanks!</p>
Converting Video Vision Transformer(TimeSformer,ViViT) for Regression Tasks

Taimoor-R: <p>I have been trying to use TimeSformer and ViViT, I have managed to convert it into a regression model by changing the loss function and setting the output of the mlp to 1. However what i understand is that a video vision transformer takes a video clip as an input(broken into frames) and outputs a single value corresponding to that input clip. I would like the model to output a value for each frame of the clip input so instead of outputing 1 value it outputs 32 values. Can you guide me in this regards.</p>
Normalization with small batches

hadarshavit: <p>Hi,<br>
I have an linear layer that takes as an input a 32 (large) items, for example, with batch size of 2:</p>
<pre><code class="lang-auto">tensor([[ 1477.7378,  2719.6460,  1040.3798,  1215.5387,  3542.7241,   266.8930,
          5944.8560, -2105.2986,  4023.8545,  -932.9826,  1617.7998,  4028.0327,
         -2225.9929, -1581.6738, -3424.4250, -1329.6610,  5483.1172,  5975.4663,
          4483.0156,  -373.3383,   482.2250,  -557.7921,  1385.3356,   196.1219,
          -375.9175,  1787.6195, -3516.9705,  -687.9647,  2663.0601,  -576.0363,
            86.2473,  5765.6387],
        [ 1744.0067,  3214.6289,  1228.2401,  1436.6631,  4191.4390,   318.6791,
          7028.2715, -2485.2910,  4753.7217, -1105.6101,  1920.9307,  4758.9941,
         -2634.7754, -1861.2491, -4049.8877, -1578.7089,  6478.5850,  7065.4326,
          5293.2305,  -442.4786,   566.5157,  -658.5790,  1643.0204,   237.8128,
          -450.6484,  2104.6516, -4151.9263,  -815.9917,  3144.3291,  -675.0918,
            94.2207,  6817.9795]])
</code></pre>
<p>Due to the big numbers, I’m trying to normalize the inputs. However, I’m using batch size of 4 which makes batch normalization very unstable during training (as it uses mean and variance of the batch).<br>
I tried using layer normalization, but it results in very similar outputs for different inputs, for example for the values above:</p>
<pre><code class="lang-auto">tensor([[ 0.1271,  0.5954, -0.0378,  0.0283,  0.9057, -0.3294,  1.8113, -1.2237,
          1.0871, -0.7818,  0.1799,  1.0886, -1.2692, -1.0263, -1.7211, -0.9313,
          1.6372,  1.8229,  1.2602, -0.5708, -0.2482, -0.6403,  0.0923, -0.3561,
         -0.5717,  0.2440, -1.7560, -0.6894,  0.5740, -0.6472, -0.3975,  1.7437],
        [ 0.1264,  0.5955, -0.0382,  0.0283,  0.9070, -0.3283,  1.8119, -1.2227,
          1.0864, -0.7826,  0.1828,  1.0881, -1.2704, -1.0236, -1.7218, -0.9335,
          1.6366,  1.8238,  1.2585, -0.5711, -0.2492, -0.6400,  0.0942, -0.3541,
         -0.5737,  0.2414,  -1.7543, -0.6902,  0.5730, -0.6453, -0.3999,  1.7449]])

</code></pre>
<p>This makes it impossible to differentiate between the two examples.<br>
Is there a normalization layer that suits this problem?<br>
Maybe something that performs per feature normalization (like batch normalization) but uses the running mean and variance?<br>
Thanks!</p>

tataganesh: <p>Could you elaborate more on why batch normalization causes training to be unstable when batch size is 4?</p>

hadarshavit: <p>Batch normalization uses the mean and the variance of each feature in the batch. 4 examples are not representative of the full dataset (and can be very far from the mean and the variance of the full dataset)</p>

tataganesh: <p>And when you say “different inputs”, do you mean different inputs to layer norm, or different inputs to the network?</p>

hadarshavit: <p>Different inputs to the layer norm (the output of the layer before)</p>
Sampling from a PyTorch dataset

papoo13: <p>I would like to randomly sample from a PyTorch dataset and create a PyTorch dataset from the samples.<br>
Currently, I am using data.random_split but I’m not sure if this is the best approach.<br>
This is the code:</p>
<pre><code class="lang-auto">def generate_random_coreset(dataset, num_samples):
    # dataset is a pytorch dataset
    return data.random_split(dataset, [len(dataset) - num_samples, num_samples])
</code></pre>
<p>Does PyTorch offer a possible function to solve this?<br>
I would appreciate any guidance and hits given on this question.</p>

srishti-git1110: <p>Could you please describe your use case in more detail? Also please checkout <code>torch.utils.data.DataLoader</code> which allows automatic batched loading of data possibly using parallel worker processes.</p>

papoo13: <p>Thank you for the reply.<br>
My use case is creating a coreset (or memory) from the task dataset in a continual learning setting. I would like to do the sampling before using the dataloaders.</p>

tataganesh: <p>You can use <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.RandomSampler" rel="noopener nofollow ugc">torch.utils.data.RandomSampler</a> to create a Random Sampler for your Dataloader.<br>
E.g.</p>
<pre><code class="lang-auto">random_sampler = data.RandomSampler(dataset, num_samples=num_samples)
dataloader = data.DataLoader(dataset, batch_size=k, sampler=random_sampler)
</code></pre>
<p>Note that use can pass a <a href="https://pytorch.org/docs/stable/generated/torch.Generator.html#torch.Generator" rel="noopener nofollow ugc">generator</a> object  to <code>RandomSampler</code> to get the same subset of random samples every time.</p>
Constraint the numeric outputs softly

Hyunjun_Cho: <p>I want the average of multiple input numbers and output numbers to follow a specific number.<br>
Of course, there are several exisiting methods such as L1 and MSE. But since I’m working on the task dealing with long sequences, I want to induce the model to follow that particular number.<br>
Suppose a given number is 0.3, and the average is 0.4. Then, instead of making the mean precisely 0.3, I would like to give the model a kind of penalty so that it can infer fewer values.</p>
<p>Thank you</p>

tataganesh: <p>Your input and output is not very clear to me. Can you provide an example input sequence, the expected output and how the output is computed in this case?</p>
Questions about batch training with dictionary index

ChineseBest: <p>Hi, I have a quick question about how to use batch training in my data as acclecration.</p>
<p>My network contians input layer with different dimensions, and I use a index for each data to indicate which layer will the data be fed. However, if I intend to batch all of these data, are there any approaches I can use for batch training? If I directly use a list, the error I received is:</p>
<p>TypeError: unhashable type: ‘list’</p>
<p>Could you please help me? Thanks a lot!</p>

Samuel_Bachorik: <p>Hi can you show model, code, training loop…</p>

ChineseBest: <p>Hi, emm My code is like:</p>
<p>out = model(emblist, graphlist, graphindexlist)<br>
all the *list represent list, and this model is a graph neural network (based on pyg).</p>
<p>Thanks.</p>
Seq2Seq regression problem with attention

WowPy: <p>Hello,</p>
<p>I wrote the following code to solve a Seq2Seq regression problem. My implementation is based on the GRU and multi-head attention. The performance is horrible. I tried playing with the hyperparameters, but nothing changed. This led me to think it was a network architecture issue.</p>
<pre><code class="lang-auto">class Seq2Seq(nn.Module):
    def __init__(self, input_size, output_size, hidden, num_heads):
        super(Seq2Seq, self).__init__()
        self.encoder = nn.GRU(input_size, hidden, 2)
        self.decoder = nn.GRU(hidden, hidden, 2)
        self.multihead_attn = nn.MultiheadAttention(hidden, num_heads)
        self.linear  = nn.Linear(hidden, output_size)
        self.init_weights()

    def init_weights(self):
        self.linear.weight.data.normal_(0, 0.1)

    def forward(self, x):
        encoded, _ = self.encoder(x)
        decoded, _ = self.decoder(encoded)
        attention_output, _ = self.multihead_attn(decoded, decoded, decoded)
        out = self.linear(attention_output)
        return out

D_in    = 4
D_out   = 1
hidden = 16
num_heads = 4
seq2seq = Seq2Seq(input_size=D_in, output_size=D_out, hidden=hidden, num_heads=num_heads)
inputs = torch.rand((7, 100, D_in))
outputs = seq2seq(inputs)

</code></pre>
<p>Any suggestions are highly appreciated</p>

vdw: <p>I don’t have a real answer, just some food for thoughts:</p>
<ul>
<li>
<p>I’m not sure intuitive it is to use <code>nn.MultiHeadAttention</code> on the output of a <code>nn.GRU</code>. <code>nn.MultiHeadAttention</code> basically implements self-attention which generally assumes that the sequence elements are “independent” like word (vectors). However the output of a <code>nn.GRU</code> is different as the output at step T captures to some extend the outputs from all previous steps (T-1).</p>
</li>
<li>
<p>At lease from by basic experience, transformers a difficult to train from scratch; usually you use pretrained models.</p>
</li>
<li>
<p>Strictly speaking your model does not implement a Seq2Seq task but a sequence labeling tasks. i.e., you get an output for each input word/item. I actually can see what kind of regression problem you’re trying to solve</p>
</li>
<li>
<p>Have you tried a more basic model by just using the <code>nn.GRU</code>? How do the results compare? It’s often better to first try a simple architectures and then extending it.</p>
</li>
</ul>

WowPy: <p><a class="mention" href="/u/vdw">@vdw</a>, many thanks for the insights. When I did not use multi-head attention, I got better results. Thanks again for the useful points.</p>
Batch size and DistributedDataParallel

Anita: <p>Hi,</p>
<p>My question is that: does the DataLoader batch_size parameter actually represents the batch_size per process (or per GPU) in Pytorch DistributedDataParallel? Or is DataLoader batch_size parameter divided by the world_size and each get a subportion of that batch size?</p>
<p>According to my implementation for 1 node with 3 GPUs, when I set batch_size to 1, then each process loads the input with a batch dimension as 1, while when I change batch_size to 3, then each process loads the input with batch dimension as 3. On the second approach, should the batch size have been divided on each process/GPU and each get input with batch size dimension as 1?</p>
<p>I am a beginner in PyTorch and just wanted to make sure that my implementation working this way is correct. My model architecture runs with only batch_size=1, and also wanted to make sure that each GPU loads the data correctly and on a node level, I will then end up with batch_size of 3.</p>

eqy: <p>I believe this is similar to the ImageNet example, which divides the total batch size by the number of GPUs:</p>
<p><a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py#L160" rel="noopener nofollow ugc">examples/main.py at main · pytorch/examples (github.com)</a></p>
how to get correct typing from nn.Module.__call__ ?

RmZeta2718: <p><strong>What I want</strong>: correct typing when I call nn modules</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
linear = nn.Linear(3, 4)
tensor = torch.ones(3)
out = linear(tensor)
reveal_type(out)  # Expected: torch.Tensor; Got: Any
</code></pre>
<p>IDEs (like VSCode or PyCharm) and linters (like <code>mypy</code>) don’t get expected typing.</p>
<p>I’ve found some workaround <a href="https://discuss.pytorch.org/t/adding-typing-to-call-of-nn-module/118295">here</a>, by adding typing to <code>__call__</code> . This works fine when creating my own module, but I can’t do it for <code>nn.Linear</code> and all other modules in <code>nn</code> .</p>
<p>I also accidentally found that back in pytorch 1.4, the typing was correct, because <code>.pyi</code> files were provided back then and it actually utilized the <code>__call__</code> trick above:</p>
<pre><code class="lang-python"># pytorch 1.4
# torch/nn/modules/linear.pyi 
class Linear(Module):
    # several lines skipped ...
    def forward(self, input: Tensor) -&gt; Tensor: ...  # type: ignore
    def __call__(self, input: Tensor) -&gt; Tensor: ...  # type: ignore
</code></pre>
<p>Apparently the pyi file is removed now and <a href="https://github.com/pytorch/pytorch/issues/36915#issuecomment-692999314" rel="noopener nofollow ugc">according to this thread</a> it’s inlined. But the inlined version seems not working.</p>
<p>I’m new to pytorch and that’s all I know, someone could tell me what went wrong?</p>
<p><strong>Environment</strong></p>
<ul>
<li>python 3.9</li>
<li>pytorch 1.13</li>
</ul>

ConvolutionalAtom: <p>A relevant issue: <a href="https://github.com/pytorch/pytorch/issues/74746" rel="noopener nofollow ugc">issue link</a></p>
ValueError: source code string cannot contain null bytes

Yellow-Champagne: <p>D:\Anaconda\envs\py-zhu\python.exe D:/zgy/GCC/GCC-SFCN-master/train.py<br>
Traceback (most recent call last):<br>
File “D:/zgy/GCC/GCC-SFCN-master/train.py”, line 12, in <br>
from tensorboardX import SummaryWriter<br>
ValueError: source code string cannot contain null bytes</p>

yadunandan_kini: <p>Hi,</p>
<p>I have a similar issue. Did you manage to solve the issue?</p>
<p>Thanks.</p>
Adding layers to pretrained model

edna891: <p>Hello,</p>
<p>I’m using a fine-tuned FasterRCNN model for detecting 2D bounding boxes on the image. After that I have to estimate the 3D position of those bounding boxes, for that I have more inputs for example the intrinsic camera parameters. What I want to do is adding layers to the pretrained model. If I want to embed the pretrained model into my new model, the return value of that is only the loss values, see <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/generalized_rcnn.py#L118" rel="noopener nofollow ugc">here</a>.</p>
<p>Example Code of my model extension</p>
<pre><code class="lang-auto">class BoundingBox3DNet(nn.Module):
    def __init__(self, model):
        super(BoundingBox3DNet, self).__init__()

        self.model = model

        self.my_new_layers = nn.Sequential(
            nn.Linear(1000, 100),
            nn.ReLU(),
            nn.Linear(100, 2)
        )

    def forward(self, images, calibs, targets):
        x = self.model(images, targets)

        return x
</code></pre>
<p>In the forward pass I get only the loss values instead of the detections which I want as input for my model. Also I want from the FasterRCNN model the feature maps, how can I also do this?</p>
<p>Thanks<br>
edna891</p>
Avoiding the for loop

WowPy: <p>I have the following implementation</p>
<pre><code class="lang-auto">def T_gradient(T, x):
    gradT = torch.empty((len(x), N, 2))
    for inc in range(N):
        gradT[:, inc, :] = torch.autograd.grad(T[:, inc, :], x, torch.ones((T.size()[0], 1), device=device),create_graph=True, retain_graph=True)[0]
    return gradT

</code></pre>
<p>T is obtained from a seq2seq model which has the size of (number of examples, N, 1), where N is the sequence length. My question is there a way to avoid using the for loop. x has the size of (number of examples, 2)</p>
Learning from label proportions

mayne: <p>Can someone please provide a simple implementation of this based on simple tensors and not large image data? I cannot understand how this works when looking at their code.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://paperswithcode.com/paper/learning-from-label-proportions-by-learning">
  <header class="source">

      <a href="https://paperswithcode.com/paper/learning-from-label-proportions-by-learning" target="_blank" rel="noopener nofollow ugc">paperswithcode.com</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:382/248;"><img src="https://production-media.paperswithcode.com/thumbnails/paper/2203.02496.jpg" class="thumbnail" width="382" height="248"></div>

<h3><a href="https://paperswithcode.com/paper/learning-from-label-proportions-by-learning" target="_blank" rel="noopener nofollow ugc">Papers with Code - Learning from Label Proportions by Learning with Label Noise</a></h3>

  <p>Implemented in one code library.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Is there a layer normalization for Conv2D

Zhang_Chi: <p>is there a layer normalization for 2d feature maps which has the same function with tf.contrib.layers.layer_norm in tensorflow?</p>

smth: <p>from what i see, no one has implemented it yet.</p>

Naruto-Sasuke: <p><a href="https://github.com/pytorch/pytorch/pull/2112" rel="nofollow noopener">LayerNorm</a></p>

Zhang_Chi: <p>if batch size is always 1,can i do it by reshape a (1,C,H,W) tensor to (1,CHW),and then tranpose it to (CHW,1) and apply batchnorm ?  then reshape it back to (1,c,h,w)</p>

loicsacre: <p>Use the <a href="https://pytorch.org/docs/stable/generated/torch.nn.GroupNorm.html" rel="noopener nofollow ugc">GroupNorm</a> as followed:</p>
<p><code>nn.GroupNorm(1, out_channels)</code></p>
<p>It is equivalent with LayerNorm. It is useful if you only now the number of channels of your input and you want to define your layers as such</p>
<pre><code class="lang-auto">nn.Sequential(nn.Conv2d(in_channels, out_channels, kernel_size, stride), nn.GroupNorm(1, out_channels), nn.ReLU())
</code></pre>

harisgulzar1: <p>Now InstanceNorm2d is implemented in pytorch which can be used as LayerNorm for 2DConv.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.InstanceNorm2d.html#torch.nn.InstanceNorm2d" rel="noopener nofollow ugc"><code>InstanceNorm2d</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html#torch.nn.LayerNorm" rel="noopener nofollow ugc"><code>LayerNorm</code></a> are very similar, but have some subtle differences. <a href="https://pytorch.org/docs/stable/generated/torch.nn.InstanceNorm2d.html#torch.nn.InstanceNorm2d" rel="noopener nofollow ugc"><code>InstanceNorm2d</code></a> is applied on each channel of channeled data like RGB images, but <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html#torch.nn.LayerNorm" rel="noopener nofollow ugc"><code>LayerNorm</code></a> is usually applied on entire sample and often in NLP tasks. Additionally, <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html#torch.nn.LayerNorm" rel="noopener nofollow ugc"><code>LayerNorm</code></a> applies elementwise affine transform, while <a href="https://pytorch.org/docs/stable/generated/torch.nn.InstanceNorm2d.html#torch.nn.InstanceNorm2d" rel="noopener nofollow ugc"><code>InstanceNorm2d</code></a> usually don’t apply affine transform.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.InstanceNorm2d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.InstanceNorm2d.html</a></p>
LayerNorm Misunderstanding

Andreas_Binder: <p>Hey guys! Out of interest, I wanted to reimplement the nn.LayerNorm functionality but I cannot wrap my head around a dummy example; I expected the result of both ref and out to be the same. For context, the embedding is supposed to be a single sentence (batch_size = 1) with two words and each word dimension equals to two. Thank you a lot!</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn 

embedding = torch.FloatTensor([[[2,1],[3,4]]])

layer_norm = nn.LayerNorm([2])
ref = layer_norm(embedding)

mean = embedding.mean(dim=(1,2)) 
std = embedding.std(dim=(1,2)) 
out = (embedding - mean) / (std + 1e-5)


out, ref
=&gt;
 (tensor([[[-0.3873, -1.1619],
          [ 0.3873,  1.1619]]]), tensor([[[ 1.0000, -1.0000],
          [-1.0000,  1.0000]]], grad_fn=&lt;NativeLayerNormBackward0&gt;))
</code></pre>

crcrpar: <p>It seems like your script doesn’t apply affine transformation after the normalization (ref: <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html" class="inline-onebox" rel="noopener nofollow ugc">LayerNorm — PyTorch 1.13 documentation</a>).<br>
I think it’d be necessary to initialize weight and bias parameters (gamma and beta in the doc above) and apply them to <code>out</code> in your snippet, e.g. <code>weight * out + bias</code>.</p>

Andreas_Binder: <p>the default values for the weights are only ones and only zeros for the biases, that’s why I left them out or am I mistaken?</p>
PyTorch & RTX4090

samAN2012: <p>Hello,</p>
<p>I would like to setup a computer for deep learning using a RTX4090 GPU (only one for the moment)</p>
<p>Could someone confirm that PyTorch works correclty with this GPU? If so, could you also provide the software configuration that you use ?</p>
<p>I see that the Nvidia official container 22.12-py3 (latest) does not officially support the ada-lovelace architecture, do we have any preliminary date for an official release supporting this GPU ?</p>
<p>Thanks for your support</p>

ptrblck: <p>The PyTorch pip wheels and conda binaries with CUDA 11.7 for nightly and stable releases as well as the (experimental) binaries with CUDA 11.8 will work for your 4090.<br>
To install the binaries with the 11.8 CUDA runtime you would need to manually change the URLs and e.g. replace <code>cu117</code> with <code>cu118</code> as they are not posted in the install instructions yet.</p>
Advice on dataset/dataloader for videos

xu555: <p>Hello great minds,</p>
<p>I need some advice about my dataset/dataloader. Any comments are welcomed!</p>
<p>I have 100 videos.<br>
Every video has around 300 frames.<br>
Every video has its own json file including frame number, <ins>info1</ins> and <ins>info2</ins>.</p>
<p>I would like the dataset/dataloader:</p>
<p>for each video:</p>
<ol>
<li>Take <strong>n frames</strong>(adjacent) (eg. frames[0,n], frames[1,n+1], …, frames[end-n, end])</li>
<li>Check if each of the <strong>n frames</strong> meet <ins>info1</ins> and <ins>info2</ins> request, if yes, take this <strong>n frames</strong> as one sample(input) , if no, ignore it and try with next <strong>n frames</strong>.</li>
</ol>
<p>Continues with other 99 videos.</p>
<p>When loading the dataset, I could either</p>
<ol>
<li>Load two samples from the same video</li>
<li>Load two samples from different videos</li>
<li>Just load one sample</li>
</ol>
<p>It is a mess in my brain about how to arrange the code. Please offer me some advice. Thank you in advance!!!</p>
How to change a slice of a slice of a tensor?

tueboesen: <p>I am trying to use two boolean tensors to select the relevant indices of another tensor which I wish to modify.<br>
The following will run, but it won’t actually change x.<br>
I would like to change the elements selected from x by m1 on m2 and then change those elements in x, but I’m not sure how this should be done?</p>
<pre><code class="lang-auto">import torch
x = torch.zeros(10)
m1 = torch.zeros(10,dtype=torch.bool)
m1[4] = 1
m1[8] = 1
m2 = torch.zeros(m1.sum(),dtype=torch.bool)
m2[-1] = 1

x[m1][m2] -= 1 # I would like for x to now have one element that is nonzero, but it doesn't.
</code></pre>

J_Johnson: <p>This works:</p>
<pre><code class="lang-auto">import torch
x=torch.zeros(10)
y=torch.arange(x.size(0))
m1=torch.zeros_like(x, dtype=torch.bool)

m1[[4,8]]=1
y=y[m1]
x[y[-1]]=-1
print(x)
</code></pre>
<blockquote>
<p>tensor([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0., -1.,  0.])</p>
</blockquote>
Error while importing torch

Vinayaka_Hegde: <p>Hi, I made changes to dataloader.py and worker.py files to add some logging and time_perf analysis. Since then, I’m getting the below error.</p>
<p>I have not changed any other files and have not broken any code.</p>
<p>I think there is a circular import error but unable to figure out what exactly the issue is. I don’t want to update/re-install pytorch again as it takes lot of time to setup again.</p>
<p>Please help me !! Thanks !</p>
<p>PS:<br>
PyTorch is installed in Nvidia Jetson Orin</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b08a6d2eb28f0d2cb2f7c2f078bd056e5191df09.png" data-download-href="https://discuss.pytorch.org/uploads/default/b08a6d2eb28f0d2cb2f7c2f078bd056e5191df09" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b08a6d2eb28f0d2cb2f7c2f078bd056e5191df09.png" alt="image" data-base62-sha1="pbKBXh16jKqiDPQuqtSOr5gxNtD" width="690" height="123" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1924×344 21.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you describe what exactly you have changed, which caused this <code>ImportError</code>?</p>

Vinayaka_Hegde: <p>No worries, I was able to debug it</p>
Torchvision::nms ops cannot be found

Shu_Li: <p>So I am working with <a href="https://github.com/pytorch/android-demo-app/tree/master/ImageSegmentation" rel="noopener nofollow ugc">the Android demo APP for image segmentation with Pytorch</a>.</p>
<p>The demo APP works just fine with deeplabv3.<br>
But instead of using the default deeplabv3 models, I am trying to use <strong>detecton2</strong>.</p>
<p>Here is <strong>what I have achieved and tested successfully in Python</strong>:</p>
<ul>
<li>converted detectron2 to TorchScript using tracing. Validated and got correct inference results.</li>
<li>Optimized the TorchScript model for mobile.</li>
<li>Saved the optimized torchscript model for lite interpreter (.ptl).</li>
</ul>
<p><strong>IDE environment:</strong></p>
<ul>
<li>Android Studio.</li>
<li>pytorch lite version: implementation ‘org.pytorch:pytorch_android_lite:1.13.0’</li>
<li>torchvision lite version: implementation ‘org.pytorch:pytorch_android_torchvision_lite:1.13.0’</li>
</ul>
<p><strong>Error</strong><br>
Following ops cannot be found: [torchvision::nms].<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49f6c121afcfdd1d26f843a27a77e9f3bda9f6b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/49f6c121afcfdd1d26f843a27a77e9f3bda9f6b7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49f6c121afcfdd1d26f843a27a77e9f3bda9f6b7_2_690x151.png" alt="image" data-base62-sha1="ayjwvdskZ6rBZnzQ4pgXdEVQ2gv" width="690" height="151" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49f6c121afcfdd1d26f843a27a77e9f3bda9f6b7_2_690x151.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49f6c121afcfdd1d26f843a27a77e9f3bda9f6b7_2_1035x226.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49f6c121afcfdd1d26f843a27a77e9f3bda9f6b7_2_1380x302.png 2x" data-dominant-color="302D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3733×822 313 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do any one know how to solve this issue?</p>
<p>Any help and ideas will be appreciated.</p>

Shu_Li: <p>Solved this problem by adding extra dependency:</p>
<pre><code class="lang-auto">implementation 'org.pytorch:torchvision_ops:0.13.0'
</code></pre>
<p>To solve the duplicate files, extra packagingOptions are added as well:</p>
<pre><code class="lang-auto">
        pickFirst 'lib/arm64-v8a/libfbjni.so'
        pickFirst 'lib/armeabi-v7a/libfbjni.so'
        pickFirst 'lib/x86/libfbjni.so'
        pickFirst 'lib/x86_64/libfbjni.so'
        pickFirst 'lib/arm64-v8a/libpytorch_jni_lite.so'
        pickFirst 'lib/armeabi-v7a/libpytorch_jni_lite.so'
        pickFirst 'lib/x86/libpytorch_jni_lite.so'
        pickFirst 'lib/x86_64/libpytorch_jni_lite.so'
        exclude 'META-INF/native-image/**'
</code></pre>
<p>Now detectron2 successfully loaded in the demo APP.</p>
Logging/printing once in DistributedDataParallel

dgm: <p>is there a correct way to do, instead of something like checking if the device is 0 ?<br>
e.g. how to acquire a lock ? for logging/printing once on multi-processing settings</p>
<p>Thanks</p>

XWu: <p>I would suggest logging/printing on rank 0 process. Is there any reason not doing this?</p>
Building Torch-vision from Source - Missing CSRC Directory

Ali_Arastehfard: <p>Hello! I am trying to build torch-vision from source. I simply clone the GitHub repository and then run <code>python setup.py install</code>. However, when I go to the install destination I cannot locate the CSRC folder as in here <a href="https://github.com/pytorch/vision/tree/main/torchvision/csrc" rel="noopener nofollow ugc">torchvision/csrc</a>. Am I missing a requirement?</p>

ptrblck: <p>The <code>csrc</code> directory contains the C++ and CUDA source code and won’t be copied to the install folder. The methods should however be available in the built <code>_C.so</code> library and you could check it via e.g. <code>nm -D /PATH/TO/VISION/_C.so | grep -i roi_align</code> or any other kernel name defined in <code>csrc</code>.</p>
Stopping a GPU Process

papoo13: <p>I am running my deep learning code on GPU and my OS is ubuntu. How to stop the code when it is in the middle of the execution so the process stops occupying the GPU? in simpler term, what is the correct way to stop the code? I use ctrl+Z but the processes pile up on the GPU and I need to kill them at some point.<br>
I appreciate your guidance.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/papoo13">@papoo13</a>,</p>
<p>Using <code>Ctrl + Z</code> suspends a job (which can then be resumed). You can cancel a job via <code>Ctrl + C</code>. If you want to resume a job, which has been suspended via <code>Ctrl + Z</code>, you can type <code>fg</code> to resume the job.</p>

papoo13: <aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="2" data-topic="171394">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>can type <code>fg</code> to resume the job</p>
</blockquote>
</aside>
<p>but Ctrl+C only gives me --KeyboardIntrrupt–</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="2" data-topic="171394">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>. If you want to resume a job, which has been suspended via <code>Ctrl + Z</code>, you can type <code>fg</code> to resume the job.</p>
</blockquote>
</aside>
<p><code>Ctrl + C</code> kills a job. Can’t resume that.</p>
<p><code>Ctrl + Z</code> suspends a job, which you can resume via <code>fg</code></p>

papoo13: <p>When I type Ctrl+C, I see the message I mentioned above. So the terminal is inaccessible. What should ones do after typing Ctrl+C and seeing KeyboardIntrupt?</p>
Pytorch FaceNet MTCNN model for facial features

Rohit_Thakur: <p>I am using MTCNN to detect face in an image, FACENET to extract and save features from each detected face and OpenCV Gaussian Blur filter to mask the detected faces.<br>
My end goal is to find the target face in the masked image by comparing saved facial features(or any other method) and unmask target face only. But the features detected by FaceNet for the face in the full image and a cropped face are different.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09138046a26be76767530423294e9632788a2ff2.jpeg" alt="test1" data-base62-sha1="1ii4QdxEC36sxyHGGk3yLd3UkkW" width="420" height="420"></p>
<p>Here is how my code looks:</p>
<pre><code class="lang-auto">def face_and_features(img):
    boxes, _ = mtcnn.detect(img)
    print(boxes)

    for i, box in enumerate(boxes):
        a,b,c,d = box
        x1,y1,x2,y2 = int(a), int(b), int(c), int(d)
        face = img[y1:y2, x1:x2]
        cv2.imwrite(f"face{i}.jpg",face)
        
        face = cv2.resize(face, (160, 160))
        face = face.transpose((2, 0, 1))
        face = torch.from_numpy(face).float()
        face = face.unsqueeze(0)

        features = facenet(face)

        filename = "face_{}.npy".format(i)
        np.save(filename, features.detach().numpy())
        with open("bounding_boxes.txt", "a") as f:
            f.write("{}, {}, {}, {}, {}, {}, {} \n".format(x1, y1, x2, y2, filename, datetime.datetime.now(), frame_number))
    return features
       
def masking(img):
    filename = "bounding_boxes.txt"
    masked_img = img.copy()
    with open(filename,'r') as file:
        for line in file:
            x,y,w,h,f_name,time, f_no = line.split(",")
            x,y,w,h = int(x), int(y), int(w), int(h)
            roi_color = masked_img[y:h, x:w]
            masked_img[y:h, x:w] = cv2.GaussianBlur(roi_color, (51,51), 0)
    return masked_img, f_name, time, f_no

def compare_features(target_face, saved_features):
    target_face = cv2.resize(target_face, (160, 160))
    target_face = target_face.transpose((2, 0, 1))
    target_face = torch.from_numpy(target_face).float()
    target_face = target_face.unsqueeze(0)

    target_features = facenet(target_face)
    filename = "target_face.npy"
    np.save(filename, target_features.detach().numpy())
    target_features = np.load(filename)
    similarity_scores = []
    for i, feature in enumerate(saved_features):
        score = cosine_similarity(target_features, feature)
        # score = distance.euclidean( target_features, feature)
        # score = np.linalg.norm(feature - target_features)
        similarity_scores.append((i, score))
    print(similarity_scores)
    return max(similarity_scores, key=lambda x: x[1])
</code></pre>