akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to get lower triangular and diagonal matrices separately with torch.linalg.ldl_factor

AlphaBetaGamma96: <p>Hi All,</p>
<p><strong>TL;DR</strong> how can I use <code>torch.linalg.ldl_factor</code> to get a lower triangular and diagonal matrix separately rather than a compact representation with pivoting?</p>
<p>I have a question about how exactly <code>torch.linalg.ldl_factor</code> is used to extract the lower triangular and diagonal components of a matrix. I understand a compact representation involving pivoting is used, but I’m not familiar with it.</p>
<p>For example, the <a href="https://en.wikipedia.org/wiki/Cholesky_decomposition" rel="noopener nofollow ugc">Wiki article</a> on Cholesky decompositions has an LDL decomposition example,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03f17a32be4404b0a90f5175ed31a7ac4d7da2b0.png" alt="image" data-base62-sha1="ySNiP2grlCPBbFxakCx6ZKlb3O" width="683" height="137"></p>
<p>However, when computing this within pytorch it gives a different representation. For example,</p>
<pre><code class="lang-auto">matrix = torch.tensor([[4,12,-16],
                       [12,37,-43],
                       [-16,-43,98]], dtype=torch.float32)
LD, pivot = torch.linalg.ldl_factor(matrix)
print(LD)
"""
returns 
tensor([[ 4.0000,  0.0000,  0.0000],
        [ 3.0000, 34.0000,  0.0000],
        [-4.0000,  0.1471,  0.2647]])
"""
print(pivot)
#returns tensor([1, 3, 3], dtype=torch.int32)
</code></pre>
<p>So I assume there’s some way to take <code>LD</code> and <code>pivots</code> and use them to compute <code>L</code> and <code>D</code> separately? Is there a way to do this efficiently? Is this possible in pytorch?</p>
How can i get seperate gradient instead of a summed one

Yuhan_PING: <p>Dear all,</p>
<p>I’m trying to use autograd.grad to calculate partial derivatives. However, I found that the grad results are summed up results, but I want seperate ones.<br>
The code is as follows:</p>
<blockquote>
<p>def df4_dx(p, c):<br>
dist_grad = torch.autograd.grad(outputs=dist, inputs=[p], grad_outputs=torch.ones_like(dist), create_graph=True)<br>
print("dist_grad xxxxxxx= ", dist_grad)</p>
<p>p2 = torch.tensor([[1.0, 2.0], [1.0, 4.0], [1.0, 6.0], [1.0, 8.0]], requires_grad=True)<br>
ci2 = torch.tensor([[0, 0], [0, 1], [1,2]])</p>
<p>z4 = df4_dx(p2, ci2)<br>
dfdx, dfdy = torch.zeros(p2.shape[0], ci2.shape[0]), torch.zeros(p2.shape[0], ci2.shape[0])<br>
print("z4 = ", z4)</p>
</blockquote>
<p>Here I calculate the distance between pi and ci, there are 2 ci so I’m expecting [dfdx at c0, dfdx at c1], however I get [dfdx at c0 + dfdx at c1]. I can get the seperate resutls if i do the autograd for <code>dist[:, 0]</code> and <code>dist[:, 1]</code> which acts like a for-loop. May I ask is there any better way to achieve this? Thanks.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/yuhan_ping">@Yuhan_PING</a>, You might want to have a look at the <code>torch.func</code> package, as that allows for per-sample gradients instead of summed gradients via the use torch <code>torch.func.vmap</code></p>
In PTQ, is possible to quantize activation in per channel mode?

111357: <p>As far as I know, we can quantize weight in per-channel mode by using <code>default_per_channel_weight_observer</code>. But I did not see any <code>per_channel</code> related configuration for activation.<br>
Is possible to do it for activation?</p>

jcaip: <p>Hi <a class="mention" href="/u/111357">@111357</a>,</p>
<p>Sorry for the late reply. You should be able to set this by changing the qconfig. See <a href="https://pytorch.org/blog/quantization-in-practice/#qconfig" rel="noopener nofollow ugc">here</a> for more details.<br>
I think you should be able to use <code>default_per_channel_weight_observer</code>, from the implementation <a href="https://github.com/pytorch/pytorch/blob/67e0913de9b400d0f32950f3f03dddd39f44c450/torch/ao/quantization/observer.py#L1564" rel="noopener nofollow ugc">here</a> it looks like it’s just calling PerChannelObserver with dtype=int8.</p>
[Need recommend] Good book for 3D vision with deep learning?

davidlee: <p>Hi all,</p>
<p>I want to start to study 3D vision with deep learning like 3D CNN, 3D object detection, etc.<br>
Is there any good book to start with??<br>
Please recommend!</p>
<p>Thanks!</p>

Ted: <p>Hi,</p>
<p>This is not really a “book” recommendation. Check out this Ph.D. thesis. It covers a lot of work on 3D objects using Deep Learning.</p>
<p><a href="http://ai.ucsd.edu/~haosu/" rel="nofollow noopener">http://ai.ucsd.edu/~haosu/</a><br>
<a href="http://ai.ucsd.edu/~haosu/papers/thesis_finalversion.pdf" rel="nofollow noopener">http://ai.ucsd.edu/~haosu/papers/thesis_finalversion.pdf</a></p>
<pre><code class="lang-auto"></code></pre>

davidlee: <p>Hi Ted,<br>
Thank you for your comment. It would be very helpful.<br>
I think there is no textbook related to 3D vision deep learning yet. haha<br>
Thank you anyway!!</p>

Ananya_Verma: <p>Seems like the link to the paper doesn’t work now. Here is the updated link: <a href="https://stacks.stanford.edu/file/druid:wc254sr0483/main-augmented.pdf" rel="noopener nofollow ugc">https://stacks.stanford.edu/file/druid:wc254sr0483/main-augmented.pdf</a></p>
Anomaly Detection using Bounding Box approach

Apurv25: <p>Hi all,</p>
<p>Previously I had implemented an anomaly detection based on Densenet architecture and achieved good results on road scene(nuScene) data.</p>
<p>Now, I am trying to create a bounding box version using the Unet decoder and SSD head without any anchor generator.</p>
<p>I am bit confused with the number of classes here. In pixelwise wise segmentation, the binary classification is a well known problem, however, in case of detection, how is the number of classes handled?<br>
For example, we have bbox labels for the anomalies, but not for the background, so how should I modify the loss function? I am not looking for complete code but conceptual answers would be most helpful to clarify my understanding.</p>
<p>Let me know if you require more details.</p>
<p>Thanks.</p>
Transforms for TensorDataset()

Mohamed_Farag: <p>Hi,<br>
I am new to Pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/baby.png?v=12" title=":baby:" class="emoji" alt=":baby:" loading="lazy" width="20" height="20"> and I want to load the dataset using <code>TensorDataset()</code></p>
<pre><code class="lang-auto">train_dataset = TensorDataset(X_train, y_train)

test_dataset = TensorDataset(X_val, y_val)
</code></pre>
<p>how to add transforms to this method? as I have already checked the normal dataset class and it was working normally.</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>

ptrblck: <p><code>TensorDataset</code> doesn’t accept transformations and will just index the passed tensors.<br>
If you want to apply transformations on these tensors you could create a custom <code>Dataset</code>, e.g. via:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, data, target, transform=None, target_transform=None):
        self.data = data
        self.target = target
        self.transform = transform
        self.target_transform = target_transform
        
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, index):
        x = self.data[index]
        if self.transform:
            x = self.transform(x)
        
        y = self.target[index]
        if self.target_transform:
            y = self.target_transform(y)
            
        return x, y
</code></pre>

Mohamed_Farag: <p>Too cool! I am surprised by the torch’s flexibility actually <img src="https://discuss.pytorch.org/images/emoji/apple/heart_eyes.png?v=12" title=":heart_eyes:" class="emoji" alt=":heart_eyes:" loading="lazy" width="20" height="20"></p>
How to avoid device binding when convert pytorch model to trochscript?

JohnHerry: <p>Hi, all<br>
I am trying to convert a VITS tts model into torchscript.<br>
I simplly used the “torch.jit.trace()”.   but there is newly created tensor in the model, the tensor is a length-scaled tensor.<br>
“x = torch.arange(max_length, dytpe=length.dtype, device=length.device)”<br>
sadly, after the convertion, the torchscript model keeps the device on tensor “x” as a hard-code device there, as a result, it can not inference on any other device from the one that convertion process had used.<br>
I had tried to use torch.jjit.script instead of trace, it can not even make success conversion.<br>
So it is there any way for the torch.jit.trace() convertion to support dynamic device in my instance?</p>

ptrblck: <p>I think this limitation in tracing is expected and scripting was the recommended approach to allow “dynamic” behavior. I don’t know if you are running into any issues using scripting but also note that TorchScript is in maintenance mode.</p>

JohnHerry: <p>Thanks ptrblck,<br>
I  have planed to try to transform the VITS model into ONNX format now. because inference speed of the torchscript VITS model  is Unbearably SLOW,  It cost more then ten seconds to synthesize a text of moderate length on RTX3090 ,  Which may be caused by many recurrent structures in the VITS model.  But the JIT convertion makes no imporovement on the inferene speed, and far more worse.</p>
Input image size to pretrained resnet34

Anastasia_Berlianna: <p>i’m using pretrained resnet34 as encoder in my unet model, in pretrained resnet34 documentation, the input image size is 224x224x3, however i’ve tried input image 224x224x3 and 512x512x3 and model performance on both inputs is not much different. Is it ok to use 512x512x3 input considering this input image quality is better than 224x224x3?</p>

eqy: <p>If you are using a pretrained model (without finetuning) then not necessarily getting better accuracy with higher resolution is expected as higher resolution images can present a different distribution of object scales than what the model was trained on. This paper is a good read about the details of what can cause this behavior: <a href="https://arxiv.org/abs/1906.06423" class="inline-onebox" rel="noopener nofollow ugc">[1906.06423] Fixing the train-test resolution discrepancy</a></p>

vahvero: <p>The pretrained models contain resize operation within the model. When using vision’s models, one should always be careful, as most of the logic is embedded into the model and not visible to the user. Personally I am not a fan of this approach.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; model = torchvision.models.resnet34()
&gt;&gt;&gt; A = torch.rand(4,3,256,256)
&gt;&gt;&gt; B = torch.rand(4,3,512,512)
&gt;&gt;&gt; model(A)
tensor([[-0.0159,  0.6987,  0.0230,  ...,  0.5298,  0.7964, -0.7284],
        [ 0.0107,  0.6738,  0.1062,  ...,  0.5328,  0.8554, -0.8342],
        [ 0.0376,  0.7694,  0.0367,  ...,  0.4300,  0.7086, -0.7498],
        [-0.1892,  0.6690,  0.0665,  ...,  0.4711,  0.8034, -0.7326]],
       grad_fn=&lt;AddmmBackward0&gt;)
&gt;&gt;&gt; model(B)
tensor([[-0.0178,  0.6623,  0.1057,  ...,  0.4388,  0.8645, -0.8103],
        [ 0.0054,  0.7318,  0.0728,  ...,  0.4826,  0.8127, -0.7748],
        [-0.0430,  0.7653,  0.0638,  ...,  0.5089,  0.8085, -0.7793],
        [-0.0374,  0.6968,  0.1174,  ...,  0.5085,  0.7960, -0.8011]],
       grad_fn=&lt;AddmmBackward0&gt;)

</code></pre>
Ram crashing when transforming form numpy to torch tensor

Mohamed_Farag: <p>Hi,<br>
I am having a problem now, I am running my code on google colab with 12 GB Ram, I have images stored at numpy files and I want to transfer them to tensors, but the ram is crashing.</p>
<p>This is my code:</p>
<pre><code class="lang-auto"> X_train, X_val, y_train, y_val = train_test_split(X,y, test_size = 0.10, random_state = 87, stratify = y)

X_train = torch.from_numpy(X_train).float().cuda() # transform to torch tensor
y_train = torch.from_numpy(y_train).float().cuda()

X_val = torch.from_numpy(X_val).float().cuda()
y_val = torch.from_numpy(y_val).float().cuda()
</code></pre>

ptrblck: <p>These calls <code>torch.from_numpy(X_train).float()</code> will create a new copy on the host if the tensor is not already in <code>float32</code> and will thus increase the memory usage.<br>
Would it be possible to directly load the numpy arrays in their desired <code>dtype</code>?<br>
Alternatively, you might want to move the tensor to the GPU and transform it to<code>float32</code> there, which would of course launch another copy kernel, but would reduce the host memory usage.</p>

Mohamed_Farag: <p>Thanks a lot! it was too helpful <img src="https://discuss.pytorch.org/images/emoji/apple/heart_eyes.png?v=12" title=":heart_eyes:" class="emoji" alt=":heart_eyes:" loading="lazy" width="20" height="20"></p>
Can I access and call submodules of DDP model when training?

Junxuan_Chen: <p>Suppose I have a module combined some submodule, like:</p>
<pre><code class="lang-auto">class Comb(nn.Module):
    def __init__(self, encoder, decoder):
        super(Comb, self).__init__()
        self.encoder = encoder
        self.decoder = decoder
</code></pre>
<p>When distribute training, I warp the module with DDP:</p>
<pre><code class="lang-auto">comb = DDP(comb)
raw_comb = comb.module
</code></pre>
<p>Now in training loop, I want to use the submodule of the comb, like:</p>
<pre><code class="lang-auto">feature = raw_comb.encoder(input)
out = raw_comb.decoder(feature)
loss = loss_fn(out, label)
loss.backward()
</code></pre>
<p>Will this behavior break the DDP module gradient sync?</p>

ptrblck: <p>Yes, if you are accessing the internal <code>.module</code> the DDP wrapper won’t be used and you would execute the raw model on each device separately.</p>

Junxuan_Chen: <p>Your suggestion saved me 2 hours of time to debug not to mention I don’t know how to check the gradient sync.  Thank you.</p>

ptrblck: <p>If you want to dig a bit into the code and see how <code>DDP</code> works check <a href="https://discuss.pytorch.org/t/is-it-ok-to-use-methods-other-than-forward-in-ddp/176509/2">this post</a> as I’ve shared a few references to see how the hooks are used and where the internal module is called.</p>
Quantizing a torch::jit::Module or updating a quantized model's weights

tdomhan: <p>Hi everyone,<br>
I have a PyTorch model in C++ which is a torch::jit::Module. The model has been quantized to int8 and traced in PyTorch. I can load it via torch::jit::load. I would like to programmatically update the weights, but haven’t found a way to do so yet. I have the following code which finds the modules that contain _packed_params, but I’m not sure how to go about updating the weights:</p>
<pre><code class="lang-auto">void copy_weights_quantized(torch::jit::Module&amp; src_module, torch::jit::Module&amp; dst_module) {
    std::map&lt;std::string, torch::jit::Module&gt; src_param_map;
    for (const auto&amp; pair : dst_module.named_modules()) {
        src_param_map.emplace(pair.name, pair.value);
        for (const auto&amp; pair2 : pair.value.named_attributes()) {
            
            if (pair2.name == "_packed_params._packed_params") {
                std::cout &lt;&lt; "    Attribute: " &lt;&lt; pair2.name &lt;&lt; " mod: "&lt;&lt; pair2.value.isModule() &lt;&lt; " capsule: " &lt;&lt; pair2.value.isCapsule() &lt;&lt; std::endl;
                
                auto packed_weight_holder = pair2.value.toObjectRef().getSlot(0).toCapsule();
                c10::intrusive_ptr&lt;LinearPackedParamsBase&gt; packed_weight = c10::static_intrusive_pointer_cast&lt;LinearPackedParamsBase&gt;(packed_weight_holder);
                // pair2.value.toObjectRef().getSlot(0).
            }
        }   
    }
}
</code></pre>
<p>Alternatively if there was a way to quantize a torch::jit::Module to int8 in C++ that would also be great. I haven’t found the equivalent of <code>quantize_dynamic</code> yet. Any pointers are appreciated.</p>
How to install pytorch in enviroment of python3.6 and RTX3060 GPU,CUDA11.6?

lmk163: <p>My system environment is in python3.6,RTX3060 and CUDA11.6,I want to install pytorch,but I did not found corresponding version to download,Can somebody tell me what should I do？</p>

eqy: <p>You do not need to install a PyTorch version that has a matching version of CUDA. The CUDA version specified in the PyTorch wheel is the CUDA version that the wheel was built and packaged with. Could you check if the 11.8 pip wheels (install with <code>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</code>) works for you?</p>
Correlation loss function

Mark_Esteins: <p>Hi,</p>
<p>I have a correlation function that amplifies the tails and I’m trying to port it to PyTorch. The loss function looks like this (preds and targets are a single column):</p>
<pre><code class="lang-auto">def strong_tails_corr(preds, target):
  ranked_preds = (preds.rank(method="average").values - 0.5) / preds.count()
  gauss_ranked_preds = stats.norm.ppf(ranked_preds)
  centered_target = target - target.mean()
  preds_p15 = np.sign(gauss_ranked_preds) * np.abs(gauss_ranked_preds) ** 1.5
  target_p15 = np.sign(centered_target) * np.abs(centered_target) ** 1.5
  return np.corrcoef(preds_p15, target_p15)[0, 1]
</code></pre>
<p>Any ideas on how I can do that? What is troubling me the most is how to build ranked_preds and gauss_ranked_preds, the rest I think I have it:</p>
<pre><code class="lang-auto">def corrcoef_torch(preds, targets):
    preds = preds - preds.mean(dim=0)
    preds = preds / preds.norm(dim=0)
    targets = targets - targets.mean(dim=0)
    targets = targets / targets.norm(dim=0)
    return (preds * targets).sum()

from torch.distributions import Normal
import torchsort
normal = Normal(0,1)
def strong_tails_corr(preds, target):
	ranked_preds = (torchsort.soft_rank(preds, regularization_strength=.0001) - 0.5) / preds.size(dim=0)
	gauss_ranked_preds = normal.icdf(ranked_preds) 
	centered_target = target - target.mean(dim=0)
	preds_p15 = torch.sign(gauss_ranked_preds) * torch.abs(gauss_ranked_preds) ** 1.5 #hadamard product
	target_p15 = torch.sign(centered_target) * torch.abs(centered_target) ** 1.5 #hadamard product
    return corrcoef_torch(preds_p15, target_p15)
</code></pre>
<p>Thanks in advance</p>
Should I use an IterableDataset? Data is similar to a directory of videos and model would look at sequences within each video

AZForward: <p>So I am not working with video files but a custom file type of video game replays.  The files contain game states and player inputs, and I’m trying to build a model that can predict player inputs.  I will need to do a fair amount of preprocessing and I would like to test out a few different model types (RNN, LSTM, Transformers/Attention).  Since the nature of the data is sequential, I will need to take subsequences from within each file.</p>
<p>For now I am not implementing a Dataset nor a Dataloader.  I am following this tutorial which doesn’t use them either and I’m afraid it will make the training much slower.  I suppose I’ll find out for sure when I get there but in the mean time I would appreciate any help or insight on this.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></p>
<p>Thank you!</p>

AZForward: <p>This might be a better explanation of what I am trying to do.</p>
<p>I have a directory called text_files that has the following:</p>
<pre><code class="lang-auto">text1.txt
text2.txt
text3.txt
</code></pre>
<p>The content of each file is as follows:</p>
<blockquote>
<p>my first text file<br>
here is another text file that is a little longer<br>
a third text file for extra measure</p>
</blockquote>
<p>I need to iterate through each text file and return a single word at a time.  Assume that each word maps to a token, and I’ll use 0 as a padding token.  I would like my dataloader to load each record like so:</p>
<blockquote>
<p>[my, 0, 0, 0, 0, 0, 0, 0, 0, 0, ]<br>
[my, first, 0, 0, 0, 0, 0, 0, 0, 0, ]<br>
[my, first, text, 0, 0, 0, 0, 0, 0, 0, ]<br>
[my, first, text, file, 0, 0, 0, 0, 0, 0, ]<br>
[here, 0, 0, 0, 0, 0, 0, 0, 0, 0, ]<br>
[here, is, 0, 0, 0, 0, 0, 0, 0, 0, ]<br>
[here, is, another, 0, 0, 0, 0, 0, 0, 0, ]</p>
</blockquote>
<p>and so on.  I’m running my head around in circles trying to figure out which combination of dataset, dataloader and collate function I would need to incorporate this.</p>

AZForward: <p>After posting this, I think my use case is unique enough that I’ll just build my own class to handle everything.</p>
Complex numbers - Cuda 11.6 documentation warning

Danfoa: <p><a href="https://pytorch.org/docs/stable/complex_numbers.html" rel="noopener nofollow ugc">Documentation on Complex numbers</a> has a note indicating that the user should use Cuda 11.6 without additional information on why this is recommended.</p>
<p>Cuda 11.6 is considerably old and does not support new OS versions (e.g., Ubuntu 22.04). Therefore, can anyone explain why is Cuda 11.6 is recommended and not any of the newer versions of it? What features are missing in other versions? what works/what doesn’t?</p>

ptrblck: <p>You can ignore the note as it’s related to an actual slow compilation time in <code>11.3</code> described <a href="https://github.com/pytorch/pytorch/issues/79876">here</a>. We should remove it as <code>11.3</code> isn’t even built anymore in our binary matrix.</p>
<p>EDIT: related PR: <a href="https://github.com/pytorch/pytorch/pull/100118" class="inline-onebox">Remove CUDA 11.6 note from complex docs by ptrblck · Pull Request #100118 · pytorch/pytorch · GitHub</a></p>
RuntimeError: Expected input_lengths to have value at most 144, but got value 190 (while checking arguments for ctc_loss_cpu)

respect.yojic: <p>Hello! Does somebody know what is this error means ‘RuntimeError: Expected input_lengths to have value at most 144, but got value 190 (while checking arguments for ctc_loss_cpu)’? I tried to implement conformer model and 144 - model dimension, but I can’t understand what is 190 number because I didn’t use this number.</p>
<p>torch.Size([32, 598, 80]) - input.shape<br>
torch.Size([32]) - inputs_length.shape<br>
torch.Size([32, 78]) - target.shape<br>
torch.Size([32]) - target_length.shape</p>

ptrblck: <p>I guess the <code>input_length</code> contains the length of <code>190</code> and raises the error.<br>
Using the example from the docs reproduces it:</p>
<pre><code class="lang-python"># works

# Target are to be padded
T = 50      # Input sequence length
C = 20      # Number of classes (including blank)
N = 16      # Batch size
S = 30      # Target sequence length of longest target in batch (padding length)
S_min = 10  # Minimum target length, for demonstration purposes
# Initialize random batch of input vectors, for *size = (T,N,C)
input = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_()
# Initialize random batch of targets (0 = blank, 1:C = classes)
target = torch.randint(low=1, high=C, size=(N, S), dtype=torch.long)
input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long)
target_lengths = torch.randint(low=S_min, high=S, size=(N,), dtype=torch.long)
ctc_loss = nn.CTCLoss()
loss = ctc_loss(input, target, input_lengths, target_lengths)
loss.backward()

# fails

input_lengths[0] = 190
loss = ctc_loss(input, target, input_lengths, target_lengths)
# RuntimeError: Expected input_lengths to have value at most 50, but got value 190 (while checking arguments for ctc_loss_cpu)
</code></pre>

respect.yojic: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you. But how can I fix it? I made collate function to pad sequence but it doesn’t work</p>

ptrblck: <p>The <code>input_length</code> vector should contain valid lengths related to the <code>input</code> tensor.<br>
In my example you can see that <code>T = 50</code> defines the input sequence length. Thus, the <code>input_lengths</code> tensor should contain values indicating the length in <code>[0, 50]</code>. Setting the <code>input_length</code> to a larger value than the actually sequence dimension in <code>input</code> is invalid and will raise the error so check why your <code>input_length</code> tensor contains these invalid lengths.</p>
How to print torch.compile'd graph?

vadimkantorov: <p>The FX graphs can be printed with <code>.code</code> / <code>.graph</code>: <a href="https://pytorch.org/docs/stable/fx.html" class="inline-onebox" rel="noopener nofollow ugc">torch.fx — PyTorch 2.0 documentation</a></p>
<p>Can torch.compile’d graphs be inspected (in order to see if there’re any graph breaks)? Should I set <code>TORCH_COMPILE_DEBUG=1</code>?</p>

marksaroufim: <p>Yup that’s the best flag it’ll show you the fx and inductor IRs</p>

vadimkantorov: <p>Is it possible to get the fx IR by accessing some compiled model attribute? I also saw somehwere some <code>_dynamo.explain(...)</code> method, but not sure if it will print the IR…</p>

marksaroufim: <p>This tutorial has some good examples of how to get the relevant fx and aten IR from a torch program <a href="https://colab.research.google.com/drive/1Zh-Uo3TcTH8yYJF-LLo5rjlHVMtqvMdf#scrollTo=MWUvovoo6KYK" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>As far as getting the actual IRs that inductor they should be printed here if running with debug flag <a href="https://github.com/pytorch/pytorch/blob/main/torch/_inductor/compile_fx.py#L182" class="inline-onebox" rel="noopener nofollow ugc">pytorch/compile_fx.py at main · pytorch/pytorch · GitHub</a></p>

vadimkantorov: <p>Thank you for your answers! Btw how does one print Triton compiled code? Basically trying to understand what got fused in the graph and what did not…</p>

marksaroufim: <p>afk now but last I remember <code>TORCH_COMPILE_DEBUG=1 python train.py</code> should spew out a bunch of logs, look for a line that looks like <code>tmp/inductor/somelongstringsetc</code> if you open that folder you should be able to inspect the actual generated triton code</p>
<p>Also note that you can figure out what got fused or not by looking at the inductor IR pre and post fusion again using the same debug flag but I personally found the triton code to be more readable</p>
<p>And since you’re probably curious you can also inspect the actual PTX code as well <a href="https://gist.github.com/msaroufim/3dfe4e7769fc9139d1489be8582659d8" class="inline-onebox" rel="noopener nofollow ugc">triton-ptx.py · GitHub</a></p>
Solving many independent optimization problems with `torch.optim`

cwindolf: <p>I’m interested in doing the torch version of something like this, in Scipy:</p>
<pre><code class="lang-python">import numpy as np
from scipy.optimize import minimize

def objective(x):
    return np.mean(x ** 2)

# a batch of initial conditions
# batch size is 101, problem dimension is 3
x0s = np.random.normal(size=(101, 3))
# use a BFGS to solve these problems independently
xs = [minimize(objective, x0).x for x0 in x0s]
</code></pre>
<p>In PyTorch, I could write</p>
<pre><code class="lang-auto">import torch
from torch.optim import LBFGS
xs = torch.tensor(x0s, requires_grad=True)
opt = LBFGS([xs])
def closure():
    opt.zero_grad()
    # objective is the MSE, and we sum the independent objectives
    # xs.shape is (101, 3) as above
    loss = torch.mean(xs ** 2, 1).sum()
    loss.backward()
    return loss
opt.step(closure)
</code></pre>
<p>and this works, in the sense that the code runs; however, it solves the problem as a (101*3)-dimensional problem rather than as 101 separate 3-dimensional problems, which is much more expensive when using a method which tries to approximate the Hessian, like LBFGS. It would probably also lead to a difference in stopping criteria with the Scipy code above.</p>
<p>I’m wondering, is my understanding here correct, and if so is there any way to run optimizations in a truly independent way with torch? Maybe some vmap strategy?</p>

cwindolf: <p>After some experiments scaling up the problem size (both the number of problems and the problem dimension), I’ve concluded that I don’t understand the full picture. torch is much faster than scipy even on huge problems. So, I’m happy about that, but would still welcome any perspectives on these questions.</p>
<p>For reference, the scaling benchmark I ran was:</p>
<ul>
<li>
<p><em>scipy</em>, in a notebook so that <code>%timeit</code> works:</p>
<pre><code class="lang-python">import numpy as np
from scipy.optimize import minimize

def objective(x):
    return np.mean(np.diff(x) ** 2) + x[9] ** 2

# a batch of initial conditions
# batch size is 101, problem dimension is 3
x0s = np.random.normal(size=(10001, 10))
# use a BFGS to solve these problems independently
%timeit xs1 = np.array([minimize(objective, x0, method="L-BFGS-B").x for x0 in x0s])
</code></pre>
<p>[30.8 s ± 160 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)]</p>
</li>
<li>
<p><em>torch</em>:</p>
<pre><code class="lang-auto">%%timeit
xs2 = torch.tensor(x0s, requires_grad=True)
opt = LBFGS([xs2], max_iter=100)
def closure():
    opt.zero_grad()
    # sum the independent objectives
    loss = (torch.mean(torch.diff(xs2, 1) ** 2, 1) + xs2[:, 9] ** 2).sum()
    loss.backward()
    return loss
opt.step(closure)
</code></pre>
<p>[80.1 ms ± 2.19 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)]</p>
</li>
</ul>
<p>The objective values are comparable (torch’s is slightly smaller).</p>
<p>This is all just on CPU. Checking htop, torch doesn’t exceed 100% CPU, so it’s fair to run scipy in serial.</p>
How to use Captum on a network with matrix output

flyingpig: <p>I am trying to use Integrated Gradients on a network whose output is of dimension (batchsize, 12, 2).</p>
<p>ig = IntergratedGradients(model)<br>
attr, delta = ig.attribute(input)<br>
then it says that I should specify the target. What parameter should I use for the target? I could not find any reasonable explanation on this.</p>

Z4K: <p>Assuming you are doing classification, the target is the index of the target class.</p>