akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Multiplication of activation function with learnable parameter/scalar

nmourdou: <p>Hello guys,</p>
<p>I am creating a custom neural network and I want to create a linear combination of two outputs of different layers after applying a corresponding non-linearity. So for example, if x and y are the outputs of two layers and f is an activation function (relu, softpluts etc), I wish to compute:<br>
a*f(x)+(1-a)*f(y)<br>
I have tried many things, like using the expand operator for the scalar with the right dimensions of tensor x and y but I keep getting the error :<br>
" unsupported operand type(s) for *: ‘Tensor’ and ‘Softplus’ "</p>
<p>Any ideas on how to make it work?</p>
<p>Thank you in advance</p>

ptrblck: <p>I think you are missing the initialization of the <code>nn.Softplus</code> module and might be running into this error:</p>
<pre><code class="lang-python">torch.randn(1) + nn.Softplus(torch.tensor(1.))
&gt; TypeError: unsupported operand type(s) for +: 'Tensor' and 'Softplus'
</code></pre>
<p>You would have to create the module first before applying it onto the tensor or use the functional API:</p>
<pre><code class="lang-python">torch.randn(1) + nn.Softplus()(torch.tensor(1.))
torch.randn(1) + F.softplus(torch.tensor(1.))
</code></pre>

nmourdou: <p>That worked. Thank you very much for the answer!</p>
Lazy execution in PyTorch?

hidefromkgb: <p>Hi everyone.</p>
<p>I`d like to implement a custom backend for PT that would support lazy network execution: e.g. when ops get added they are not executed right away but rather accumulate the topology, and only when some trigger happens — e.g. a target tensor gets read — is the previously accumulated topology optimized, compiled and executed on a custom non-CPU device.</p>
<p>Is that at all possible in PT? I couldn`t find much documentation on how to write PT backends of the kind I just described.</p>

hidefromkgb: <p>Okay, now that I know more about PT I can answer my own question.</p>
<p>The solution is to use TorchScript and work directly with the traced graph.<br>
For that you`ll need a torch::jit backend.<br>
Here`s <a href="https://jott.live/markdown/Writing%20a%20Toy%20Backend%20Compiler%20for%20PyTorch" rel="noopener nofollow ugc">a useful article</a> on the topic. Bear in mind however that it`s already quite outdated, so e.g. CustomFuseGraph would not be of much help so you`ll need to create fusion groups (i.e. meta-primitives you`ll be in charge of compiling and executing) by hand — substituting the existing prims via SubgraphUtils::createSingletonSubgraphAndUpdateAliasing() and SubgraphUtils::mergeNodeIntoSubgraphAndUpdateAliasing().</p>
BERT with a GRU layer, wrong dimensions

PizzaMargherita: <p>Hey all, struggling a bit with the input dimensions.</p>
<p>I have a BERT model (from huggingface) and want to add a GRU.<br>
<code>self.bert = BertModel.from_pretrained('bert-base-uncased')</code></p>
<p>I don’t understand the dimensions yet. Getting this error:<br>
<code>RuntimeError: input must have 3 dimensions, got 2</code>.</p>
<p>The GRU is defined as follows.<br>
<code>self.gru = nn.GRU(input_size=768*2 , hidden_size=64, num_layers=2)</code></p>
<p>The output of the BERT is<br>
<code>torch.cat((outputs1.last_hidden_state[:,0], outputs1.last_hidden_state[:,max_q_len]), 1)</code><br>
with <code>torch.Size([32, 1536])</code></p>
<p>I can see that the output has 2 dimensions, but what’s the third one that GRU expects? According to the docs the input needs to be of shape <code>seq_len, batch, input_size</code>.</p>
<p>Would be glad if someone could help me out here. Thanks!</p>
What do I need for an Alexa-like system?

newbieee: <p>Hi guys,</p>
<p>I would like to build an Alexa like system for some sub sets of questions. For this I have a few questions, (1) what kind of problems would this task belong to as well as (2) whether this kind of task is feasible for a ML beginner.</p>
<p>Longer Version:</p>
<ol>
<li>What kind of problem is this?<br>
**Input: **<br>
i) My basic idea would be to first transform the speech to text - there are probably ready-made solutions for this.<br>
ii) Analyze / tag the text from (i) to filter out the question<br>
E.g. from the question “What is the name of the most visited website”.<br>
=&gt; filter “name most visited website”</li>
</ol>
<p>**Output: **<br>
iii)To answer the filtered question (“name most visited website”), I would run it against a database full of statements and return the statement with the highest match to the question.<br>
Here, the database could possibly contain statements like:<br>
E.g. “Pytorch(dot)org is the most visited website”.<br>
 → Object: Pytorch(dot)org, Statement: most visited website<br>
and then this statement would most likely match the question from above and then you would return the object of the statement (Pytorch(dot)org).</p>
<p>**Statement-Database: **<br>
I would like to build the statement database by parsing Wikipedia. For example, if the system is to answer questions about Machine Learning, I would first parse and save the Wikipedia page of Machine Learning and call all the links of the article and parse and save them in the same way and this until I have a certain number of articles.<br>
I would then analyze / parse the parsed texts just like the questions above and add them to a database.</p>
<p>My question here would be, if this kind of approach would make sense, to what kind of problems this task may belong, so that I can better inform myself.</p>
<p>I’ve done a few things so far with sentiment analysis (NLP) and image recognition. Do you think such a prototypical project would be feasible with the knowledge within half a year? I currently only have an AMD R9 390 - is Google Colab enough or would I have to buy a new graphics card for this?</p>
<p>Ty</p>

newbieee: <p>bring up my post - bump</p>
Basic doubt regarding channel of an image for squeeze and excitation network

Alok1: <p>Hi, I am working on an image reconstruction problem, where images in my dataset are of size (1, 1, 128,128 )  To improve the current results I came across Squeeze and Excitation block, which allows the network to feature recalibration (emphasize informative features and suppress less useful features)<br>
however, Squeeze and Excitation network works with an image channel greater than 1. is there any possibility that I can still use this network by converting 1 channel to a 3 channel image.<br>
Has anyone worked with SE network here? any advice would be a great help.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67f8f1ab0f2e6b8964b1ec19e00a4c8775192d2a.png" data-download-href="https://discuss.pytorch.org/uploads/default/67f8f1ab0f2e6b8964b1ec19e00a4c8775192d2a" title="sq"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67f8f1ab0f2e6b8964b1ec19e00a4c8775192d2a_2_690x157.png" alt="sq" data-base62-sha1="ePMxs2fn9L8UiPLChEBvzBoD4G6" width="690" height="157" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67f8f1ab0f2e6b8964b1ec19e00a4c8775192d2a_2_690x157.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67f8f1ab0f2e6b8964b1ec19e00a4c8775192d2a_2_1035x235.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67f8f1ab0f2e6b8964b1ec19e00a4c8775192d2a_2_1380x314.png 2x" data-dominant-color="F3EFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">sq</span><span class="informations">1425×326 71 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>