akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there any improvement in accuracy if the activation function of styleGAN is changed?

username: YuhskeHujisaki
cooked: <p>Is there any improvement in accuracy if the activation function of styleGAN is changed? Or will it be less accurate?</p>

username: Andrew_Legend
cooked: <p>It depends what you mean when you say accuracy. If you mean the accuracy of the discriminator then perhaps? You actually have to test the activation in order to find out. ReLU/Leaky ReLU and in some scenarios PReLU have been shown to work very well due to their properties.</p>
<p>However, it has been shown over and over again that The loss/accuracy of discriminator has nothing to do with image quality, so keep that in mind.</p>
<p>Do you mind further elaborating?</p>
About of torch.svd error

username: charlotte9
cooked: <p>code: U, S, V = torch.svd(torch.mm(src_vec, tgt_vec.t()))<br>
sinpa = torch.sqrt(1 - torch.pow(cospa,2))<br>
rsd_loss = torch.norm(sinpa,1)</p>
<p>When I train model ,and I use rsd_loss  as loss values, pytorch always give me a error:<br>
Intel MKL ERROR: Parameter 4 was incorrect on entry to SLASCL.<br>
RuntimeError: svd_cuda: the updating process of SBDSDC did not converge (error: 23)<br>
Even I moved the Tensors to cpu but still getting the same issue.</p>
<p>Then I found that after training twenty or thirty epoch, Nan and INF appeared in the gradient, so the weight of update also changed to nan, and all the features obtained by network changed to Nan, which were passed into torch.svd(). SVD calculation reported the above error.<br>
Some say it‘s a pytorch version problem,  and I have tried several version:1.10.0、1.7.0、0.4.1，I get same error as above.<br>
How should this problem be solved？ Which version of pytorch should I use？</p>
Custom Dataset: Best Practices for Transformations on Training Set

username: janwinkler1
cooked: <p>Hello,</p>
<p>I am working on an image segmentation task. There are two folders, “inputs” and “masks”, which both contain the corresponding pairs of the dataset (i.e., input/ipt_xy.npy corresponds to masks/msk_xy.npy).</p>
<p>My dataset class looks like this:</p>
<pre><code class="lang-auto">class DatasetBlaBla(Dataset):
    def __init__(self, root_path, ipt, tgt, transform=None):
        super(DatasetBlaBla, self).__init__()
        self.root_path = root_path
        self.ipt = ipt
        self.tgt = tgt
        self.transform = transform
        
    def __len__(self):
        l1 = os.listdir(os.path.join(self.root_path, self.ipt)) 
        l2 = os.listdir(os.path.join(self.root_path, self.tgt))
        number_files_inp = len(l1)
        number_files_tgt = len(l2)

        if number_files_inp == number_files_tgt:
            return number_files_inp


    def __getitem__(self, idx):
        img_path_input_patch = os.path.join(self.root_path, self.ipt, f"ipt_{idx}.npy")
        img_path_tgt_patch = os.path.join(self.root_path, self.tgt, f"tgt_{idx}.npy")
        
        input_patch = np.load(img_path_input_patch)
        tgt_patch = np.load(img_path_tgt_patch)  
        
        
        if self.transform:
            input_patch = self.transform(input_patch)
            tgt_patch = self.transform(tgt_patch)
            
        return input_patch, tgt_patch
</code></pre>
<p>When creating the dataset, one instance is created, which I then split into train/val/test using:</p>
<pre><code class="lang-auto">train_set, val_set, test_set = torch.utils.data.random_split(dataset, [train_size, val_size, test_size])
</code></pre>
<p>Finally, we come to the question:</p>
<p>What are best practices, in this case, to apply transformations on the train_set only?<br>
I have looked through the forum and found a variety of approaches; however, I still wondered whether there is a distinct best practice for my use case. Also, I do not want to split the folders into train/val/test subfolders because the reshuffling will be an issue with my amount of data.</p>
<p>Thank you for your help,</p>
<p>Cheers</p>

username: ptrblck
cooked: <p>My preference would be to create three difference datasets using the desired transformations for the training, validation, and test sets. This approach makes it clear that the <code>train_dataset</code> also uses the <code>train_transform</code> only. Once this is done, create the training, validation, and test indices via any kind of splitting (<code>sklearn.model_selection.train_test_split</code> is quite popular) and wrap the datasets into a <code>Subset</code> with the corresponding indices.</p>
<p>There are certainly other approaches, so let’s also hear from other users. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: janwinkler1
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for your answer!</p>
Visualize vision transformer forward pass

username: Khawar_Islam
cooked: <p>How i can visualize the forward function in vision transformer (pyramid vision transformer)?.<br>
I know about CNN visualization but i do not have any idea.  I want to visualize x from the first and last line</p>
<blockquote>
<p>class PatchEmbed(nn.Module):<br>
“”" Image to Patch Embedding<br>
“”"</p>
<pre><code>def __init__(self, img_size=112, patch_size=8, in_chans=3, embed_dim=768):
    super().__init__()
    img_size = to_2tuple(img_size)
    patch_size = to_2tuple(patch_size)

    self.img_size = img_size
    self.patch_size = patch_size
    # assert img_size[0] % patch_size[0] == 0 and img_size[1] % patch_size[1] == 0, \
    #     f"img_size {img_size} should be divided by patch_size {patch_size}."
    self.H, self.W = img_size[0] // patch_size[0], img_size[1] // patch_size[1]
    self.num_patches = self.H * self.W
    self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)
    self.norm = nn.LayerNorm(embed_dim)

def forward(self, x):
    B, C, H, W = x.shape

    x = self.proj(x).flatten(2).transpose(1, 2)
    x = self.norm(x)
    H, W = H // self.patch_size[0], W // self.patch_size[1]

    return x, (H, W)
</code></pre>
</blockquote>
Is torch.Tesnor with requires_grad=True equal to Variable?

username: matic_lauko
cooked: <p>I’m new to PyTorch and AI in general. I have seen that Torch computational graph is created(we are able to compute gradients) when set use requires_grad=True. But, most of the code on GitHub and on Tutorials I’ve used use Variables. But, from reading I can’t see a difference.<br>
What is the difference, if there is any?<br>
Thanks for explanation or link!</p>

username: Henry_Chibueze
cooked: <p>In the latest versions of pytorch, <code>Variable()</code> is no more a thing. You can simply use <code>torch.tensor([...]).requires_grad_(True)</code></p>

username: bradezard
cooked: <p>You’re looking at very old code, as Variable was removed well before the 1.x.x release of pytorch. A Variable was a Tensor that required grad, but now you can just have tensors require grads directly without the wrapper. I’d suggest trying to find more up-to-date examples while you’re learning though, as there have been more than a few changes since variable was removed and you’ll likely find other points of friction trying to update old code whilst you’re learning it.</p>
Merge the non-over lap tensor back

username: wayne1226
cooked: <p>Now I have a tensor list with for same-size-tensor A=[a,b,c,d], say [3,224,224], what is the simplest way to merge them back to one tensor [[a,b],[c,d]] with size [3,448,448]?</p>

username: ptrblck
cooked: <p>I think <code>F.fold</code> should work assuming you want to create an output tensor where each “window” of the input is preserved:</p>
<pre><code class="lang-python"># setup
channels = 3
h = 4
offset = channels * h * h
a = torch.arange(offset).view(channels, h, h).float()
b = torch.arange(offset).view(channels, h, h).float() + offset
c = torch.arange(offset).view(channels, h, h).float() + 2*offset
d = torch.arange(offset).view(channels, h, h).float() + 3*offset

A = [a, b, c, d]

out = torch.stack(A, dim=1)
out = out.permute(0, 2, 3, 1).reshape(channels*h*h, h)
res = F.fold(out, h*2, h, stride=h)
print(res)

# tensor([[[  0.,   1.,   2.,   3.,  48.,  49.,  50.,  51.],
#          [  4.,   5.,   6.,   7.,  52.,  53.,  54.,  55.],
#          [  8.,   9.,  10.,  11.,  56.,  57.,  58.,  59.],
#          [ 12.,  13.,  14.,  15.,  60.,  61.,  62.,  63.],
#          [ 96.,  97.,  98.,  99., 144., 145., 146., 147.],
#          [100., 101., 102., 103., 148., 149., 150., 151.],
#          [104., 105., 106., 107., 152., 153., 154., 155.],
#          [108., 109., 110., 111., 156., 157., 158., 159.]],

#         [[ 16.,  17.,  18.,  19.,  64.,  65.,  66.,  67.],
#          [ 20.,  21.,  22.,  23.,  68.,  69.,  70.,  71.],
#          [ 24.,  25.,  26.,  27.,  72.,  73.,  74.,  75.],
#          [ 28.,  29.,  30.,  31.,  76.,  77.,  78.,  79.],
#          [112., 113., 114., 115., 160., 161., 162., 163.],
#          [116., 117., 118., 119., 164., 165., 166., 167.],
#          [120., 121., 122., 123., 168., 169., 170., 171.],
#          [124., 125., 126., 127., 172., 173., 174., 175.]],

#         [[ 32.,  33.,  34.,  35.,  80.,  81.,  82.,  83.],
#          [ 36.,  37.,  38.,  39.,  84.,  85.,  86.,  87.],
#          [ 40.,  41.,  42.,  43.,  88.,  89.,  90.,  91.],
#          [ 44.,  45.,  46.,  47.,  92.,  93.,  94.,  95.],
#          [128., 129., 130., 131., 176., 177., 178., 179.],
#          [132., 133., 134., 135., 180., 181., 182., 183.],
#          [136., 137., 138., 139., 184., 185., 186., 187.],
#          [140., 141., 142., 143., 188., 189., 190., 191.]]])
</code></pre>

username: wayne1226
cooked: <p>This is exactly what I need, thank you for your reply.</p>
Use argmax indices to select values from a tensor

username: jjhh
cooked: <p>I have 2 tensors <code>A</code> and <code>B</code> both having a shape of 2 x 10 x 5 x 2</p>
<p>I can get the max values and their indices with <code>values, indices = A.max(dim=-2)</code></p>
<p>How do I use the <code>indices</code> to select the corresponding values in tensor <code>B</code>?</p>

username: KarthikR
cooked: <p>I tried with a smaller tensor but should work for you:</p>
<p>input1  = torch.randn(2,2,3)<br>
input2  = torch.randn(2,2,3)<br>
print(input2)</p>
<p>values1, indices1 = input1.max(dim=-1,  keepdim=True)<br>
print(’\n’,indices1, ‘\n’)</p>
<p>print(torch.gather(input2, -1, indices1))</p>

username: jjhh
cooked: <p>Legend, thank you very much Karthik.</p>

username: dGermen
cooked: <p>I am having a similar problem, rather than pulling the values from the A, I want to edit the values at the indexes’ locations. Any suggestion?</p>
Adding GT Boxes to Proposals in RoIHeads

username: willjstone
cooked: <p>Hi There, and thanks in advance for your time.</p>
<p>In the <code>select_training_samples</code> method of the <code>RoIHeads</code> class (from <code>torchvision/models/detection/roi_heads.py</code>), the ground truth boxes for each image are concatenated onto the proposal tensors via the <code>add_gt_proposals</code> method. A few lines later, the <code>assign_targets_to_proposals</code> method is called, and if I understand this correctly it will end up matching the GT targets perfectly to the same GT boxes that were just added to the proposal tensors. I am not sure why this is desirable.</p>
<p>Is my understanding correct? If so, why is this happening?</p>
Inference my model on TRT Engine file

username: salidw
cooked: <p>Hello,</p>
<p>I had converted my *.pt file to onnx and validated its inference. Everything was OK.<br>
When I converted my Onnx file to TRT Engine file and did inference it on Jetson Nano.<br>
The predicted output was ZERO matrix like the below (Even my image input has values).</p>
<p>Predicted output:<br>
[[ 0.  0.  0. …,  0.  0.  0.]<br>
[ 0.  0.  0. …,  0.  0.  0.]<br>
[ 0.  0.  0. …,  0.  0.  0.]<br>
…,<br>
[ 0.  0.  0. …,  0.  0.  0.]<br>
[ 0.  0.  0. …,  0.  0.  0.]<br>
[ 0.  0.  0. …,  0.  0.  0.]]</p>
<p>Image input (as Numpy): For example<br>
[[[ 0.73339844  0.74121094  0.74121094 …,  0.1451416   0.11767578<br>
0.12158203]<br>
[ 0.73339844  0.73339844  0.73339844 …,  0.08233643  0.06274414<br>
0.10198975]<br>
[ 0.73730469  0.72558594  0.74121094 …,  0.07452393  0.05883789<br>
0.10198975] ]]</p>
<p>This is my inference function:<br>
<strong>def predict(img):</strong><br>
cuda.memcpy_htod_async(d_input, img, stream)<br>
context.execute_async_v2(bindings, stream.handle, None)<br>
cuda.memcpy_dtoh_async(output, d_output, stream)<br>
stream.synchronize()<br>
return output</p>
<p>Please advise why the inference result was ZERO matrix. Thanks.</p>

username: ptrblck
cooked: <p>I would recommend to post the question in a TensorRT board or repository, as it doesn’t seem to be PyTorch-specific.</p>
How `nn.Embedding` works with DistributedDataParallel?

username: jh_shim
cooked: <p>As I know, DDP (DistributedDataParallel) works only when all parameters of the given module participate to calculate loss.<br>
In case of <code>nn.Embedding</code>, I think some parameters of the module can be not used in forward pass.<br>
However, Transformers such as BERT works well with DDP.<br>
I don’t understand how <code>nn.Embedding</code> can work with DDP.</p>

username: ptrblck
cooked: <p>Even if not all indices of the embedding weight matrix are used, the parameter itself would still be used and would thus get a valid gradient (zeros for the unused indices), so DDP shouldn’t complain about it.</p>
Defining one class in another one

username: saran
cooked: <p>How to implement this part of tensorflow code  in pytorch i have tried<br>
but I am getting the “Not Implemented error”</p>
<p>this (ConvBNR) is subclass implemented in main class(ConvolutionNet)</p>
<pre><code class="lang-auto">class ConvBNR(keras.Model):  
           def __init__(self, ch, kernel_size=3, strides=1, padding='same'):
                 super(ConvBNR, self).__init__()
        
                   self.model = keras.models.Sequential([
                   layers.Conv2D(ch, kernel_size =  kernel_size, strides=strides, padding=padding,
                          kernel_regularizer=tf.keras.regularizers.l2(0.01)),
                  layers.BatchNormalization(),
                  ])
        
         def call(self, x, training=None): 
         x = self.model(x, training=training) 
         return x ""






class ConvolutionNet(Model): # Vitamon network except inception layer
    # Set layers.
    def __init__(self, num_classes):
        super (ConvolutionNetself).__init__()
        # Convolution Layer with 32 filters and a kernel size of 5.
        self.conv1 = ConvBNR(64)
        
        # self.concat1 = layers.Concatenate()

        # Convolution Layer with 64 filters and a kernel size of 3.
        self.conv2 = ConvBNR(64)
        # Max Pooling (down-sampling) with kernel size of 2 and strides of 2. 
        self.maxpool1 = layers.MaxPool2D(2, strides=2)
        
        self.conv3 = ConvBNR(64, kernel_size=3)
        # Max Pooling (down-sampling) with kernel size 
        #some others layers  ....
   def call(self, x, training=False): 
        x = self.conv1(x, training=training)
        # print(x.shape)
        x = self.conv2(x, training=training)
        x = self.maxpool1(x)
        x = self.conv3(x, training=training)
</code></pre>

username: ptrblck
cooked: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/implementing-a-sub-class-in-parent-class-in-pytorch/155738">here</a> or what would the difference be?</p>
Manually update model parameters from a vector

username: romano-francesco
cooked: <p>Hi everyone, I’m tryng to update manually the parameter of my model by a simple update rule like: new_param = old_param + lr * vector. I don’t know how to do it properly since online i found how to manually update from grads or multiplyng params with a constant, but in my case i need to update each single parameter differently and my vector is with monodimensional shape, while the model parameters are not organized like this. It’s my first experience with pytorch, I hope you can help me</p>

username: Karthik_Ganesan
cooked: <p>Hi, you need to provide more info. What are you updating the parameters for? Is this for a single layer or all layers? if its multiple layers, is it a different <code>vector</code> for each layer?</p>
<p>Is this <code>vector</code> static or is it trained along with the network?</p>

username: romano-francesco
cooked: <p>The vector is a monodimensional array that covers all the parameters os all the layers of the network. It is a natural gradient update for a RL algorithm and it changes at each iteration of the algo, but inside the single iteration it can be considered static. My problem is that doing the usual: for param in model.parameters(), we have that param has a particular shape. Maybe I should cut and reshape my vector in a way to make it compatible with the param shape</p>

username: Karthik_Ganesan
cooked: <p>I’m not sure how you can have a single vector shared by all the layers since the different layers will (presumably) have different shapes.</p>
<p>So it sounds like this is a trainable parameter per layer? If so, the easiest option might be for you to just add it as a parameter per layer. You can then adjust the learning rate for this parameter as needed (maybe per epoch as you say). That way, PyTorch will automatically train this vector for you without you having to adjust the sizes or anything.</p>
<p>If this isn’t what you want, can you maybe share a small code snippet to show what you have in mind or  what you are trying to do. I don’t think I still fully understand what you are trying to do.</p>
Pytorch dataset image loading

username: lanka
cooked: <p>hi,<br>
pls explain why is this error( <strong>NameError</strong> : name ‘ds’ is not defined)</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets, transforms

path = 'D:/work/train'

try:
    ds = datasets.ImageFolder(root=path)
except:
    pass

print(int(len(ds)))
</code></pre>
<p><strong>NameError</strong> : name ‘ds’ is not defined</p>

username: ptrblck
cooked: <p><code>ds = dataset.ImageFolder(root=path)</code> seems to raise an exception, which is caught in the <code>expect</code> block and thrown away since you are just using <code>pass</code> instead of notifying the user. <code>ds</code> is never initialized and <code>len(ds)</code> will raise the next exception.</p>

username: lanka
cooked: <p>yes. The problem is some folders(labels) are empty. so any other way to load the dataset with empty folders?</p>

username: ptrblck
cooked: <p>You need to provide a path to a valid dataset where each subfolder contains valid image data types.<br>
If some of the folders are empty, <a href="https://github.com/pytorch/vision/blob/d6e39ff76c82c7510f68a7aa637f015e7a86f217/torchvision/datasets/folder.py#L98-L103">this error</a> will be raised, so remove these empty folders.</p>
FSDP doesn't reduce the GPU memory usage

username: edrt
cooked: <p>hi, I’m working on refactor the <a href="https://github.com/facebookresearch/deit/blob/main/main.py#L336" rel="noopener nofollow ugc">deit</a> code to integrate FSDP to train deit_large with larger single gpu batch.<br>
After import related FSDP module, I simply change my code [ref this <a href="https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/" rel="noopener nofollow ugc">link</a>] to</p>
<pre><code class="lang-auto">if args.distributed:
        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
        model = FSDP(
            model,
            fsdp_auto_wrap_policy=default_auto_wrap_policy,
        )
</code></pre>
<p>then I hit two problems:</p>
<ol>
<li>the fsdp model cannot run with model EMA.</li>
<li>as metioned above, the GPU memory usage is not reduced as expected.</li>
</ol>
<p>Here is an example for Q2:<br>
run with FSDP:</p>
<pre><code class="lang-auto">GPU memory by nvidia-smi: 
18191MiB
terminal output by deit:
Epoch: [0]  [  200/40036]  eta: 10:42:06  lr: 0.000001  loss: 0.5857 (0.6690)  
time: 1.0125  data: 0.0036  max mem: 13402
</code></pre>
<p>run without FSDP:</p>
<pre><code class="lang-auto">GPU memory by nvidia-smi: 
19569MiB
terminal output by deit: 
Epoch: [0]  [ 200/5004]  eta: 1:47:13  lr: 0.000001  loss: 0.7104 (0.7275)  
time: 1.1631  data: 0.0002  max mem: 17334
</code></pre>
<p>Both of the stats are run with:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch \
           --nproc_per_node=8 --use_env train.py \
           --model deit_large_patch16_LS \
           --data-path ${imagenet_dir} \
           --output_dir ${output_dir} \
           --nb-classes 1000 \
           --batch 32 \
           --lr 3e-3 --epochs 1 \
           --weight-decay 0.05 --sched cosine \
           --input-size 224 \
           --reprob 0.0 \
           --smoothing 0.0 --warmup-epochs 5 --drop 0.0 \
           --seed 0 --opt lamb \
           --warmup-lr 1e-6 --mixup .8 --drop-path 0.6 --cutmix 1.0 \
           --unscale-lr --repeated-aug \
           --bce-loss  \
           --color-jitter 0.3 --ThreeAugment \
           --no-model-ema
</code></pre>
<p>Can anyone help?</p>

username: pbelevich
cooked: <p><a class="mention" href="/u/edrt">@edrt</a> you should not do</p>
<pre><code class="lang-auto">model = DDP(model, ...)
model = FSDP(model, ...)
</code></pre>
<p>please look carefully at the tutorial that you use, it actually does:</p>
<pre><code class="lang-auto">model = DDP(model(), ...)
fsdp_model = FSDP(model(), ...)
</code></pre>
<p>So, just remove DDP line</p>

username: edrt
cooked: <p>Thanks for your reply.<br>
I tried the <code>model()</code> instead of <code>model</code> first, but the error came out with missing input of calling forward pass with <code>model()</code>.<br>
I also tried to remove the DDP line, but the gpu memory usage didn’t reduce as expected.</p>

username: pbelevich
cooked: <p><a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a> could you please help?</p>
[HIRING] Lead Machine Learning Engineer 💰 200,000 - 220,000 USD / year

username: jharp
cooked: <p>[HIRING][Chicago, Illinois, USA, Onsite / Remote]</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/office.png?v=12" title=":office:" class="emoji" alt=":office:" loading="lazy" width="20" height="20"> Cision, based in Chicago, Illinois <img src="https://discuss.pytorch.org/images/emoji/apple/us.png?v=12" title=":us:" class="emoji" alt=":us:" loading="lazy" width="20" height="20"> is looking for a Lead Machine Learning Engineer</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/wrench.png?v=12" title=":wrench:" class="emoji" alt=":wrench:" loading="lazy" width="20" height="20"> Tech used: Machine Learning, Network, PyTorch, Spark, TensorFlow</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/moneybag.png?v=12" title=":moneybag:" class="emoji" alt=":moneybag:" loading="lazy" width="20" height="20"> 200,000 - 220,000 USD / year</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/mag_right.png?v=12" title=":mag_right:" class="emoji" alt=":mag_right:" loading="lazy" width="20" height="20"> More details and option to apply: <a href="https://devitjobs.us/jobs/Cision-Lead-Machine-Learning-Engineer?utm_source=otherg&amp;utm_medium=otherg&amp;utm_campaign=otherg" class="inline-onebox" rel="noopener nofollow ugc">Lead Machine Learning Engineer Job in Chicago | Cision</a></p>
Saving the predictions to CSV

username: Wookie
cooked: <p>I am doing something wrong trying to append the results from the models prediction.</p>
<p>My attempt is to append the predictions to a new variable and the convert that variable with the over 40,000 outcomes to array to save into CSV.</p>
<p>My attempt is as follows:</p>
<pre><code class="lang-auto">totalpreds = []
import seaborn as sns
nb_classes = 2

confusion_matrix = torch.zeros(nb_classes, nb_classes)
with torch.no_grad():
    for i, (inputs, classes) in enumerate(testloader):
        inputs = inputs
        classes = classes
        outputs = clf(inputs)
        _, preds = torch.max(outputs, 1)
        
        # Saving predictions
        arr = preds.cpu().numpy()
        totalpreds.append(arr)
        
# write CSV
np.savetxt('output.csv', totalpreds)

</code></pre>
<p>It’s not working… I am sure it’s something wrong in my thinking here. Any help would be appreciated!</p>

username: ArchieGertsman
cooked: <p>Consider working with Pandas when dealing with CSVs:</p>
<pre><code class="lang-auto">import pandas as pd

n_preds = 4
output_dim = 3

# random data in your format
preds = [np.random.rand(output_dim) for _ in range(n_preds)]

# convert data into dataframe
df = pd.DataFrame(preds)

# save data to csv
df.to_csv('output.csv')
</code></pre>
How do I raise a constant to the power of a tensor?

username: Andrew_Legend
cooked: <p>I want to take raise 2 to the power of a tensor A and then have it return another tensor B but with B with all elements of B be 2 but raised to the coresponding exponent. For example:</p>
<pre><code class="lang-auto">A= [1, 2, 3]

B= Function(2, A)

B= [2, 4, 8]
</code></pre>
<p>However I still haven’t figured out how to make such a function. Any ideas?</p>

username: AlphaBetaGamma96
cooked: <p>Could you try <code>2**A</code>?</p>

username: Andrew_Legend
cooked: <p>Wouldn’t that cause issues with back-propagation tho?</p>

username: AlphaBetaGamma96
cooked: <p>I don’t see why not, give it a go and see if it works. Or you can quickly do something like,</p>
<pre><code class="lang-auto">A = torch.randn(3, requires_grad=True)
B = 2**A
print(B.grad_fn) #if there's a grad function, back-prop is fine
</code></pre>

username: Andrew_Legend
cooked: <p>It prints &lt;PowBackward1 object at 0x00000173EF314808&gt; so I suppose it works</p>
Gradient on parts of feature maps

username: L-Reichardt
cooked: <p>Hello,</p>
<p>I need some advice on how to best approach this problem. I want to compare feature maps between a network with K-classes. For simplicity, lets presume the feature maps (with batch size 1) are each a vector/tensor with size (1, 1, K).</p>
<p>I only want to compare and update depending on single values, which will change their position in K, depending on the input. If I slice the feature map and set grad=False, It will set the entire gradient to false. Basically I am looking for something like an “ignore index”, where I can train only on the result of e.g. (1, 1, K=4), not updating for the rest of K-4 values.</p>
<p>Id appreciate any help and pointers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Best Regards,<br>
L</p>
How to accelerate this vanilla program with 2 torch.topk()?

username: fried-chicken
cooked: <p>What I want to do is:</p>
<p>given a tensor with shape like [32,100] as input, I want to get 2 tensor with shape [32,50] and [32,50] as output, where the first [32,50] contains the top-50-largest elements in each rows and the second [32,50] contains the top-50-smallest.</p>
<p>So far, I know I can use torch.topk() twice to get the desired two tensors, but I assume it isn’t efficient enough (since topk is time-consuming) and I think there should be someway to directly get the second tensor by the result of first topk.</p>
<p>All in all, I want to know what is the best practice for implement program above with only one calling of torch.topk()?</p>

username: KFrank
cooked: <p>Hi Fried!</p>
<aside class="quote no-group" data-username="fried-chicken" data-post="1" data-topic="155773" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fried-chicken/48/45494_2.png" class="avatar"> fried-chicken:</div>
<blockquote>
<p>the first [32,50] contains the top-50-largest elements in each rows and the second [32,50] contains the top-50-smallest.</p>
</blockquote>
</aside>
<p>In this specific use case you want, at the end, all 100 elements split up<br>
a certain way.  So you will be better off calling <code>sort()</code> once, rather than<br>
calling <code>topk()</code> one or more times:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.12.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; t = torch.randn  (32, 100)
&gt;&gt;&gt;
&gt;&gt;&gt; tsort = t.sort (dim = 1)[0]
&gt;&gt;&gt;
&gt;&gt;&gt; topbig = tsort[:, 50:].flip (1)
&gt;&gt;&gt; topsml = tsort[:, :50]
&gt;&gt;&gt;
&gt;&gt;&gt; topbig[0, :5]
tensor([3.0777, 2.6153, 2.5442, 2.3476, 1.8371])
&gt;&gt;&gt; topsml[0, :5]
tensor([-2.2356, -2.2051, -2.1979, -2.1629, -1.8965])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: fried-chicken
cooked: <p>Stupid for me. Definitely works for me. Thank you a lot.</p>
Shouldn't use resnet for not image type data?

username: farahnaz
cooked: <p>i have a data that it is ROICorrelation matrix from fmri image. i fed it into resnet and got a good performance but I don’t know if it is right to use resnet for nonimage data or not?</p>

username: ptrblck
cooked: <p>While the original use case was an ImageNet classification, I wouldn’t see any valid points against training or fine-tuning this model to your use case.<br>
It also seems to work, which is great. What is your concern about using this model?</p>

username: farahnaz
cooked: <p>HI. thank you for your response.<br>
my concern is that my input data isn’t image type.it is kind of extracted feature from fmri images.</p>
Resnet 50 with contrastive loss

username: Aysan
cooked: <p>Hi,<br>
I want to do multi-class classification with a pre-trained resnet50 with contrastive loss. I want to calculate the loss between the actual label and the label predicted by the model. But while searching, I came across some questions.<br>
first, should I necessarily use supervised contrastive learning to use contrastive loss? The method is explained in this <a href="https://github.com/HobbitLong/SupContrast/tree/8117ece43eea5a38fb154f587601ceeadbe416b3" rel="noopener nofollow ugc">link</a>. It seems that it first generates more data from the initial data and then calculates the loss.<br>
secondly, there is also some prepared function like <a href="https://kevinmusgrave.github.io/pytorch-metric-learning/losses/#contrastiveloss" rel="noopener nofollow ugc">this</a>, But I do not know how to use them. or do you know other prepared functions?<br>
Thank you in advance.</p>
Transformer not able to do bracket matching

username: yamsk
cooked: <p>I have been trying to apply transformers to classify simpler sequences. One such example is in my earlier post with working code:</p>
<aside class="quote quote-modified" data-post="1" data-topic="155369">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/8c91f0/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/transformer-layer-weights-not-updating/155369">Transformer layer weights not updating</a> 
  </div>
  <blockquote>
    Hi: 
I am trying to classify simple sequence data. 
Examples: 
([‘bbRcabbSab’, 
‘abRbbccSca’, 
‘baabRcaaaS’, 
‘abRbabaSaa’, 
‘acRcbacSbc’, 
‘bcRbaacSbc’, 
‘RbbaaabaSc’, 
‘abRacabSba’, 
‘RabcbSbcba’, 
‘ccRbcaaSba’], 
[1, 1, 0, 1, 1, 1, 0, 1, 0, 1]) 
If R is in position 2 and S is in position len(s)-3 then the target is 1 and 0 otherwise. 
I am trying to use a transformer for this. The model I am using is: 
class PositionalEncoding(nn.Module):
    def __init__(self, dim_model, dropout_p, max_len=5…
  </blockquote>
</aside>

<p>However, the code does not work for the following bracket matching examples. I was told that matching brackets is quite a tricky problem for transformers as it has high circuit complexity. But if someone wants to take a crack at it. Or try it for any other simpler sequences.</p>
<p>Bracket matching examples.<br>
([‘()()()()()’, ‘()()))()()’, ‘())()())()’, ‘(((())))))’, ‘()()))()()’],<br>
[1, 0, 0, 0, 0])</p>
Seq2Seq focused attention regularization

username: marcpaga
cooked: <p>I am working on a seq2seq model with attention.<br>
When visualizing attention sometimes the attention score is spread across the time axis, and sometimes it clearly focusses on a particular time point (see below).</p>
<pre><code class="lang-auto"># toy example
spread_attention = [0.1, 0.2, 0.2, 0.2, 0.1, 0.1, 0.1]
focused_attention = [0.0, 0.0, 0.9, 0.1, 0.0, 0.0, 0.0]
</code></pre>
<p>I was wondering if there’s a way to reinforce the focused attention instead of the spread attention. Perhaps as an additional loss term, like a regularization factor.</p>
<pre><code class="lang-auto">reg_att = attention_spreadness(attention)
loss = cross_entropry_loss + reg_att
loss.backward()
</code></pre>
<p>I was thinking of using a negative L1 or L2 regularization, but they both use the sum of weights, and in attention all weights sum up to 1. Then I thought about using a threshold layer:</p>
<pre><code class="lang-auto">def attention_spreadness(attention):
    return -torch.nn.functional.threshold(-attention, -0.5, 0).sum()

attention_spreadness(torch.Tensor([0.1, 0.2, 0.2, 0.2, 0.1, 0.1, 0.1])) 
# &gt; 1.0
attention_spreadness(torch.Tensor([0.0, 0.0, 0.9, 0.1, 0.0, 0.0, 0.0])) 
# &gt; 0.1
</code></pre>
<p>In this manner, it basically adds the values for all the scores below a certain threshold, and that can be used as a penalizing term.</p>
<p>Does anyone know better approaches to do this? Perhaps some literature on similar attempts at this?</p>
Are there any techniques that haven't been researched in similar image search?

username: YuhskeHujisaki
cooked: <p>I’d like to study similar image search, but is there anything still uncertain in this field? What do you think is necessary, although it has not been studied yet?</p>

username: Andrej
cooked: <p>Hello,<br>
I am not an expert in this field, but is it this what you’re looking for?</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/Zhenye-Na/image-similarity-using-deep-ranking">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/Zhenye-Na/image-similarity-using-deep-ranking" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/acb249f7ae5e185320be0f8450322a0a87bfc938f1005769c56772c18a536e57/Zhenye-Na/image-similarity-using-deep-ranking" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/Zhenye-Na/image-similarity-using-deep-ranking" target="_blank" rel="noopener nofollow ugc">GitHub - Zhenye-Na/image-similarity-using-deep-ranking: 🖼️  𝖀𝖓𝖔𝖋𝖋𝖎𝖈𝖎𝖆𝖑...</a></h3>

  <p>🖼️  𝖀𝖓𝖔𝖋𝖋𝖎𝖈𝖎𝖆𝖑 PyTorch implementation of "Learning Fine-grained Image Similarity with Deep Ranking" (arXiv:1404.4661) - GitHub - Zhenye-Na/image-similarity-using-deep-ranking: 🖼️  𝖀𝖓𝖔𝖋𝖋𝖎𝖈...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Intel Job: Pytorch developer with experience on large scale training

username: joshua_mora
cooked: <p>Hello.<br>
Our team at Intel is looking for a pytorch developer with experience on large scale training.<br>
More details on the job description can be found at<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://intel.wd1.myworkdayjobs.com/assets/images/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://intel.wd1.myworkdayjobs.com/External/job/US-Texas-Austin/AI-Workload-Performance-Engineer_JR0146901" target="_blank" rel="noopener nofollow ugc">intel.wd1.myworkdayjobs.com</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://intel.wd1.myworkdayjobs.com/External/job/US-Texas-Austin/AI-Workload-Performance-Engineer_JR0146901" target="_blank" rel="noopener nofollow ugc">Workday</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Additionally, you can ping me directly via the message system or at linkedin at  <a href="https://www.linkedin.com/in/joshuamora/" rel="noopener nofollow ugc">https://www.linkedin.com/in/joshuamora/</a></p>

username: OuisYasser
cooked: <p>Sorry I know I’m not a good developer but I wanna ask what are the requirements ? Or the tasks in other words ?<br>
Thank you</p>

username: joshua_mora
cooked: <p>you can read the job description to get an idea of what skills we are looking for.</p>
<p>Regards,<br>
Joshua</p>
Does ctx.save_for_backward() do anything in inference mode?

username: zfzhang
cooked: <p>Hi,<br>
When defining custom Function class, we define static forward()/backward() methods, and in forward() we use <strong>ctx.save_for_backward(…)</strong> to record the input tensors for backward pass.</p>
<p>I’m wondering, when in inference mode (training = false), will the above <strong>ctx.save_for_backward(…)</strong> call still record the tensors for the never arriving backward pass?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Based on <a href="https://discuss.pytorch.org/t/save-for-backward-in-no-grad-mode/57316/2">this response</a> from <a class="mention" href="/u/alband">@albanD</a> it should not save any tensors if it’s executed in a <code>with torch.no_grad()</code> context (and I would also assume the same for <code>with torch.inference_mode()</code>).</p>

username: zfzhang
cooked: <p>Thanks! Didn’t realized there was a similar question.</p>
Gradient function of a nn.module

username: Tia
cooked: <p>Hello all,  suppose I have a scalar function f(t,x) that is a nn.Module:</p>
<pre><code class="lang-python">class f(nn.Module):
    def __init__(self):
        super(f, self).__init__()

        self.net = nn.Sequential(
            nn.Linear(2, 50),
            nn.Tanh(),
            nn.Linear(50, 1),
        )

    def forward(self, t, x):
        return self.net(x)
</code></pre>
<p>I would like to build a <strong>nn.module</strong> that takes the input (t,x) and outputs the gradient of f with respect to x, roughly like below:</p>
<pre><code class="lang-python">class dfdx(nn.Module):
    def __init__(self):
        super(dfdx, self).__init__()

    def forward(self, t, x):
        return df_dx
</code></pre>
<p>In this case, x=[x1, x2] is a 2x1 tensor, so df_dx should be a 2x1 tensor [df/dx1, df/dx2].</p>
<p>I found a <a href="https://stackoverflow.com/questions/64978232/pytorch-how-to-compute-second-order-jacobian" rel="noopener nofollow ugc">similar question</a>,  and tried the following</p>
<pre><code class="lang-python">class f(nn.Module):

    def __init__(self):
        super(f, self).__init__()

        self.net = nn.Sequential(
            nn.Linear(2, 50),
            nn.Tanh(),
            nn.Linear(50, 1),
        )
    def forward(self, t, x):
        return self.net(x)

    def compute_u_x(self, t, x):
        self.u_x = torch.autograd.functional.jacobian(self, x, create_graph=True)
        self.u_x = torch.squeeze(self.u_x)
        return self.u_x
</code></pre>
<p>and hope that</p>
<pre><code class="lang-python">func = f()
func.compute_u_x
</code></pre>
<p>would achieve the goal (Though I am not sure if <code>func.compute_u_x</code> is a <strong>nn.module</strong>). But got the following error:</p>
<pre><code class="lang-python">TypeError: forward() missing 1 required positional argument: 'x'
</code></pre>
<p>I understand why I get this error but have no idea how to fix it. Is there any way to solve the problem? Any help is highly appreciated!</p>
How can I get the vector of features for each pixel using a Resnet?

username: Kathena
cooked: <p>how can I get vector of features (features extraction) for pixel by pixel using a Resnet 50 or a Resnet 101 (Pytorch)? My image resolution is 474x496, so I want to extract 474x496 vectors at the end.</p>

username: ptrblck
cooked: <p>I don’t think that’s possible without changing the model architecture, since the “standard” ResNets will decrease the spatial size of the activation maps, so that you won’t be able to directly map the features to the input pixels.<br>
You could of course try to keep the spatial size equal by changing the conv layer setups and removing the pooling layers, but I would expect this model might not be that easy to train as the activations would be huge.</p>
Combine two torch array with a alternative style

username: miraboreasu
cooked: <pre><code class="lang-auto">x = torch.ones(2, 2, 2, requires_grad=False, dtype=torch.float64)
y = torch.zeros(2, 2, 2, requires_grad=False, dtype=torch.float64)
</code></pre>
<p>How can I make the result looks like</p>
<pre><code class="lang-auto">Slicing[:,:,0]
1010
1010
Slicing[:,:,1]
1010
1010
</code></pre>
<p>Thank you</p>

username: KFrank
cooked: <p>Hi Miraboreasu!</p>
<aside class="quote no-group" data-username="miraboreasu" data-post="1" data-topic="155705" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/6bbea6/48.png" class="avatar"> miraboreasu:</div>
<blockquote>
<pre><code class="lang-auto">Slicing[:,:,0]
1010
1010
Slicing[:,:,1]
1010
1010
</code></pre>
</blockquote>
</aside>
<p>If I understand your use case correctly, the following should work:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; x = torch.ones (2, 2, 2, requires_grad = False, dtype = torch.float64)
&gt;&gt;&gt; y = torch.zeros (2, 2, 2, requires_grad = False, dtype = torch.float64)
&gt;&gt;&gt; torch.stack ((x, y), dim = 2).reshape (2, 4, 2)
tensor([[[1., 1.],
         [0., 0.],
         [1., 1.],
         [0., 0.]],

        [[1., 1.],
         [0., 0.],
         [1., 1.],
         [0., 0.]]], dtype=torch.float64)
&gt;&gt;&gt; #   thus:
&gt;&gt;&gt; torch.stack ((x, y), dim = 2).reshape (2, 4, 2)[:, :, 0]
tensor([[1., 0., 1., 0.],
        [1., 0., 1., 0.]], dtype=torch.float64)
&gt;&gt;&gt; torch.stack ((x, y), dim = 2).reshape (2, 4, 2)[:, :, 1]
tensor([[1., 0., 1., 0.],
        [1., 0., 1., 0.]], dtype=torch.float64)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Saving map<string, map<string, torch::Tensor>> objects in LibTorch

username: JamesDickens
cooked: <p>My use case in LibTorch is as follows. I have a map&lt;string, map&lt;string, torch::Tensor&gt;&gt;<br>
object in C++ that I would like to serialize for FAST reading for a torch::data::Dataset&lt;&gt; subclass.</p>
<p>Supposing I write a function</p>
<pre><code class="lang-auto"># include &lt;torch/torch.h&gt;

using namespace std;

void save_serialized_obj(map&lt;string, map&lt;string, torch::Tensor&gt;&gt;, string save_fp) {
      

} 

map&lt;string, map&lt;string, torch::Tensor&gt;&gt; load_serialized_obj(string save_fp) {


}
</code></pre>
<p>What is the easiest/fastest way to achieve this functionality?</p>
Probleme whith torch_geometric.nn.TopKPooling

username: DRISS_ELALAOUI
cooked: <p>please I wanted to use the maximum using TopKPooling from torch_geometric.nn in this way:<br>
in the init() part</p>
<pre><code class="lang-auto">self.pool1 = TopKPooling(8, ratio=1)
</code></pre>
<p>in the forward() part</p>
<pre><code class="lang-auto">x, edge_index, _, batch, _, _ = self.pool1(x, edges, None, batch)
</code></pre>
<p>whit batch = 256<br>
but i got this error message</p>
<pre><code class="lang-auto">RuntimeError: The expanded size of the tensor (256) must match the existing size (1058) at non-singleton dimension 0.  Target sizes: [256].  Tensor sizes: [1058]
</code></pre>
<p>can you help me to solve this problem?<br>
and thank you</p>
Torch.sum - too many indices for tensor of dimension 1

username: DRISS_ELALAOUI
cooked: <p>please I have a “targets” and “outputs” tensor which have a size [batch-size,slate_length]. “targets” contains values: True or False and “outputs” contains values in [0; 1]<br>
when I execute these lines:</p>
<pre><code class="lang-auto">outputs_order = torch.argsort(outputs, descending=True, dim=-1)
targets_sorted_by_outputs = torch.gather(targets, dim=-1, index=outputs_order)
hits_score = torch.sum(targets_sorted_by_outputs[:,:k], dim=1) / torch.sum(targets, dim=1)
</code></pre>
<p>I get this error message</p>
<pre><code class="lang-auto">13    outputs_order = torch.argsort(outputs, descending=True, dim=-1)
     14    targets_sorted_by_outputs = torch.gather(targets, dim=-1, index=outputs_order)
---&gt; 15    hits_score = torch.sum(targets_sorted_by_outputs[:,:k], dim=1) / torch.sum(targets[:,:k], dim=1)

IndexError: too many indices for tensor of dimension 1
</code></pre>
<p>how i can solve this problem? and thank you</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="DRISS_ELALAOUI" data-post="1" data-topic="155625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/driss_elalaoui/48/35900_2.png" class="avatar"> DRISS_ELALAOUI:</div>
<blockquote>
<p>I have a “targets” and “outputs” tensor which have a size [batch-size,slate_length]</p>
</blockquote>
</aside>
<p>I don’t think this is the case, as <code>targets_sorted_by_output</code> seems to have a single dimension and raises the <code>IndexError</code> as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">k = 5
targets_sorted_by_outputs = torch.randn(10)
targets_sorted_by_outputs[:, :k] 
# IndexError: too many indices for tensor of dimension 1
</code></pre>

username: DRISS_ELALAOUI
cooked: <p>Thank you very much for this valuable help</p>
Parallel training of a network with multiple branches

username: haruko
cooked: <p>Hi, I am a beginner in Python and PyTorch. So please forgive me if it is a simple question.</p>
<p>I have a model with separate branches like</p>
<pre><code class="lang-python">class mynet(nn.Module):
    def __init__(self, num):
        super(mynet, self).__init__()
        self.subnet = nn.ModuleList()
        for _ in range(num):
            self.subnet.append(mysubnet())
    def forward(self, x):
        y = []
        for i in range(len(self.subnet)):
            y.append(self.subnet[i](x[i]))
        return y
</code></pre>
<p>Since the subnets in the model are small,  I wonder if it is possible to compute <code>forward</code> and  <code>loss.backward()</code>  in parallel?</p>
<p>Thanks in advance!</p>

username: eqy
cooked: <p>Without multiple hardware devices it might be difficult to realize a speedup if each subnet is contending for the same hardware resources. What are the layers used in each subnet? Would it be possible to combine them in some way (e.g., grouped convolutions for parallel convolutions)?</p>

username: rob-miller
cooked: <p>Please forgive me for hijacking this thread, but I do have the same question and would very much like some more detail and especially syntax.</p>
<p>In big picture I’m looking to define something like the net image in <a href="https://discuss.pytorch.org/t/how-to-train-multi-branch-output-network/95199">this post</a>, but with the arrows reversed.  In other words, I want the input to be a set of identical-size tensors which each process through one or more layers of their own subnets (learning local representations) before feeding into one larger layer for further processing.  The target hardware is a GPU, and as the original poster noted it seems like looping over the subnets will be inefficient.</p>
<p>In my specific case I have a set of voxels of interest (so not a cube), and within each voxel a specific x,y,z coordinate (real values) and a 10-bit one-hot encoding for the point in the voxel (also reals since tensors can have only one dtype).  Thus for this example an input size of (N, 13) where N is fixed and on the order of 50-100 depending on the model.  Ideally I want each of the 13-element vectors to process in their own fully-connected multi-layer subnets before feeding into a single larger layer.</p>
<p>The use of convolutional layers is intriguing, but I am not clear on the required syntax to define non-overlapping subnets without necessitating smaller output layers for the subnets as I see in most examples for convolutional layers.</p>
<p>Interested to read constructive suggestions and comments on any of this.</p>