While using torch_xla.it shows the following error

Question:
518   torch_xla._XLAC._xla_step_marker(
519       torch_xla._XLAC._xla_get_default_device(), [],

– 520       wait=xu.getenv_as(‘XLA_SYNC_WAIT’, bool, False))
521   # Only emit metrics from the first local device index, to avoid emitting the
522   # same values from different threads.
RuntimeError: Failed precondition: From /job:tpu_worker/replica:0/task:0:
The TPU system has not been initialized.
[[{{node XRTCompile}}]]

Context:
Hi, for all pytorch/xla (or pytorch on tpu) questions, please open an issue in https://github.com/pytorch/xla 65.
Thanks!


Question:
Hi everyone,
I am currently using gcp in order to test the TPU on pytorch.
My code comes from https://colab.research.google.com/github/pytorch/xla/blob/master/contrib/colab/mnist-training-xrt-1-15.ipynb 2 . I change or remove some part of codes like the TPU_ADDRESS.
During the training, I got an exeption message :

2019-11-21 18:22:14.125827: I    2364 tensorflow/compiler/xla/xla_client/mesh_service.cc:168] Waiting to connect to client mesh master (300 seconds) localhost:40359
2019-11-21 18:22:14.141768: I    2369 tensorflow/compiler/xla/xla_client/mesh_service.cc:168] Waiting to connect to client mesh master (300 seconds) localhost:40359
2019-11-21 18:22:14.137212: I    2365 tensorflow/compiler/xla/xla_client/mesh_service.cc:168] Waiting to connect to client mesh master (300 seconds) localhost:40359
2019-11-21 18:22:14.159541: I    2376 tensorflow/compiler/xla/xla_client/mesh_service.cc:168] Waiting to connect to client mesh master (300 seconds) localhost:40359
2019-11-21 18:22:14.170106: I    2380 tensorflow/compiler/xla/xla_client/mesh_service.cc:168] Waiting to connect to client mesh master (300 seconds) localhost:40359
2019-11-21 18:22:14.182444: I    2384 tensorflow/compiler/xla/xla_client/mesh_service.cc:168] Waiting to connect to client mesh master (300 seconds) localhost:40359
2019-11-21 18:22:14.191538: I    2392 tensorflow/compiler/xla/xla_client/mesh_service.cc:168] Waiting to connect to client mesh master (300 seconds) localhost:40359
Device :  xla:1
2019-11-21 18:22:22.612051: I    2364 tensorflow/compiler/xla/xla_client/computation_client.cc:195] Fetching mesh configuration for worker tpu_worker:0 from mesh service at localhost:40359
Device :  xla:0
2019-11-21 18:22:23.098120: I    2380 tensorflow/compiler/xla/xla_client/computation_client.cc:195] Fetching mesh configuration for worker tpu_worker:0 from mesh service at localhost:40359
Device :  xla:0
2019-11-21 18:22:23.538515: I    2384 tensorflow/compiler/xla/xla_client/computation_client.cc:195] Fetching mesh configuration for worker tpu_worker:0 from mesh service at localhost:40359
2019-11-21 18:22:23.689589: I    2376 tensorflow/compiler/xla/xla_client/computation_client.cc:195] Fetching mesh configuration for worker tpu_worker:0 from mesh service at localhost:40359
2019-11-21 18:22:23.776546: I    2392 tensorflow/compiler/xla/xla_client/computation_client.cc:195] Fetching mesh configuration for worker tpu_worker:0 from mesh service at localhost:40359
Device :  xla:0
Device :  xla:0
Device :  xla:0
2019-11-21 18:22:24.179791: I    2369 tensorflow/compiler/xla/xla_client/computation_client.cc:195] Fetching mesh configuration for worker tpu_worker:0 from mesh service at localhost:40359
2019-11-21 18:22:24.274278: I    2365 tensorflow/compiler/xla/xla_client/computation_client.cc:195] Fetching mesh configuration for worker tpu_worker:0 from mesh service at localhost:40359
Device :  xla:0
Device :  xla:0
Traceback (most recent call last):
File “test.py”, line 217, in
xmp.spawn(_mp_fn, args=(FLAGS,), nprocs=FLAGS[‘num_cores’],start_method=‘fork’)
File “/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py”, line 173, in spawn
start_method=start_method)
File “/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 149, in start_processes
while not context.join():
File “/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 107, in join
(error_index, name)
Exception: process 5 terminated with signal SIGSEGV

My code is this one :
 import collections
from datetime import datetime, timedelta
import os
import requests
import threading

_VersionConfig = collections.namedtuple('_VersionConfig', 'wheels,server')
VERSION = "xrt==1.15.0"  #@param ["xrt==1.15.0", "torch_xla==nightly"]
CONFIG = {
    'xrt==1.15.0': _VersionConfig('1.15', '1.15.0'),
    'torch_xla==nightly': _VersionConfig('nightly', 'XRT-dev{}'.format(
        (datetime.today() - timedelta(1)).strftime('%Y%m%d'))),
}[VERSION]
DIST_BUCKET = 'gs://tpu-pytorch/wheels'
TORCH_WHEEL = 'torch-{}-cp36-cp36m-linux_x86_64.whl'.format(CONFIG.wheels)
TORCH_XLA_WHEEL = 'torch_xla-{}-cp36-cp36m-linux_x86_64.whl'.format(CONFIG.wheels)
TORCHVISION_WHEEL = 'torchvision-{}-cp36-cp36m-linux_x86_64.whl'.format(CONFIG.wheels)

# Update TPU XRT version
def update_server_xrt():
  print('Updating server-side XRT to {} ...'.format(CONFIG.server))
  url = 'http://{TPU_ADDRESS}:8475/requestversion/{XRT_VERSION}'.format(
      TPU_ADDRESS=os.environ['TPU_IP_ADDRESS'].split(':')[0],
      XRT_VERSION=CONFIG.server,
  )
  print('Done updating server-side XRT: {}'.format(requests.post(url)))

#update = threading.Thread(target=update_server_xrt)
#update.start()
#update.join()

# Result Visualization Helper
import math
from matplotlib import pyplot as plt

M, N = 4, 6
RESULT_IMG_PATH = '/tmp/test_result.png'

def plot_results(images, labels, preds):
  images, labels, preds = images[:M*N], labels[:M*N], preds[:M*N]
  inv_norm = transforms.Normalize((-0.1307/0.3081,), (1/0.3081,))

  num_images = images.shape[0]
  fig, axes = plt.subplots(M, N, figsize=(11, 9))
  fig.suptitle('Correct / Predicted Labels (Red text for incorrect ones)')

  for i, ax in enumerate(fig.axes):
    ax.axis('off')
    if i >= num_images:
      continue
    img, label, prediction = images[i], labels[i], preds[i]
    img = inv_norm(img)
    img = img.squeeze() # [1,Y,X] -> [Y,X]
    label, prediction = label.item(), prediction.item()
    if label == prediction:
      ax.set_title(u'\u2713', color='blue', fontsize=22)
    else:
      ax.set_title(
          'X {}/{}'.format(label, prediction), color='red')
    ax.imshow(img)
  plt.savefig(RESULT_IMG_PATH, transparent=True)

  # Define Parameters
FLAGS = {}
FLAGS['datadir'] = "/tmp/mnist"
FLAGS['batch_size'] = 128
FLAGS['num_workers'] = 4
FLAGS['learning_rate'] = 0.01
FLAGS['momentum'] = 0.5
FLAGS['num_epochs'] = 10
FLAGS['num_cores'] = 8
FLAGS['log_steps'] = 20
FLAGS['metrics_debug'] = False

import numpy as np
import os
import time
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch_xla
import torch_xla.core.xla_model as xm
import torch_xla.debug.metrics as met
import torch_xla.distributed.parallel_loader as pl
import torch_xla.distributed.xla_multiprocessing as xmp
import torch_xla.utils.utils as xu
from torchvision import datasets, transforms

class MNIST(nn.Module):

  def __init__(self):
    super(MNIST, self).__init__()
    self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
    self.bn1 = nn.BatchNorm2d(10)
    self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
    self.bn2 = nn.BatchNorm2d(20)
    self.fc1 = nn.Linear(320, 50)
    self.fc2 = nn.Linear(50, 10)

  def forward(self, x):
    x = F.relu(F.max_pool2d(self.conv1(x), 2))
    x = self.bn1(x)
    x = F.relu(F.max_pool2d(self.conv2(x), 2))
    x = self.bn2(x)
    x = torch.flatten(x, 1)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return F.log_softmax(x, dim=1)

def train_mnist():
  torch.manual_seed(1)

  # Get and shard dataset into dataloaders
  norm = transforms.Normalize((0.1307,), (0.3081,))
  train_dataset = datasets.MNIST(
      os.path.join(FLAGS['datadir'], str(xm.get_ordinal())),
      train=True,
      download=True,
      transform=transforms.Compose(
          [transforms.ToTensor(), norm]))
  test_dataset = datasets.MNIST(
      os.path.join(FLAGS['datadir'], str(xm.get_ordinal())),
      train=False,
      download=True,
      transform=transforms.Compose(
          [transforms.ToTensor(), norm]))
  train_sampler = torch.utils.data.distributed.DistributedSampler(
    train_dataset,
    num_replicas=xm.xrt_world_size(),
    rank=xm.get_ordinal(),
    shuffle=True)
  train_loader = torch.utils.data.DataLoader(
      train_dataset,
      batch_size=FLAGS['batch_size'],
      sampler=train_sampler,
      num_workers=FLAGS['num_workers'],
      drop_last=True)
  test_loader = torch.utils.data.DataLoader(
      test_dataset,
      batch_size=FLAGS['batch_size'],
      shuffle=False,
      num_workers=FLAGS['num_workers'],
      drop_last=True)

  # Scale learning rate to world size
  lr = FLAGS['learning_rate'] * xm.xrt_world_size()

  # Get loss function, optimizer, and model
  device = xm.xla_device()
  print("Device : ", device)
  model = MNIST().to(device)
  optimizer = optim.SGD(model.parameters(), lr=lr, momentum=FLAGS['momentum'])
  loss_fn = nn.NLLLoss()

  def train_loop_fn(loader):
    tracker = xm.RateTracker()
    model.train()
    for x, (data, target) in enumerate(loader):
      optimizer.zero_grad()
      output = model(data)
      loss = loss_fn(output, target)
      loss.backward()
      xm.optimizer_step(optimizer)
      tracker.add(FLAGS['batch_size'])
      #if x % FLAGS['log_steps'] == 0:
       # print('[xla:{}]({}) Loss={:.5f} Rate={:.2f} GlobalRate={:.2f} Time={}'.format(
       #     xm.get_ordinal(), x, loss.item(), tracker.rate(),
      #      tracker.global_rate(), time.asctime()), flush=True)

  def test_loop_fn(loader):
    total_samples = 0
    correct = 0
    model.eval()
    data, pred, target = None, None, None
    for data, target in loader:
      output = model(data)
      pred = output.max(1, keepdim=True)[1]
      correct += pred.eq(target.view_as(pred)).sum().item()
      total_samples += data.size()[0]

    accuracy = 100.0 * correct / total_samples
    #print('[xla:{}] Accuracy={:.2f}%'.format(
       # xm.get_ordinal(), accuracy), flush=True)
    return accuracy, data, pred, target

  # Train and eval loops
  accuracy = 0.0
  data, pred, target = None, None, None
  for epoch in range(1, FLAGS['num_epochs'] + 1):
    begin = time.time()
    para_loader = pl.ParallelLoader(train_loader, [device])
    train_loop_fn(para_loader.per_device_loader(device))
    xm.master_print("Finished training epoch {0} in {1} sec".format(epoch, time.time()-begin))

    #para_loader = pl.ParallelLoader(test_loader, [device])
    #accuracy, data, pred, target  = test_loop_fn(para_loader.per_device_loader(device))
    #if FLAGS['metrics_debug']:
     # xm.master_print(met.metrics_report(), flush=True)

  return accuracy, data, pred, target

# Start training processes
def _mp_fn(rank, flags):
  global FLAGS
  FLAGS = flags
  torch.set_default_tensor_type('torch.FloatTensor')
  accuracy, data, pred, target = train_mnist()
  #if rank == 0:
    # Retrieve tensors that are on TPU core 0 and plot.
    #plot_results(data.cpu(), pred.cpu(), target.cpu())

xmp.spawn(_mp_fn, args=(FLAGS,), nprocs=FLAGS['num_cores'],start_method='fork')

#train_mnist()

Moreover if I use only train_mnist function instead of xmp.spawn, the training works but take around 13 seconds which is slower than in Colab (around 3 seconds) so I suspect than something is wrong.
The only difference I have with colab is that :

update = threading.Thread(target=update_server_xrt) is commented
put the good TPU_ADDRESS instead of Colab TPU IP

I used the custom image with pytorch XLA already installed on GCP. Should I installed something else ? I think there should be no issue with the code as it works on colab,
I am using torch XLA nightly.
I thank you for your advices

Context:
Hi, sorry for the late reply! Just want to let you know that for all pytorch/xla (or pytorch on tpu) questions, please open an issue in https://github.com/pytorch/xla 71.
Thanks!


Pytorch TPU printing 1 epoch performance 8 times

Question:
is the code attached below bug free?
def train_model():
    global train_dataset, valid_dataset

    torch.manual_seed(42)

    train_sampler = torch.utils.data.distributed.DistributedSampler(
        train_dataset,
        num_replicas=xm.xrt_world_size(),
        rank=xm.get_ordinal(),
        shuffle=True)

    train_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=BATCH_SIZE,
        sampler=train_sampler,
        num_workers=0,
        drop_last=True) # print(len(train_loader))

    '''valid_sampler = torch.utils.data.distributed.DistributedSampler(
        train_dataset,
        num_replicas=xm.xrt_world_size(),
        rank=xm.get_ordinal(),
        )'''

    valid_loader = torch.utils.data.DataLoader(
        valid_dataset,
        batch_size=BATCH_SIZE,
        #sampler=valid_sampler,
        shuffle=False,
        num_workers=0,
        drop_last=True)

    #xm.master_print(f"Train for {len(train_loader)} steps per epoch")
    LOGGER.debug(f"Train for {len(train_loader)} steps per epoch")
    # Scale learning rate to num cores
    lr  = 0.0001 * xm.xrt_world_size()

    # Get loss function, optimizer, and model
    device = xm.xla_device()

    #model = model()
    '''
    for param in model.base_model.parameters(): # freeze some layers
        param.requires_grad = False'''

    global model

    model = model.to(device)

    criterion = torch.nn.BCEWithLogitsLoss() #  MSELoss
    optimizer = optim.Adam(model.parameters(), lr=lr, weight_decay=5e-4)
    scheduler = OneCycleLR(optimizer,
                           lr,
                           div_factor=10.0,
                           final_div_factor=50.0,
                           epochs=NUM_EPOCH,
                           steps_per_epoch=len(train_loader))

    def train_loop_fn(loader):
        tracker = xm.RateTracker()
        model.train()

        #xm.master_print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        LOGGER.debug('Epoch {}/{}'.format(epoch, num_epochs - 1))
        #xm.master_print('-' * 10)
        LOGGER.debug('-' * 10)
        scheduler.step()

        running_loss = 0.0
        tk0 = tqdm(loader, total=int(len(train_loader)))
        counter = 0
        for bi, d in enumerate(tk0):
            inputs = d["image"]
            labels = d["label"].view(-1, 1)
            inputs = inputs.to(device, dtype=torch.float)
            labels = labels.to(device, dtype=torch.float)
            optimizer.zero_grad()
            #with torch.set_grad_enabled(True):
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            #loss = criterion(outputs, torch.max(labels, 1)[1])
            loss.backward()
            xm.optimizer_step(optimizer)
            running_loss += loss.item() * inputs.size(0)
            #print(running_loss)
            counter += 1
            tk0.set_postfix(loss=(running_loss / (counter * BATCH_SIZE)))
        epoch_loss = running_loss / len(train_loader)
        #xm.master_print('Training Loss: {:.8f}'.format(epoch_loss))
        LOGGER.debug('Training Loss: {:.8f}'.format(epoch_loss))

    def test_loop_fn(loader):
        tk0 = tqdm(loader, total=int(len(valid_loader)))
        counter = 0
        total_samples, correct = 0, 0
        for bi, d in enumerate(tk0):
            inputs = d["image"]
            labels = d["label"].view(-1, 1)
            inputs = inputs.to(device, dtype=torch.float)
            labels = labels.to(device, dtype=torch.float)
            optimizer.zero_grad()

            with torch.no_grad():

                output = model(inputs)

                pred = output.max(1, keepdim=True)[1]
                correct += pred.eq(labels.view_as(pred)).sum().item()
                total_samples += inputs.size()[0]
        accuracy = 100.0 * correct / total_samples
        #print('[xla:{}] Accuracy={:.4f}%'.format(xm.get_ordinal(), accuracy), flush=True)
        model.train()
        return accuracy

    # Train - valid  loop
    accuracy = []
    for epoch in range(1, num_epochs + 1):
        start = time.time()
        para_loader = pl.ParallelLoader(train_loader, [device])
        train_loop_fn(para_loader.per_device_loader(device))

        para_loader = pl.ParallelLoader(valid_loader, [device])
        accuracy.append(test_loop_fn(para_loader.per_device_loader(device)))
        #xm.master_print("Finished training epoch {}  Val-Acc {:.4f} in {:.4f} sec".format(epoch, accuracy[-1],   time.time() - start))

        LOGGER.debug("Finished training epoch {}  Val-Acc {:.4f} in {:.4f} sec".format(epoch, accuracy[-1],   time.time() - start))
        valauc = accuracy[-1]
        if(epoch>4):
            xm.save(model.state_dict(), f"./epoch{epoch}valauc{valauc}.bin")
    return accuracy

def _mp_fn(rank, flags):
    global acc_list
    torch.set_default_tensor_type('torch.FloatTensor')
    res = train_model()

FLAGS={}
xmp.spawn(_mp_fn, args=(FLAGS,), nprocs=8, start_method='fork')

1st epochs train log looks like this :
2020-05-09 12:21:29,371 DEBUG Train for 445 steps per epoch
2020-05-09 12:21:29,710 DEBUG Train for 445 steps per epoch
2020-05-09 12:21:29,721 DEBUG Train for 445 steps per epoch
2020-05-09 12:21:29,911 DEBUG Train for 445 steps per epoch
2020-05-09 12:21:30,561 DEBUG Epoch 1/6
2020-05-09 12:21:30,564 DEBUG ----------
2020-05-09 12:21:31,065 DEBUG Epoch 1/6
2020-05-09 12:21:31,076 DEBUG ----------
2020-05-09 12:21:31,120 DEBUG Epoch 1/6
2020-05-09 12:21:31,130 DEBUG ----------
2020-05-09 12:21:31,390 DEBUG Epoch 1/6
2020-05-09 12:21:31,426 DEBUG ----------
2020-05-09 12:21:32,629 DEBUG Train for 445 steps per epoch
2020-05-09 12:21:33,573 DEBUG Train for 445 steps per epoch
2020-05-09 12:21:33,748 DEBUG Train for 445 steps per epoch
2020-05-09 12:21:33,883 DEBUG Train for 445 steps per epoch
2020-05-09 12:21:34,889 DEBUG Epoch 1/6
2020-05-09 12:21:34,914 DEBUG ----------
2020-05-09 12:21:35,573 DEBUG Epoch 1/6
2020-05-09 12:21:35,613 DEBUG ----------
2020-05-09 12:21:35,823 DEBUG Epoch 1/6
2020-05-09 12:21:35,845 DEBUG ----------
2020-05-09 12:21:36,128 DEBUG Epoch 1/6
2020-05-09 12:21:36,171 DEBUG ----------
2020-05-09 12:35:08,162 DEBUG Training Loss: 11.22450873
2020-05-09 12:35:08,172 DEBUG Training Loss: 11.19612112
2020-05-09 12:35:08,309 DEBUG Training Loss: 11.18398799
2020-05-09 12:35:08,352 DEBUG Training Loss: 11.16665337
2020-05-09 12:35:08,362 DEBUG Training Loss: 11.20103131
2020-05-09 12:35:08,357 DEBUG Training Loss: 11.19919075
2020-05-09 12:35:08,368 DEBUG Training Loss: 11.19310062
2020-05-09 12:35:08,386 DEBUG Training Loss: 11.21970569
2020-05-09 12:39:31,562 DEBUG Finished training epoch 1  Val-Acc 50.5348 in 1080.4523 sec
the validation accuracy calculation is slow,somehow validation accuracy is using 1 core for calculation where in train phase it is using 8 cores,how do i solve this issue? i need to make the validation calculation fast, also in all epoch i see same validation accuracy, maybe i have bug in my code? another thing is,if i train this model for 8-10 epoch then kaggle kernel doesn’t finish commit,it gives error that’s not visible,so maybe somewhere in my code i am requesting more memory and getting OOM for that? also in my code if i try sampler=valid_sampler for valid_loader then i get error. please help me find bugs in my code,thank you a lot in advance

Context: FYI this conversation has been moved to https://github.com/pytorch/xla/issues/2054 118.


The computational complexity of torch.cat

Question:
how can I calculate the complexity O of torch.cat and torch.stack ?
and thank you

Context:
They allocate a target blob of the required size and then copy the values, so it is proportional to the total size. In degenerate cases (0-sized tensors) you would also have things that perform with the number of operands, for non-degenerate cases this is bounded by the total size.
Strictly speaking, the number of dimensions also plays a role, but in terms of asymptotic analysis you would soon discover that there is a hard upper bound for the number of dimensions you can use with PyTorch anyway.
The most common pitfall to avoid is to grow tensors by using growing_list = cat(growing_list, new_item) in a loop because that will give quadratic complexity (in the iteration size from having effort proportional to i in the i-th loop), which becomes noticeable quite fast.
Best regards
Thomas


Low gpu utilization whith very fast dataloading

Question:
Hello everyone, I can’t find the bottleneck of my training program.
As you can see the gpus utilization is low 43% (I am using bsz of 40 which consumes 12GB/16GB).
The bottleneck isn’t from the dataloading as you can see. It seems to come from the forward and backward…

image1354×406 70 KB

When I check the operators execution time on CPU, for some reason they consume CPU time. I am only using native pytorch tensors in my forward and loss computation.
https://imgur.com/a/VdVqTTY 1
I profiled the forward separately, and it show a CPU usage of 2s. But the individual operations don’t sum to 100% CPU total. I am confused.
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                                          model_forward         0.40%       8.407ms       100.00%        2.106s        2.106s       0.000us         0.00%      51.656ms      51.656ms             1
                                              aten::bmm         0.14%       3.033ms         0.30%       6.262ms     313.100us      23.576ms        45.64%      23.576ms       1.179ms            20
                                maxwell_sgemm_128x64_nn         0.00%       0.000us         0.00%       0.000us       0.000us      18.129ms        35.10%      18.129ms       2.014ms             9
                                             aten::add_         0.52%      10.907ms         0.55%      11.569ms     236.102us      11.983ms        23.20%      11.983ms     244.551us            49
void at::native::unrolled_elementwise_kernel<at::nat...         0.00%       0.000us         0.00%       0.000us       0.000us      11.282ms        21.84%      11.282ms     245.261us            46
                                          aten::softmax         0.00%      67.000us         0.34%       7.056ms     705.600us       0.000us         0.00%       8.552ms     855.200us            10
                                         aten::_softmax         0.08%       1.584ms         0.33%       6.989ms     698.900us       8.552ms        16.56%       8.552ms     855.200us            10
void (anonymous namespace)::softmax_warp_forward<flo...         0.00%       0.000us         0.00%       0.000us       0.000us       7.342ms        14.21%       7.342ms       1.049ms             7
                                maxwell_sgemm_128x64_tn         0.00%       0.000us         0.00%       0.000us       0.000us       5.623ms        10.89%       5.623ms     401.643us            14
                                           aten::linear         0.02%     344.000us         2.03%      42.657ms       1.066ms       0.000us         0.00%       2.733ms      68.325us            40
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 2.106s
Self CUDA time total: 51.656ms

Thank your for your help.

Context: Is this model using the C++ API rather than Python? It might be useful to share some of the training loop code to check if the profile is missing something.


Expected scalar type Float but found ComplexDouble

Question:
Hi,
I’m trying to run a NN with complex tensors.
I keep getting the following error: RuntimeError: expected scalar type Float but found ComplexDouble
I suspect maybe it is because the weights are not complex?
If so- how do I change it?
> class NeuralNet(nn.Module):
>     def __init__(self, input_nodes, hidden1_nodes, hidden2_nodes, hidden3_nodes, output_nodes):
>         super(NeuralNet, self).__init__()
>         self.type(torch.complex128)
>         self.fc1 = nn.Linear(input_nodes, hidden1_nodes)
>         self.relu = nn.ReLU()
>         self.fc2 = nn.Linear(hidden1_nodes, hidden2_nodes)
>         self.fc3 = nn.Linear(hidden2_nodes, hidden3_nodes)
>         self.fc4 = nn.Linear(hidden3_nodes, output_nodes)

Thanks!

Context:
Mam:

self.type(torch.complex128)

use .to these days instead of .type (which is a very odd function and the most surprising thing is that it hasn’t been deprecated 2 years ago).
.to (or .type) changes the state of the NeuralNet. This also changes all submodules. But if you add submodules later, these will not be party to the change but will have the default dtype (float).
this suggests that moving the .to to after the submodule registration should work better.

Best regards
Thomas


Question:
Hi,
So I just read the torch 1.10 release notes and came across the new constant time conjugate of complex tensors.
I would appreciate if you could help me find any information about this constant time algorithm.
(PyTorch`s tweet about this release also mentions constant time transpose. Knowing about that will be great as well)
Thank you.

Context:
Hi Anand! .conj() is now an O(1) operation because all it does is flips a bit on the tensor to indicate conjugation. However, the “materialization” of this lazy conjugation is still O(n).
Example:
x = torch.randn(100, dtype=torch.cfloat)
y = x.conj() # O(1)
z = x + y # materialization of conjugation [O(n)] + addition [O(n)]

However, in some operations like matrix multiplication, we can fuse conjugation with the operation because the underlying library that’s being called provides the option to indicate conjugation, and that’s where you’d see a significant improvement in performance (due to fewer reads and writes) in many cases.


Pytorch 1.10 complex number change is horrible, please change it back

Question:
pytorch 1.10 says

x = torch.tensor([1+2j])
y = x.conj()
z = y.imag
print(z.is_neg())
True

z.add_(2)
print(x)
tensor([1.-0.j])

This is anti-math! People who us complex number will not expect this behavior. Please change it back to the way in 1.9.

Context: cc @anjali411 who is working on this topic


Good advice to double the input and output channel from the pre-trained model?

Question:
Hi,
I have loaded the pre-trained ResNet-50 model, and now I want to change the input and output channels of all layers. Not only the first or last layers. But maintaining the same model structure.
is there any way to do this?
Here is a simple example of what I want to do.
class CustomCNN(nn.Module):
    def __init__(self):
        super(MiniCustomCNN, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 6, (5, 5)),
            nn.ReLU(),
            nn.MaxPool2d((2, 2)),
            nn.Conv2d(6, 8, (5, 5)),
            nn.ReLU(),
            nn.MaxPool2d((2, 2))
        )
        self.fc = nn.Sequential(
            nn.Linear(200, 40),
            nn.Linear(40, 10)
        )

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)
        return x

model = CustomCNN()
# To do something...

The model summary results I am aiming for…

CustomCNN(
(features): Sequential(
(0): Conv2d(6, 12, kernel_size=(5, 5), stride=(1, 1))
(1): ReLU()
(2): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
(3): Conv2d(12, 16, kernel_size=(5, 5), stride=(1, 1))
(4): ReLU()
(5): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
)
(fc): Sequential(
(0): Linear(in_features=400, out_features=80, bias=True)
(1): Linear(in_features=80, out_features=10, bias=True)
)
)

Simply, I want to double(or customize) the input and output channels from the pre-build model.
Any help will be appreciated.
Thanks.

Context:
Not a professional here but here is how I’d approach the situation.
Here is what I understood from your question, you have downloaded a pre-trained model, and now you want to change the internal channels of the layers, but keep the layers themselves, right?
Would you mind explaining further why you would need to do this? Maybe it would shed some light on how best to solve the issue.
I’m not sure if you can change the channels without hurting the weights because you’d have to reinitialize the layers to get the channels you want.
A solution might be to use the pre-trained model as an encoder, pass your data through the encoder, then create a custom network and pass it through. e.g
class CustomCNN(nn.Module):
    def __init__(self, encoder):
        super(CustomCNN, self).__init__()
        # pass the pretrained model here
        self.encoder = encoder
        # define your custom architecture
        self.features = nn.Sequential(
            nn.Conv2d(3, 6, (5, 5)),
            nn.ReLU(),
            nn.MaxPool2d((2, 2)),
            nn.Conv2d(6, 8, (5, 5)),
            nn.ReLU(),
            nn.MaxPool2d((2, 2))
        )
        self.fc = nn.Sequential(
            nn.Linear(200, 40),
            nn.Linear(40, 10)
        )

    def forward(self, x):
       # pass through encoder first
        x = self.encoder(x)
        x = self.features(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)
        return x

model = CustomCNN()

This way, your model works on whatever channels you have.

 dhkim:

and now I want to change the input and output channels of all layers. Not only the first or last layers

I’m not sure this is possible because you have to redefine
The tricky part would be how to match the output shape of the encoder to your first Conv2D.


Complex recurrent layers produce NaN as grad

Question:
Hi all. I am trying to run the most basic single-layer RNN with complex inputs on the nightly build of PyTorch (1.10.0-CPU). The problem is that the gradient always evaluates to NaN. I’ve tried all recurrent layers (RNN, GRU, LTSM) with the same result. Here is the model:
class CRNN(torch.nn.Module):
    def __init__(
        self,
        input_dim: int,
        output_dim: int,
        **kwargs
    ):
        super().__init__(**kwargs)
        self.model = torch.nn.RNN(input_dim, output_dim,
                                  batch_first=True,
                                  dtype=torch.cfloat)
        self.loss = torch.nn.L1Loss()
        self.optimizer = torch.optim.Adam(self.parameters())

    def forward(self, x):
        x = torch.unsqueeze(x, 0)
        return self.model(x)[0]

    def fit(self, x, y):
        self.optimizer.zero_grad()
        z = torch.squeeze(self(x), 0)
        loss = self.loss(z, y)
        loss.backward()
        self.optimizer.step()
        return loss.item()

I used torch.autograd.set_detect_anomaly(True) and it gave the following results:

For RNN, the first NaN appears in AddmmBackward

For GRU and LSTM, the first NaN appears in L1LossBackward

Since I have no idea how to interpret this, I am left to hope for help from the community.

Context: Are you able to reproduce this issue using random inputs? If so, could you post the random tensor initialization so that we could try to reproduce it, please?


Rewriting Tf to Pytorch DepthwiseConv2d functor opperation

Question:
Hello, In my research I have to rewrite the following function to Pytorch. The project involves looping over the layers in the network. Does anyone know how to rewrite the following:
depthwise = DepthWiseConv2d(in_channels = inp.shape[2], out_channels = out.shape[2], kernel_size = kernel, padding=padding, stride=stride, dilation = dilation_rate)
functor = K.function([inp, K.learning_phase()], [depthwise])
depthwise_conv2d_vals = functor([test_ex, 0])[0]

Context:
I don’t know what a functor is in TF/Keras, but would assume that you could just create the nn.Conv2d layer in PyTorch and pass the input directly to it.
If that’s not the case, could you explain what this code is doing?


Complex CNN on complex images gives RuntimeError

Question:
Hi,
I am trying to train a UNet model on complex images (MRI data), and, while I can convert the model and get the complex images (both cfloat, same happens with cdouble) just fine, I get the following error during the forward pass:
RuntimeError: getCudnnDataTypeFromScalarType() not supported for ComplexDouble

And the full stack trace:
Traceback (most recent call last):
  File "<MY FOLDER>/train.py", line 105, in <module>
    main()
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 344, in _run_hydra
    run_and_report(
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 199, in run_and_report
    raise ex
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 196, in run_and_report
    return func()
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 345, in <lambda>
    lambda: hydra.run(
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 107, in run
    return run_job(
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/hydra/core/utils.py", line 126, in run_job
    ret.return_value = task_function(task_cfg)
  File "<MY FOLDER>/train.py", line 98, in main
    start_single_process(0, n_gpu, working_dir, config)
  File "<MY FOLDER>/train.py", line 83, in start_single_process
    train_worker(config)
  File "<MY FOLDER>/train.py", line 46, in train_worker
    trainer.train()
  File "<MY FOLDER>/srcs/trainer/base.py", line 81, in train
    result = self._train_epoch(epoch)
  File "<MY FOLDER>/srcs/trainer/trainer.py", line 52, in _train_epoch
    output = self.model(data)
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "<MY FOLDER>/srcs/model/unet/unet.py", line 24, in forward
    x1 = self.inc(x)
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "<MY FOLDER>/srcs/model/unet/unet_parts.py", line 25, in forward
    return self.double_conv(x)
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "<PYTHON FOLDER>/miniconda3/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: getCudnnDataTypeFromScalarType() not supported for ComplexDouble

I wasn’t although able to pinpoint the cause. Has anyone come across the same issue before?

Context: Cudnn doesn’t support the ComplexDouble data type, so you would have to disable it via torch.backends.cudnn.enabled = False and rerun the code.


How to select the eigenvector of the largest eigenvalues?

Question:
I have a matrix of dimension (…, N, N). I use torch.linalg.eig to get the eigenvalues and eigenvectors. How to select the eigenvector which has the largest eigenvalue?
Sample code here:
m = torch.rand(2, 100, 8, 8, dtype=torch.cdouble)
w, v = torch.linalg.eig(m)
_, indices = torch.sort(w.abs(), dim=-1, descending=True)
v_max = f(indices) # the shape of v_max should be (2, 100, 8)

Context:
Hi,
You can use gather with expanded indices:
import torch
seconds_in_a_day = 24 * 60 * 60
seconds_im = torch.rand(2, 100, 8, 8, dtype=torch.cdouble)
w, v = torch.linalg.eig(seconds_im)
_, indices = torch.max(w.abs(), dim=-1, keepdim=True)
print(indices.size())
print(v.size())
v_max = v.gather(-1, indices.unsqueeze(-1).expand(2, 100, 8, 1)) # expand to 1, 8 if you want rows
print(v_max.size())
print(v[0][0])
print(v_max[0][0])


Zero-padding of `ifft` does not work as expected

Question:
Hi:)
I want to calculate the inverse FFT of a truncated input vector,
Python 3.8.10 (default, May 19 2021, 18:05:58)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> x = torch.rand(5)
>>> fx = torch.fft.fft(x)
>>> fx
tensor([3.2079+0.0000j, 0.3252+0.3854j, 0.1552+0.6282j, 0.1552-0.6282j, 0.3252-0.3854j])
>>> fx[2] = 0.
>>> fx[3] = 0.
>>> fx
tensor([3.2079+0.0000j, 0.3252+0.3854j, 0.0000+0.0000j, 0.0000+0.0000j, 0.3252-0.3854j])
>>> torch.fft.ifft(fx)
tensor([0.7717+0.j, 0.5352+0.j, 0.4457+0.j, 0.6269+0.j, 0.8284+0.j])

Note that the truncation affects the highest frequency modes and since both, fx[2] and fx[3], are set to zero the result of ifft is real-valued as expected.
Naively, I would expect that torch.fft.ifft gives the very same result when the frequency components are truncated rather than set to zero,
>>> torch.fft.ifft(torch.tensor([fx[0], fx[1], fx[4]]), n=5)
tensor([0.7717+0.0000j, 0.5811+0.1863j, 0.4905-0.1098j, 0.7277-0.0625j, 0.6370-0.0139j])

but the result is now different and complex.
Obviously, I am missing something, probably, I misunderstand the parameter n?
Any ideas?

Context:
Hi Avitase!

 avitase:

>>> fx
tensor([3.2079+0.0000j, 0.3252+0.3854j, 0.0000+0.0000j, 0.0000+0.0000j, 0.3252-0.3854j])
>>> torch.fft.ifft(fx)
tensor([0.7717+0.j, 0.5352+0.j, 0.4457+0.j, 0.6269+0.j, 0.8284+0.j])

>>> torch.fft.ifft(torch.tensor([fx[0], fx[1], fx[4]]), n=5)
tensor([0.7717+0.0000j, 0.5811+0.1863j, 0.4905-0.1098j, 0.7277-0.0625j, 0.6370-0.0139j])

but the result is now different and complex.

Pytorch’s fft functions, including ifft(), pad by simply appending zeros
to the end of the input vector – they don’t insert zeros into the middle.
You can get the behavior I believe you want by using rfft() / irfft().
The following illustrates these two points:
>>> import torch
>>> torch.__version__
'1.9.0'
>>> x = torch.arange (1, 6).float()**(1.0 / 3.0)
>>> x
tensor([1.0000, 1.2599, 1.4422, 1.5874, 1.7100])
>>> fx = torch.fft.fft (x)
>>> fx
tensor([ 6.9995+0.0000j, -0.5333+0.5133j, -0.4665+0.1265j, -0.4665-0.1265j,
        -0.5333-0.5133j])
>>> torch.fft.ifft (torch.tensor ([fx[0], fx[1], 0.0, 0.0, fx[4]]))
tensor([1.1866+0.j, 1.1387+0.j, 1.4518+0.j, 1.6932+0.j, 1.5293+0.j])
>>> torch.fft.ifft (torch.tensor ([fx[0], fx[1], fx[4], 0.0, 0.0]))
tensor([1.1866+0.0000j, 1.4159-0.0493j, 1.2952-0.0760j, 1.6112-0.1535j,
        1.4905+0.2789j])
>>> torch.fft.ifft (torch.tensor ([fx[0], fx[1], fx[4]]), n = 5)
tensor([1.1866+0.0000j, 1.4159-0.0493j, 1.2952-0.0760j, 1.6112-0.1535j,
        1.4905+0.2789j])
>>> rfx = torch.fft.rfft (x)
>>> rfx
tensor([ 6.9995+0.0000j, -0.5333+0.5133j, -0.4665+0.1265j])
>>> torch.fft.irfft (torch.tensor ([rfx[0], rfx[1]]), n = 5)
tensor([1.1866, 1.1387, 1.4518, 1.6932, 1.5293])
>>> torch.fft.irfft (torch.tensor ([rfx[0], rfx[1], 0.0]), n = 5)
tensor([1.1866, 1.1387, 1.4518, 1.6932, 1.5293])

Best.
K. Frank


Question: I am training a small model with 256 kb parameters. And I set the dataset loader with num_works = 16, pin_memory = True and batch_size = 128. However, the GPU-util is still very low with 0.007%. Could you give any advice to improve the GPU-util?

Context: Is that the batch size that is the maximum possible? Have you profiled or timed parts of the code to see what the bottleneck is?


Question:
After training some epochs, the error suddenly jump. However, I have know idea how to fix it after reading the error message below. Did anyone know what happened?
By the way, the error occurred after I add if __name__=='__main__'  to wrap the training python code ,specify the dataloader num_worker and pin_memory, and these:
torch.backends.cudnn.enabled = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = True
torch.autograd.set_detect_anomaly(False)
torch.autograd.profiler.profile(False)
torch.autograd.profiler.emit_nvtx(False)

,is their any alternatively way to code it? Thank you.
from resnetmodel2 import UnetLike_Residual_AutoEncoder
import torch
import numpy as np
from dataset import SurDataset
from utils import save_checkpoint, load_checkpoint, save_some_examples, seed_everything
from torch.utils.data import DataLoader
from torchvision.utils import save_image
import torch.nn as nn
import torch.optim as optim
import config
from tqdm import tqdm
if __name__ == '__main__':
    torch.backends.cudnn.enabled = True
    torch.backends.cudnn.benchmark = True
    torch.backends.cudnn.deterministic = True
    torch.autograd.set_detect_anomaly(False)
    torch.autograd.profiler.profile(False)
    torch.autograd.profiler.emit_nvtx(False)
    seed_everything(42)
    print(config.DEVICE)
    model = UnetLike_Residual_AutoEncoder(in_channels=1, out_channels=1).to(config.DEVICE)
    optimizer = optim.Adam(list(model.parameters()),
            lr = config.LEARNING_RATE,
            betas=(0.5, 0.999))
    mse = nn.MSELoss()
    scalar = torch.cuda.amp.GradScaler()
    dataset = SurDataset(csv_dir=config.CSV_DIR, img_dir=config.IMG_DIR)
    train_set, val_set = torch.utils.data.random_split(dataset, [12000, len(dataset)-12000], )
    loader = DataLoader(dataset=train_set, batch_size=config.BATCH_SIZE, shuffle=True, num_workers=config.NUM_WORKER, pin_memory=config.PIN_MEMORY)
    val_loader = DataLoader(dataset=val_set, batch_size=config.BATCH_SIZE, num_workers=config.NUM_WORKER, pin_memory=config.PIN_MEMORY)
    loop = tqdm(loader)

    if config.LOAD_MODEL:
        load_checkpoint(
            config.CHECKPOINT, model, optimizer, config.LEARNING_RATE,
        )
    model.train()
    for epoch in range(config.NUM_EPOCHS):
        print(f"epoch {epoch+1}/{config.NUM_EPOCHS}:")
        losses = []
        for idx, (csv_, target) in enumerate(loop):
            csv_ = csv_.to(config.DEVICE)
            target = target.to(config.DEVICE)
            predict = model(csv_)
            loss = mse(predict, target)
            losses.append(loss.item())
            optimizer.zero_grad()
            loss.backward()
            # gradient descent or adam step
            optimizer.step()
            if idx % 5 == 0:
                save_some_examples(model, val_loader, epoch, folder=config.EVALUATE_FOLDERNAME)
        if config.SAVE_MODEL:# and epoch % 2 == 0:
            save_checkpoint(model, optimizer, filename=config.CHECKPOINT)

        print(f"loss at epoch {epoch+1}/{config.NUM_EPOCHS} is {np.sum(losses):.4f}.")

=> Saving checkpoint
loss at epoch 1/30 is 55.9952.
epoch 2/30:
loss at epoch 2/30 is 54.4250.
epoch 3/30:
=> Saving checkpoint
loss at epoch 3/30 is 52.9719.
epoch 4/30:
Traceback (most recent call last):
  File "C:\Users\PML\.conda\envs\floren\lib\multiprocessing\queues.py", line 238, in _feed
    send_bytes(obj)
  File "C:\Users\PML\.conda\envs\floren\lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "C:\Users\PML\.conda\envs\floren\lib\multiprocessing\connection.py", line 280, in _send_bytes
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
BrokenPipeError: [WinError 232]

Context: @ptrblck  Could you help me, thank you a lot


Trying to create parallel RNN technical proof of concept

Question:
This is a technical proof of concept prior to experimenting with an MoE RNN.
Essentially, I want to run two RNNs in parallel on the same input sequence. The caveat is that the number of parallel RNNs to run is unknown - thus the for loop.
class pRNN(nn.Module):

    def __init__(self, input_size=0, hidden_size=0, num_layers=1, bidirectional=False, dropout=0, subunit_count=1):

        super().__init__()

        self.subunit_count = subunit_count

        subunit_size = math.ceil(hidden_size / subunit_count)

        self.rnn = []
        hidden_size_remaining = hidden_size
        for i in range(self.subunit_count):
            self.rnn.append(nn.GRU(input_size=input_size, hidden_size=min(hidden_size_remaining, subunit_size), num_layers=num_layers, bidirectional=bidirectional, dropout=dropout))

        self.rnn = nn.ModuleList(self.rnn)

    def forward(self, x):

        out = None
        for i in range(self.subunit_count):
            if out is None:
                out, hidden = self.rnn[i](x)
            else:
                out2 = self.rnn[i](x)
                out = torch.cat((out, out2[0]), dim=-1)
                hidden = torch.cat((hidden, out2[1]), dim=-1)

        return out, hidden

Unfortunately, this is maybe the fifth failed attempt to come up with a technique that runs in parallel - no matter what I try it runs sequentially (including reducing the already minimal batch size of 128 by the number of RNNs). I am running on a single V100.
Any suggestions are appreciated.

Context:
You could check, if your GPU has any resources left for a parallel execution of the RNNs.
RNNs are using matrix multiplications internally, which are executed via cublas on the GPU. Generally matmuls tend to saturate the device as much as possible, so that other kernels might have to wait for their execution. You could check the profile via e.g. Nsight Systems and also take a look at e.g. this post.


[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware. error message

Question:
Hey guys ,
I’am running into this error. I don’t really know if this is only one error messsage.
My OS:
Kali Linux
Pyhton version:
3.7
pytorch,pytorch-lightning, torchaudio, torchmetrics, torchvision version:
(I’ve installed pytorch, torchaudio and torchvision thru this command for cpu only
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 -c pytorch, you can find this installation command on pytorch.org 2)
pytorch==1.7.1
pytorch-lightning==1.2.8
torchaudio==0.7.2
torchmetrics==0.2.0
torchvision==0.8.2
My error message:
/home/anaconda3/envs/speech/lib/python3.7/site-packages/torch/functional.py:516: UserWarning: stft will require the return_complex parameter be explicitly  specified in a future PyTorch release. Use return_complex=False  to preserve the current behavior or return_complex=True to return  a complex output. (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370152014/work/aten/src/ATen/native/SpectralOps.cpp:653.)
  normalized, onesided, return_complex)
/home/anaconda3/envs/speech/lib/python3.7/site-packages/torch/functional.py:516: UserWarning: The function torch.rfft is deprecated and will be removed in a future PyTorch release. Use the new torch.fft module functions, instead, by importing torch.fft and calling torch.fft.fft or torch.fft.rfft. (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370152014/work/aten/src/ATen/native/SpectralOps.cpp:590.)
  normalized, onesided, return_complex)
[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware.

After this error message is being displayed it is sort of creating an endless loop which is printing out every second a new row. So my program is being completely crashed because of this error message.
Is someone able to help me out with that or know what this error message is caused by?
Thank’s for every suggestion and help in advance:)

Context: Double post with potential answer 723.


TypeError: trace(): argument ‘input’ (position 1) must be Tensor, not method

Question:
Hey guy’s,
when I try to run my script I get this weird error and I don’t know how to fix it or what this error is caused by. This is the error:TypeError: trace(): argument 'input' (position 1) must be Tensor, not method
I hope that some of you guy’s can help me with this error and explain to me what this error is caused by.
This is my code:
import argparse
import torch
from model import SpeechRecognition
from collections import OrderedDict

def trace(model):
    model.eval()
    x = torch.rand(1, 81, 300)
    hidden = model._init_hidden(1)
    traced = torch.trace(model.forward)(example_inputs=(x, hidden))
    return traced

def main(args):
    print("loading model from", args.model_checkpoint)
    checkpoint = torch.load(args.model_checkpoint, map_location=torch.device('cpu'))
    h_params = SpeechRecognition.hyper_parameters
    model = SpeechRecognition(**h_params)

    model_state_dict = checkpoint['state_dict']
    new_state_dict = OrderedDict()
    for k, v in model_state_dict.items():
        name = k.replace("model.", "") # remove `model.`
        new_state_dict[name] = v

    model.load_state_dict(new_state_dict)

    print("tracing model...")
    traced_model = trace(model)
    #print(type(traced_model))
    print("saving to", args.save_path)
    traced_model.save(args.save_path)
    print("Done!")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="testing the wakeword engine")
    parser.add_argument('--model_checkpoint', type=str, default='checkpoint_file', required=False,
                        help='Checkpoint of model to optimize')
    parser.add_argument('--save_path', type=str, default='path/where/the/optimized/model/should/be/saved', required=False,
                        help='path to save optmized model')

    args = parser.parse_args()
    main(args)

This is the full error message:
Traceback (most recent call last):
  File "c:/Users/supre/Documents/Python Programme/neuralnet/optimize_graph.py", line 45, in <module>
    main(args)
  File "c:/Users/supre/Documents/Python Programme/neuralnet/optimize_graph.py", line 30, in main
    traced_model = trace(model)
  File "c:/Users/supre/Documents/Python Programme/neuralnet/optimize_graph.py", line 11, in trace
    traced = torch.trace(model.forward)(example_inputs=(x, hidden))
TypeError: trace(): argument 'input' (position 1) must be Tensor, not method

Thank’s for every help in advance:)

Context:
It is because your inputs to the forward function are in the wrong parenthesis. You are just passing the forward method into the trace function not the output. To fix it just do this
     traced = torch.trace(model.forward(example_inputs=(x, hidden)))


Process stuck in backward if multi-rtx3090 were enabled

Question:
Hi Guys,
When I run my AI train in multi-GPUs mode, process always stuck (hang) in backward. I traced it into torch interface torch/autograd/init.py line 145 Variable._execution_engine.run_backward. It was blocked process and never return. No matter what I have set up DataParallel or DistributedDataParallel.  The console nothing output. I guess something dead-loop in torch/_C.cpython-38-x86_64-linux-gnu.so or further. Have anyone know about this issue or workaround solution?
I compare two mode train (the single GPU and both GPUs), my train is successfully passed when only one GPUs but failed in double. And I collect the arguments of both of them when pass it into Variable.execution_engine.run_backward, they are same.
rt_frames.T = {NameError}name ‘rt_frames’ is not defined
create_graph = {bool} False
grad_tensors = {NoneType} None
grad_tensors = {tuple: 1} tensor(1., device=‘cuda:0’)
grad_variables = {NoneType} None
inputs = {tuple: 0} ()
retain_graph = {bool} False
tensors = {tuple: 1} tensor(0.6899, device=‘cuda:0’, grad_fn=)
And I also test GPU + CUDA with p2pBandwithLatencyTest and the result is OK.
Would anyone do me a favor on having a look at this issue?
My hardware environment is CPU Intel® Core™ i9-7900X CPU @ 3.30GHz, Memory 8G*8 DDR3,
GPU rtx3090
software environment is Ubuntu18.04TLS, RTX3090 Driver460.56 + cuda_11.1.TC455_06.29069683_0 + Tensorflow 2.4.1.+ Pytorch1.8.1(cu111).

Context: You could check dmesg for Xid errors and see, if a GPU might be dropped due to some issues such as a PSU, which might be too weak to drive two 3090s.


Some tensors getting left on CPU despite calling model.to(“cuda”)

Question:
Hello,
Would like to politely request another set of eyes on some problems we’re running into trying to train our model.
Background: We’re using complex64 dtype tensors generated from torchaudio stft().  We are using a git repo, complexPyTorch 3, which provides complex versions of layers etc.  We are hoping to train the model on GPU, and we have successfully trained other models on this same gpu/cuda/pytorch setup.
Here’s some relevant code from our training file:
device = torch.device("cuda")
net = model(params)
net = net.to(device)

criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr=learning_rate)

# Training

start_time = round(time())

for epoch in range(10):  # loop over the dataset multiple times

    running_loss = 0.0

    for i, data in enumerate(dataLoader, 0):

        # get the inputs; data is a list of [inputs, labels]

        inputs = data['song']

        labels = data['vocal']

        inputs = inputs.to(device)

        labels = labels.to(device)

        optimizer.zero_grad()

        outputs = net(inputs)

        loss = criterion(outputs, labels)

        loss.backward()

        optimizer.step()

        running_loss += loss.item()

print('Finished Training', str(round(time()) - start_time))

PATH = './PhATPUSSE.pth'

torch.save(net.state_dict(), PATH)

So, we attempt to move the model and all data to the GPU.
When we try to train we get this error:
RuntimeError: Tensor for argument #3 'mat2' is on CPU, but expected it to be on GPU (while checking arguments for addmm)
We followed the trace to figure out it is the weight tensors of any of our nn.Linear layers that do not make it onto the GPU.  Many other tensors do successfully make it to the GPU, such as our data and the tensors associated with Conv2d layers.  Any tips are appreciated, thanks so much!  If you would like to see any of the architecture code etc. lmk

Context: Could you post the model definition and in particular how the problematic nn.Linear layer is initialized?


Question:
I am trying to implement super resolution with a complex neural network using 91-image and set-5 datasets. I keep getting the following error at the loss functions:
cnn error1004×401 19.8 KB
This is my code
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms
from complexLayers import ComplexBatchNorm2d, ComplexConv2d, ComplexLinear
from complexLayers import ComplexDropout2d, NaiveComplexBatchNorm2d
from complexFunctions import complex_relu, complex_max_pool2d

from datasets import TrainDataset, EvalDataset

from torch.utils.data.dataloader import DataLoader

batch_size = 64

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])
train_set = TrainDataset('D:\complexPyTorch-master\SRCNN-pytorch-master\91-image_x2.h5')
train_loader = DataLoader(dataset=train_set,
                                  batch_size=64,
                                  shuffle=True,
                                  num_workers=8,
                                  pin_memory=True,
                                  drop_last=True)
test_set = EvalDataset('D:\complexPyTorch-master\SRCNN-pytorch-master\Set5_x2.h5')
test_loader = DataLoader(dataset=test_set, batch_size=1)

class ComplexNet(nn.Module):

    def __init__(self):
        super(ComplexNet, self).__init__()
        self.conv1 = ComplexConv2d(1, 64, kernel_size=9, padding=9 // 2)
        self.conv2 = ComplexConv2d(64, 32, kernel_size=5, padding=5 // 2)
        self.conv3 = ComplexConv2d(32, 1, kernel_size=5, padding=5 // 2)
        #self.relu = complex_relu()

    def forward(self,x):
        x = complex_relu(self.conv1(x))
        x = complex_relu(self.conv2(x))
        x = self.conv3(x)
        x = complex_relu(x) #this should have bene in init but it is placed here instead
        #x = self.fc2(x)
        x = x.abs()
        #x =  F.log_softmax(x)
        return x

device = device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = ComplexNet().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=5e-3, momentum=0.9)

def train(model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target =data.to(device).type(torch.complex64), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        print(output.dtype)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 100 == 0:
            print('Train Epoch: {:3} [{:6}/{:6} ({:3.0f}%)]\tLoss: {:.6f}'.format(
                epoch,
                batch_idx * len(data),
                len(train_loader.dataset),
                100. * batch_idx / len(train_loader),
                loss.item())
            )

for epoch in range(4):
    train(model, device, train_loader, optimizer, epoch)

Context:
F.nll_loss expects the targets as LongTensors containing the class indices, while you are trying to pass a FloatTensor to it as the error message explains.
If your target already contains the necessary class indices, call target = target.long() before passing it to the criterion.


Question:
Hello,
I am currently implementing complex GRU-cells as described in the paper “Complex Gated Recurrent Neural Networks”.
Working on the currently nightly (1.8.0dev20210201) and the list of supported complex gradients 2, I expected the code below to run through.
Instead, I receive the following error:
  File "...\model\spectral_rnn\spectral_rnn.py", line 75, in train
    loss.backward()
  File "...\Python38\lib\site-packages\torch\tensor.py", line 227, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "...\Python38\lib\site-packages\torch\autograd\__init__.py", line 145, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Expected isFloatingType(grad.scalar_type()) || (input_is_complex == grad_is_complex) to be true, but got false.

I also reported it as a bug as issue in the repository.
I am happy for any hint, e.g. which operation the problem could be!
Thanks,
Kinetize
def to_complex_activation(activation):
    return lambda x: torch.view_as_complex(torch.cat(
        [activation(x.real).unsqueeze(-1), activation(x.imag).unsqueeze(-1)], dim=-1))

class CGCell(nn.Module):

    def __init__(self, input_size, hidden_size):
        super(CGCell, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size

        self.wg = nn.Parameter(torch.randn(2 * hidden_size, hidden_size, dtype=torch.cfloat))
        self.vg = nn.Parameter(torch.randn(2 * hidden_size, input_size, dtype=torch.cfloat))
        self.bg = nn.Parameter(torch.randn(2 * hidden_size, dtype=torch.cfloat))

        self.w = nn.Parameter(torch.randn(hidden_size, hidden_size, dtype=torch.cfloat))
        self.v = nn.Parameter(torch.randn(hidden_size, input_size, dtype=torch.cfloat))
        self.b = nn.Parameter(torch.randn(hidden_size, dtype=torch.cfloat))

        alpha = beta = 0.5  # TODO
        self.fg = lambda x: torch.sigmoid(alpha * x.real + beta * x.imag)
        self.fa = to_complex_activation(torch.sigmoid)

    def _init_hidden(self, x):
        h = torch.zeros((x.shape[0], self.hidden_size),  dtype=torch.cfloat).to(device)

        return h

    def forward(self, x, ht_=None):
        if ht_ is None:
            ht_ = self._init_hidden(x)

        gates = ht_ @ self.wg.T + x @ self.vg.T + self.bg
        g_r, g_z = gates.chunk(2, 1)

        g_r = self.fg(g_r)
        g_z = self.fg(g_z)

        z = (g_r * ht_) @ self.w.T + x @ self.v.T + self.b
        ht = g_z * self.fa(z) + (1 - g_z) * ht_

        return ht

Context: Bug resolved, see this issue 24.


AttributeError: ‘builtin_function_or_method’ object has no attribute ‘fftn’

Question:
Hi, I am using torch version ‘1.7.1+cu110’ but fftn appears to be missing at torch.fft.fftn.
According to torch.fft — PyTorch 1.7.0 documentation 26 this should exist.
Any advice? Thanks

Context:
I guess you might be missing the import torch.fft statement and thus torch.fft.fftn will raise an error.
>>> import torch
tor>>> torch.__version__
'1.7.1'
>>> torch.fft.fftn
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'builtin_function_or_method' object has no attribute 'fftn'

>>> import torch.fft
>>> torch.fft.fftn
<built-in function fft_fftn>


How might autograd differentiate complex functions that are not complex-differentiable?

Question:
Hello Forum!
There are useful functions of a complex variable, z, that are not
“complex-differentiable,” that is, they are not analytic functions.
Two examples would be z.conj() and |z|**2 = z * z.conj().
Note that these functions are differentiable when viewed as real
functions of two real variables, x and y, where z = x + yj.
I think that when a function is analytic, autograd should return
its conventional complex derivative.  However, we would like
autograd also to do something “useful” for non-analytic functions
that are differentiable when understood as real functions.
For example, it would be nice if gradient descent would work for
minimizing |z - z0|**2 = (z - z0) * (z - z0).conj() with
respect to z (which takes on its minimum value of 0 when z = z0).
I don’t have a well-thought-out proposal for how to do this.  But I
came across this exposition that discusses some of the core issues:
The Complex Gradient Operator and the CR-Calculus 1
One facet of this issue is illustrated in this post where naive gradient
descent is shown to fail for |z| (for complex z) using autograd in
version 1.6.0:

    How to implement a custom loss function which include frobenius norm?

    Hello Yuchen!

Complex tensors are still a work in progress in pytorch, with more
functionality (and more functionality that is actually correct) being
added in successive versions.
Note that as of version 1.6.0, I don’t believe that the pytorch optimizers
accept complex Parameters, so to use pytorch’s complex machinery,
you will have to either use real Parameters that you combine into
complex tensors or write your own complex-aware optimizer.
All in all, depending on what you are doing,…

Best.
K. Frank

Context:
Hi Frank,
We actually had a very long discussion about this exact question and what to do.
It is also quite fun to see that jax and Tensorflow currently define two different gradients for these non-analytical functions!
You can see the whole discussion here: Follow the JAX or Tensorflow convention for meaning of grad() with complex inputs · Issue #41857 · pytorch/pytorch · GitHub 6
Also you can find in our doc a fairly extensive summary of the result of that discussion: Autograd mechanics — PyTorch master documentation 4
Happy to discuss it more if you’re interested!


LSTM shape problem for time series feature extraction

Question:
Hi,
I am trying to implement a feature extractor LSTM network.
The main architecture of my network was:
FeatureExtractorNetworkLSTM(
  (fenet): ModuleList(
    (0): LSTM(18, 256)
    (1): Dropout(p=0.3, inplace=False)
    (2): LSTM(256, 256)
    (3): Dropout(p=0.3, inplace=False)
    (4): LSTM(256, 256)
    (5): Dropout(p=0.3, inplace=False)
    (6): LSTM(256, 256)
    (7): Dropout(p=0.3, inplace=False)
    (8): LSTM(256, 256)
    (9): Dropout(p=0.3, inplace=False)
    (10): LSTM(256, 256)
  )
)

And then changed the architecture to:
FeatureExtractorNetworkLSTM(
  (fenet): ModuleList(
    (0): LSTM(18, 18)
    (1): Dropout(p=0.3, inplace=False)
    (2): LSTM(256, 256)
    (3): Dropout(p=0.3, inplace=False)
    (4): LSTM(256, 256)
    (5): Dropout(p=0.3, inplace=False)
    (6): LSTM(256, 256)
    (7): Dropout(p=0.3, inplace=False)
    (8): LSTM(256, 256)
    (9): Dropout(p=0.3, inplace=False)
    (10): LSTM(256, 256)
  )
)

but again this error is shown:

RuntimeError : input.size(-1) must be equal to input_size. Expected 18, got 1*

I would appreciate if you can help me how should I implement the network.
Note that using batch is not necessity.
The code of network:
class FeatureExtractorNetworkLSTM(nn.Module):
    def __init__(self,
                input_size=256, hidden_size=256, output_size=64,
                alpha=0.001,
                n_layers=3, dropout_prob=0,
                optimizer='Adam', chkpt_dir='runs/ppo'):
        super(FeatureExtractorNetworkLSTM, self).__init__()

        self.checkpoint_file = os.path.join(chkpt_dir, 'fe_LSTM_ppo')
        self.fenet = nn.ModuleList()
        for n in range(n_layers):
            if n == 0:
                self.fenet.append(nn.LSTM(input_size, input_size, 1))

            #input_size = input_size if n == 0 else hidden_size
            elif n == 1:
                self.fenet.append(nn.LSTM(input_size, hidden_size, 1))

                if dropout_prob > 0:
                    self.fenet.append(nn.Dropout(dropout_prob))

            else:
                self.fenet.append(nn.LSTM(hidden_size, hidden_size, 1))

                if dropout_prob > 0:
                    self.fenet.append(nn.Dropout(dropout_prob))

        self.fenet.append(nn.LSTM(hidden_size, output_size, 1))

        #self.fenet.append(nn.Linear(2*hidden_size, 1))

        if optimizer == 'Adam':
            self.optimizer = torch.optim.Adam(self.parameters(), lr=alpha)
        if optimizer == 'Adagrad':
            self.optimizer = torch.optim.Adagrad(self.parameters(), lr=alpha)

        if optimizer == 'RMSprop':
            self.optimizer = torch.optim.RMSprop(self.parameters(), lr=alpha)

        self.device = torch.device(
            'cuda:0' if torch.cuda.is_available() else 'cpu')
        self.to(self.device)

    def forward(self, state):
        #dist = self.actor(state)
        for i, l in enumerate(self.fenet):
            f_state = l(state)
        #dist = torch.distributions.categorical.Categorical(dist)

        return f_state

Sample of input tensor:
tensor([[ 1.0658e+00,  1.0675e+00,  1.0637e+00,  1.0652e+00,  1.4717e+04,
          6.2318e+01,  6.0329e+01,  5.8142e+01,  1.0639e+00,  1.0592e+00,
          1.0549e+00,  6.2538e+01,  2.6001e-03,  2.6311e-04,  2.3370e-03,
          1.0681e+00,  1.0558e+00,  1.0619e+00],
        [ 1.0652e+00,  1.0667e+00,  1.0647e+00,  1.0665e+00,  1.1889e+04,
          6.5088e+01,  6.2078e+01,  5.9319e+01,  1.0643e+00,  1.0595e+00,
          1.0550e+00,  6.3402e+01,  2.6151e-03,  2.2250e-04,  2.3926e-03,
          1.0684e+00,  1.0564e+00,  1.0624e+00],
        [ 1.0665e+00,  1.0668e+00,  1.0645e+00,  1.0648e+00,  7.0910e+03,
          5.8982e+01,  5.8555e+01,  5.7163e+01,  1.0644e+00,  1.0597e+00,
          1.0551e+00,  5.8190e+01,  2.4615e-03,  5.5073e-05,  2.4064e-03,
          1.0685e+00,  1.0569e+00,  1.0627e+00],
        [ 1.0648e+00,  1.0651e+00,  1.0630e+00,  1.0641e+00,  7.8530e+03,
          5.6691e+01,  5.7200e+01,  5.6324e+01,  1.0643e+00,  1.0599e+00,
          1.0552e+00,  5.7425e+01,  2.2588e-03, -1.1809e-04,  2.3769e-03,
          1.0685e+00,  1.0576e+00,  1.0630e+00],
        [ 1.0641e+00,  1.0642e+00,  1.0621e+00,  1.0626e+00,  6.8300e+03,
          5.1931e+01,  5.4325e+01,  5.4523e+01,  1.0641e+00,  1.0600e+00,
          1.0553e+00,  5.3252e+01,  1.9554e-03, -3.3719e-04,  2.2926e-03,
          1.0684e+00,  1.0580e+00,  1.0632e+00],
        [ 1.0626e+00,  1.0628e+00,  1.0610e+00,  1.0623e+00,  6.1390e+03,
          5.0850e+01,  5.3664e+01,  5.4107e+01,  1.0638e+00,  1.0601e+00,
          1.0553e+00,  4.8368e+01,  1.6675e-03, -5.0007e-04,  2.1676e-03,
          1.0681e+00,  1.0587e+00,  1.0634e+00]], device='cuda:0')

and the current input tensor shape is:
torch.Size([6, 18])

Complete traceback:
RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_4592/1425274335.py in <module>
      7     for t in range(len(env.data)):  # while not env.done
      8         # Select and perform an action
----> 9         action, probs, value = agent.choose_action(state)
     10         reward, done, _ = env.step(action)
     11

c:\analytics_ai\analytics_ai\trading_bots\fe_ppo_agent.py in choose_action(self, state)
    146             self.actor.device)
    147
--> 148         fe_states = self.fe_net(state)
    149
    150         dist = self.actor(fe_states)

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

c:\analytics_ai\analytics_ai\trading_bots\models.py in forward(self, state)
    418         #dist = self.actor(state)
    419         for i, l in enumerate(self.fenet):
--> 420             f_state = l(state)
    421         #dist = torch.distributions.categorical.Categorical(dist)
    422

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
    687             hx = self.permute_hidden(hx, sorted_indices)
    688
--> 689         self.check_forward_args(input, hx, batch_sizes)
    690         if batch_sizes is None:
    691             result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\nn\modules\rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    630                            batch_sizes: Optional[Tensor],
    631                            ):
--> 632         self.check_input(input, batch_sizes)
    633         self.check_hidden_size(hidden[0], self.get_expected_hidden_size(input, batch_sizes),
    634                                'Expected hidden[0] size {}, got {}')

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\nn\modules\rnn.py in check_input(self, input, batch_sizes)
    203                     expected_input_dim, input.dim()))
    204         if self.input_size != input.size(-1):
--> 205             raise RuntimeError(
    206                 'input.size(-1) must be equal to input_size. Expected {}, got {}'.format(
    207                     self.input_size, input.size(-1)))

RuntimeError: input.size(-1) must be equal to input_size. Expected 18, got 1

I also checked it with torch.unsqueeze(0) which converts the shape to:
torch.Size([1, 6, 18])

but did not worked.
Thanks

Context: @ptrblck Can you please help me?


Question:
I am currently working on a multi-label binary classification problem.
The output of my model is a tensor like this:
tensor([[3.0914e-08, 3.2459e-17]])
and the ground truth label looks like this:
tensor([[1, 1]])
I iterate over a custom validation DataLoader (after training for one epoch) and for every input and label I execute:
prediction = self._model(x)
loss = self._crit(output, y.float())
return loss, prediction

Where self._crit is BCELoss. Now I am trying to calculate the f1-score using sklearn.metrics via:
f1_score(label[0], (prediction[0] > 0.5).float() * 1)
I don’t know whether this is the correct approach but the f1_score always ends up being 0.
Instead of calculating this for every validation sample, should I maybe do this only for the whole validation set?
If so - how?

Context: You could store the predictions in a new tensor (or append them to a list and create the tensor after the validation loop is done) and calculate the F1 score afterwards on the entire validation set.


Question:
I was training GCN model on my Linux server and I suddenly got this error.

RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW

Pytorch version: 1.10.1+cu102
OS: Linux
Python version: Python 3.8.10
CUDA Version: 11.2

Context:
Based on this issue 6 other users were running into the same error message if

their setup was broken due to a driver/library mismatch (rebooting seemed to solve the issue)
their installed drivers didn’t match the user-mode driver inside a docker container (and forward compatibility failed due to the usage of non-server GPUs)

Was your setup working before and if so, what changed?


Question:
Hello, have a nice day!!

When to try implement the code;
def has_mask(mask_path):
img = cv2.imread(mask_path, 0)
thresh = cv2.threshold(img, 200, 255, cv2.THRESH_BINARY)[1]
if np.mean(thresh) > 0.0:
return True
else:
return False
def get_dataloader(rank, world_size):
“”“Creates the data loaders.”""
train_df = dataset.create_df(config.train_dir)
valid_df = dataset.generate_df(config.valid_dir)
# determine if an image contains mask or not
building_label_paths = train_df['building_label_path'].values.tolist()
train_has_masks = list(map(dataset.has_mask, building_label_paths))
train_df['has_mask'] = train_has_masks

def getitem(self, index):
example = {}
df_row = self.dataset.iloc[index]
image_1 = cv2.imread(df_row[‘image_1_path’], 0) / 255.0
image_2 = cv2.imread(df_row[‘image_2_path’], 0) / 255.0
rgb_image = s1_to_rgb(image_1, image_2)
if self.split == 'test':
    # no building mask should be available
    example['image'] = rgb_image.transpose((2,0,1)).astype('float32')
else:
    # load ground truth building mask
    building_mask = cv2.imread(df_row['building_label_path'], 0) / 255.0

    # compute transformations or apply augmentations if specified
    if self.transform:
        augmented = self.transform(image=rgb_image, mask=building_mask)
        rgb_image = augmented['image']
        building_mask = augmented['mask']

    example['image'] = rgb_image.transpose((2,0,1)).astype('float32')
    example['mask'] = building_mask.astype('int64')

return example

After completing one epoch, I got these errors
Traceback (most recent call last):
File “train.py”, line 240, in
mp.spawn(train, args=(config.num_epochs, WORLD_SIZE), nprocs=WORLD_SIZE, join=True)
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\multiprocessing\spawn.py”, line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method=‘spawn’)
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\multiprocessing\spawn.py”, line 157, in start_processes
while not context.join():
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\multiprocessing\spawn.py”, line 118, in join
raise Exception(msg)
Exception:
– Process 0 terminated with the following error:
Traceback (most recent call last):
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\multiprocessing\spawn.py”, line 19, in _wrap
fn(i, *args)
File “F:\Projects\FD\1\train.py”, line 212, in train
for batch in tqdm(val_loader, leave=False):
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\tqdm\std.py”, line 1180, in iter
for obj in iterable:
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\utils\data\dataloader.py”, line 435, in next
data = self._next_data()
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\utils\data\dataloader.py”, line 1085, in _next_data
return self._process_data(data)
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\utils\data\dataloader.py”, line 1111, in _process_data
data.reraise()
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch_utils.py”, line 428, in reraise
raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\utils\data_utils\worker.py”, line 198, in _worker_loop
data = fetcher.fetch(index)
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\utils\data_utils\fetch.py”, line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File “C:\ProgramData\Anaconda3\envs\torch171\lib\site-packages\torch\utils\data_utils\fetch.py”, line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File “F:\Projects\FD\1\etciDataset.py”, line 39, in getitem
building_mask = cv2.imread(df_row[‘building_label_path’], 0) / 255.0
TypeError: Can’t convert object of type ‘numpy.float64’ to ‘str’ for ‘filename’

Thanks for the help!

Context:
An example would be to pass a valid image path as a string:
img = cv2.imread('/tmp/image.jpeg')

Right now df_row['building_label_path'] is an np.float64 value, which is a floating point value and not a path string so you would check your df_row content and make sure to index the right column/row.


Question:
on torch.utils.data.sampler.SubsetRandomSampler(),  if I want to choose 5 indices out of 100 randomly,  is it right?
torch.utils.data.sampler.SubsetRandomSampler(idx, 5)
where  len(idx) = 100
in tutorial of Pytorch, I can’t find using it at this way, why?

Context:
No, your usage is wrong and won’t chose 5 out of 100 indices.
As described in the docs 1:

Samples elements randomly from a given list of indices, without replacement.

which means you are supposed to pass the indices the sampler should randomly sample from as the first argument.
E.g. if your original index is defined as idx = torch.arange(100), you could select the desired 5 indices and pass it to the sampler: SubsetRandomSampler(idx[:5]).
If you want to randomly pick these 5 indices, you could use:
idx_subset = idx[torch.randperm(len(idx))[:5]]


Question:
Hello,
I’m wondering if there is a pretty straightforward way (possibly this is trivial) to implement a neural network in PyTorch with a variable number of input features.
Many thanks,
Federico

Context:
Hi Frederico!

 Federico_Ottomano:

Thanks, what about a standard fully connected neural network?

Focusing on your word “standard,” no, a fully-connected network will
not be able to accept a variable number of input features.
Let’s say you have an input batch of shape [nBatch, nFeatures]
and the first network layer is Linear (in_features, out_features).
If nFeatures != in_features pytorch will complain about a dimension
mismatch when your network tries to apply the weight matrix of your
first Linear to the input batch.
If you want to do something like this you will have to pad your input
feature vectors to all have the same length or similarly use “dummy”
values for “missing” features in the shorter feature vectors.
Bear in mind that feature[7] in one sample has to have the same
meaning as feature[7] in another sample – otherwise your network
won’t be able to “learn” what feature[7] means.  Therefore any
scheme you use to “pad” feature vectors to the same length has to
make sure that features with the same meaning end up in the same
location in the input vectors.
Best.
K. Frank


Question:
Hi, all.
I have some questions about the torch.nn.functional.batch_norm.
I train the model, extract the model’s values with state_dict(), and then proceed with inference using the torch function based on it.
However, the value of the model implemented as a function by myself is different from the value in the original model.
I think there is a problem in the process of directly implementing the function.

13924×412 39.6 KB

When using torch.nn.functional.batch_norm, I wonder if it is necessaru to use a value other than running from state_dict() or if there is another way.
Thank you!

Context:
I don’t think you can directly compare F.conv2d with BinarizeConv2d as it’s not a plain nn.Conv2d layer, but “binarizes” the data and parameters by manipulating their .data attribute as seen here 1:
class BinarizeConv2d(nn.Conv2d):

    def __init__(self, *kargs, **kwargs):
        super(BinarizeConv2d, self).__init__(*kargs, **kwargs)

    def forward(self, input):
        if input.size(1) != 3:
            input.data = Binarize(input.data)
        if not hasattr(self.weight,'org'):
            self.weight.org=self.weight.data.clone()
        self.weight.data=Binarize(self.weight.org)

        out = nn.functional.conv2d(input, self.weight, None, self.stride,
                                   self.padding, self.dilation, self.groups)

        if not self.bias is None:
            self.bias.org=self.bias.data.clone()
            out += self.bias.view(1, -1, 1, 1).expand_as(out)

        return out


Question:
I am working on WGAN-GP with a data set having 292 images for training. I am facing few problems:
1- Generator’s graph seems totally flat even after 25 epochs.
2- Just after few epochs this error is coming “DataLoader worker (pid(s) 6740) exited unexpectedly”
Error:
E:\Users\Asus\anaconda3\lib\site-packages\torch\utils\data\dataloader.py in _try_get_data(self, timeout)
   1001             if len(failed_workers) > 0:
   1002                 pids_str = ', '.join(str(w.pid) for w in failed_workers)
-> 1003                 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
   1004             if isinstance(e, queue.Empty):
   1005                 return (False, None)

RuntimeError: DataLoader worker (pid(s) 6740) exited unexpectedly

3- Continuously flat  noise images are generating. Image is attached for further clarity.
fake_images-00101552×777 44.9 KB

Training Loop:
if epoch_flag == True:
    previous_epochs = previous_epochs
elif epoch_flag == False:
    previous_epochs = 0

cur_step = 0
generator_losses = []
critic_losses = []
for epoch in range(n_epochs):
    # Dataloader returns the batches
    for real_images, _ in tqdm(data_loader):
        cur_batch_size = len(real_images)
        real_images = real_images.to(device)

        mean_iteration_critic_loss = 0
        for _ in range(crit_repeats):
            ### Update critic ###
            crit_opt.zero_grad()
            fake_noise = get_noise(cur_batch_size, z_dim, device=device)
            fake = gen(fake_noise)
            crit_fake_pred = crit(fake.detach())
            crit_real_pred = crit(real_images)

            epsilon = torch.rand(len(real_images), 1, 1, 1, device=device, requires_grad=True)
            gradient = get_gradient(crit, real_images, fake.detach(), epsilon)
            gp = gradient_penalty(gradient)
            crit_loss = get_crit_loss(crit_fake_pred, crit_real_pred, gp, c_lambda)

            # Keep track of the average critic loss in this batch
            mean_iteration_critic_loss += crit_loss.item() / crit_repeats
            # Update gradients
            crit_loss.backward(retain_graph=True)
            # Update optimizer
            crit_opt.step()
        critic_losses += [mean_iteration_critic_loss]

        ### Update generator ###
        gen_opt.zero_grad()
        fake_noise_2 = get_noise(cur_batch_size, z_dim, device=device)
        fake_2 = gen(fake_noise_2)
        crit_fake_pred = crit(fake_2)

        gen_loss = get_gen_loss(crit_fake_pred)
        gen_loss.backward()

        # Update the weights
        gen_opt.step()

        # Keep track of the average generator loss
        generator_losses += [gen_loss.item()]

        ### Visualization code ###
        if cur_step % display_step == 0 and cur_step >= 0:
            gen_mean = sum(generator_losses[-display_step:]) / display_step
            crit_mean = sum(critic_losses[-display_step:]) / display_step
            print(f"Epoch {epoch}, step {cur_step}: Generator loss: {gen_mean}, critic loss: {crit_mean}")
            show_tensor_images(fake)
            show_tensor_images(real_images)
            step_bins = 20
            num_examples = (len(generator_losses) // step_bins) * step_bins
            plt.plot(
                range(num_examples // step_bins),
                torch.Tensor(generator_losses[:num_examples]).view(-1, step_bins).mean(1),
                label="Generator Loss"
            )
            plt.plot(
                range(num_examples // step_bins),
                torch.Tensor(critic_losses[:num_examples]).view(-1, step_bins).mean(1),
                label="Critic Loss"
            )
            plt.legend()
            plt.show()

        cur_step += 1
    all_epochs = epoch + 1 + previous_epochs
    if epoch_flag == True:
        save_fake_images(all_epochs)
    elif epoch_flag == False:
        save_fake_images(epoch + 1)

Generator’s Code:
class Generator(nn.Module):
    def __init__(self, z_dim=32, im_chan=1, hidden_dim=32):
        super(Generator, self).__init__()
        self.z_dim = z_dim
        # Build the neural network
        self.gen = nn.Sequential(
            #PrintBlock(), # [50, 32, 1, 1]
            self.make_gen_block(z_dim, hidden_dim * 2),
            #PrintBlock(), # [50, 64, 3, 3]
            self.make_gen_block(hidden_dim * 2, hidden_dim * 4, kernel_size=4, stride=1),
            #PrintBlock(), # [50, 128, 6, 6]
            self.make_gen_block(hidden_dim * 4, hidden_dim * 8),
            #PrintBlock(), # [50, 256, 13, 13]
            self.make_gen_block(hidden_dim * 8, hidden_dim * 16, kernel_size=4, stride=1),
            #PrintBlock(), # [50, 512, 16, 16]
            self.make_gen_block(hidden_dim * 16, hidden_dim * 16),
            #PrintBlock(), # [50, 512, 33, 33]
            self.make_gen_block(hidden_dim * 16 , hidden_dim * 8 , kernel_size=4, stride=1),
            #PrintBlock(), # [50, 256, 36, 36]
            self.make_gen_block(hidden_dim * 8 , hidden_dim * 4),
            #PrintBlock(), # [50, 128, 73, 73]
            self.make_gen_block(hidden_dim * 4 , hidden_dim * 2, kernel_size=4, stride=1),
            #PrintBlock(), # [50, 64, 76, 76]
            self.make_gen_block(hidden_dim * 2 , im_chan,  final_layer=True),
            #PrintBlock(), # [50, 1, 153, 153]

        )

    def make_gen_block(self, input_channels, output_channels, kernel_size=3, stride=2, final_layer=False):

        if not final_layer:
            return nn.Sequential(
                nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),
                nn.BatchNorm2d(output_channels),
                nn.ReLU(inplace=True),
            )
        else:
            return nn.Sequential(
                nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),
                nn.Tanh(),
            )

    def forward(self, noise):

        x = noise.view(len(noise), self.z_dim, 1, 1)
        return self.gen(x)

Discriminator’s Code:
class Critic(nn.Module):
    def __init__(self, im_chan=1, hidden_dim=64):
        super(Critic, self).__init__()
        self.crit = nn.Sequential(
            #PrintBlock(),
            self.make_crit_block(im_chan, hidden_dim),
            #PrintBlock(),
            self.make_crit_block(hidden_dim, hidden_dim * 2, kernel_size=4, stride=1),
            #PrintBlock(),
            self.make_crit_block(hidden_dim * 2, hidden_dim * 4),
            #PrintBlock(),
            self.make_crit_block(hidden_dim * 4, hidden_dim * 8,kernel_size=4, stride=1),
            #PrintBlock(),
            self.make_crit_block(hidden_dim * 8, hidden_dim * 8),
            #PrintBlock(),
            self.make_crit_block(hidden_dim * 8, hidden_dim * 4,kernel_size=4, stride=1),
            #PrintBlock(),
            self.make_crit_block(hidden_dim * 4, hidden_dim * 2),
            #PrintBlock(),
            self.make_crit_block(hidden_dim * 2, hidden_dim,kernel_size=4, stride=1),
            #PrintBlock(),
            self.make_crit_block(hidden_dim, 1, final_layer=True),
            #PrintBlock(),

        )

    def make_crit_block(self, input_channels, output_channels, kernel_size=3, stride=2, final_layer=False):

        if not final_layer:
            return nn.Sequential(
                nn.Conv2d(input_channels, output_channels, kernel_size, stride),
                nn.BatchNorm2d(output_channels),
                nn.LeakyReLU(0.2, inplace=True),
            )
        else:
            return nn.Sequential(
                nn.Conv2d(input_channels, output_channels, kernel_size, stride),
            )

    def forward(self, image):
        crit_pred = self.crit(image)
        return crit_pred.view(len(crit_pred), -1)

Please help me in figuring out these issues.

Context:
Finally i figured out these issues and i would like to thanks this community which enabled me to resolve my issues by myself.
No 2 issue has been resolved when i set num_workers=0.
No3 and No1 issues were interrelated and they get solved when i corrected my generator’s and discriminator’s network.
Hopefully my findings will help the community here.


Question:
Since batchnorm layer gathers statistics during the training step and reuse them later during inference, we have to define a new batchnorm layer every time it is used. The question is if this also applies to maxpooling or is it enough to define it once and use multiple times
if we take the follwing script as example is it enough to only use max_pooling1 layer for both usages
thnks
self.conv1 = nn.Conv2d(out_channel_4, out_channel_5, kernel_size=(9, 9), bias=True, stride=1, padding='same')
self.batch_Norm1 = nn.BatchNorm2d(out_channel_3, momentum=momentum)
self.conv2 = nn.Conv2d(out_channel_3, out_channel_4, kernel_size=(9, 9), bias=True, stride=1, padding='same')
self.batch_Norm2 = nn.BatchNorm2d(out_channel_4, momentum=momentum)
self.max_pooling1 = nn.MaxPool2d(2, stride=2)
self.max_pooling2 = nn.MaxPool2d(2, stride=2)

    def forward(self, x1):
        # block 1.1
        x1 = self.conv1(x1)
        x1 = self.batch_Norm1(x1)
        x1 = F.relu(x1)
        x1 = self.max_pooling1(x1)
        x1 = self.conv2(x1)
        x1 = self.batch_Norm2(x1)
        x1 = F.relu(x1)
        x1 = self.max_pooling2(x1)

Context:
Correct mean and variance of the whole dataset needed to do proper normalization, but batchnorm has access only to batch part of data each step. That’s why batchnorm is saving intermediate values and updates them each batch-step.
Maxpool only returns max value in the pool in input data. Thus, you don’t need to define different maxpooling instances if they have same parameters.


Why run torch.unique twice get different running time?

Question:
Code is:
time1 = time.time()
unique, inv =  torch.unique(src, ,sorted=False, return_inverse=True)
time2 = time.time()
src2 = src.clone()
time2 = time.time()
unique, inv =  torch.unique(src2, sorted=True, return_inverse=True)
time3 = time.time()
print(time2- time1, time3-time2)
get the time:  0.015712261199951172 0.000576972961425781
It’s very strange that time-consuming seems to have nothing to do with whether it is sorted or not. The time-consuming to run first is always higher than  later.

Context: In case you are using the GPU, you would have to synchronize the code before starting and stopping the timers since CUDA operations are executed asynchronously. The better way to profile such workloads would be to use torch.utils.benchmark.


Question:
torchtext.vocab build_vocab_from_iterator
hello, help please, gives an error
vocab = torchtext.vocab.build_vocab_from_iterator(iterate_corpus(all_sentenses), specials=["", “”],special_first=True)
TypeError: build_vocab_from_iterator() got an unexpected keyword argument ‘specials’
torch version 1.7.0
torchtext version 0.8.0
python 3.8
tokenizer = torchtext.data.get_tokenizer("spacy", language="ru_core_news_sm")

def iterate_corpus(corpus: List[str]):
        for sentense in corpus:
         yield tokenizer(sentense)

    corpus = construct_dataset("paraphrases.xml")
    all_sentenses = corpus["text_1"].tolist() + corpus["text_2"].tolist()
    vocab = torchtext.vocab.build_vocab_from_iterator(iterate_corpus(all_sentenses),
    specials=["<unk>", "<pad>"],
    special_first=True
    )

Context: You would have to update torchtext, as this the specials argument was added in torchtext==0.10.0 13.


How to access different layers of a sequential model?

Question:
Hi all…I have a deep sequential CNN model consisting of Conv, relu, batchNorm, maxPool stacked one after another. I was thinking if I could access the layers(like indexing) through a loop or by some other mechanism, so that I can perform some different operation to different layers.
Thank you

Context:
Hi @Kai123
To get an item of the Sequential use square brackets. You can even slice Sequential
import torch.nn as nn

my_model = nn.Sequential(nn.Identity(), nn.Identity(), nn.Identity())
print(my_model[0:2])


Loss and Gradients Fluctuating very rapidly

Question: I am using a 3 layered neural net with a single input and single output I am using  loss.backward() and then updating the weights but my loss fluctuates very rapidly and it never converges, However I am pretty sure that my loss function has a global minima.

Context:
It is very difficult to help you based on the abstract description that you provided in your post. This is just because there are literally millions of code scenarios that could show the behaviour that you describe, and since your description is so vague, we have no clue which of these scenarios apply to you.
You will have a far better chance of getting help with your problem if you provide the following (more concrete) information:

Complete,  executable code which exhibits the behaviour that you want to address

Ideally, this should include just enough code to demonstrate the points that you want us to address, while being executable. The fewer the lines of code involved, the easier (usually …) it is for others to read, understand, and debug it.
If possible, try to come up with code that takes in random tensors of the proper shapes as input, instead of actual data (such as images or text from the real world).

A description of what you expected from this code, and what you actually saw.


Implementation of the multiclass case for precision metric

Question:
Hi, I wanted to try the example of the multiclass case for precision metric from the ignite library with Average=True. I used the exact same values and code but still got an error. I was interested to test out different cases and binary & multilabel seem to work, but the multiclass case is not working as it should work like in the docs.
import torch
from ignite.engine import Engine
from ignite.metrics import Precision
def process_function(engine, data):
  return data[0][0], data[0][1]
default_evaluator = Engine(process_function)

metric = Precision(average=True)
metric.attach(default_evaluator, "precision")
y_true = torch.Tensor([2, 0, 2, 1, 0, 1]).long()
y_pred = torch.Tensor([
    [0.0266, 0.1719, 0.3055],
    [0.6886, 0.3978, 0.8176],
    [0.9230, 0.0197, 0.8395],
    [0.1785, 0.2670, 0.6084],
    [0.8448, 0.7177, 0.7288],
    [0.7748, 0.9542, 0.8573],
])
state = default_evaluator.run([[y_pred, y_true]])
print(state.metrics["precision"])

This is the error I get:

image1386×601 35.7 KB

In the docs  link below, you can find the same code block for the multiclass case, and they get the precision as 0.6111…

      PyTorch-Ignite

Precision — PyTorch-Ignite v0.4.7 Documentation 2

  High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

I would be very grateful if someone could point out where I was going wrong, or if I need to do something else to test the multiclass case for precision.
Thank you!

Context:
Thanks for pointing out potential issues with our documentation and doctests!
In your example, the definition of default_evaluator should be different. For our doctests it is defined here:
https://github.com/pytorch/ignite/blob/5c5837ac2528e9e9f1451dfbceacfc0deb86fe56/docs/source/conf.py#L349-L352
def eval_step(engine, batch):
    return batch

default_evaluator = Engine(eval_step)

Complete example is the following:
import torch
from ignite.metrics import Precision
from ignite.engine import Engine

def eval_step(engine, batch):
    return batch

default_evaluator = Engine(eval_step)

metric = Precision(average=True)
metric.attach(default_evaluator, "precision")
y_true = torch.Tensor([2, 0, 2, 1, 0, 1]).long()
y_pred = torch.Tensor([
[0.0266, 0.1719, 0.3055],
[0.6886, 0.3978, 0.8176],
[0.9230, 0.0197, 0.8395],
[0.1785, 0.2670, 0.6084],
[0.8448, 0.7177, 0.7288],
[0.7748, 0.9542, 0.8573],
])
state = default_evaluator.run([[y_pred, y_true]])
print(state.metrics["precision"])
> 0.6111111

Hope this helps