Question:
Hi. I am trying to build a standalone evaluation script using Pytorch Ignite. Yet, I am having trouble getting the desired output.
From the following code:
evaluator = create_evaluator(cfg, model, tokenizer, selected_metrics, logger)

    @evaluator.on(Events.ITERATION_COMPLETED(every=1) | Events.COMPLETED)
    def log_info():
        metrics_output = "\n".join(
            [f"\t{k}: {v}" for k, v in evaluator.state.metrics.items()]
        )

        logger.info(
            f"\nEvaluation time (seconds): {evaluator.state.times['COMPLETED']:.2f}\nValidation metrics:\n {metrics_output}"
        )

    try:
        state = evaluator.run(val_dataloader,)
    except Exception as e:
        logger.exception("")
        raise e

I get the following truncated output:

2022-01-06 16:32:30,537 Test INFO:
Evaluation time (seconds): 0.00
Validation metrics:
…
2022-01-06 16:32:32,628 Test INFO:
Evaluation time (seconds): 0.00
Validation metrics:
2022-01-06 16:32:32,628 Test INFO:
Evaluation time (seconds): 30.94
Validation metrics:
meteor: 0.2809806329259838
ter: 491.0379854211612

As you can see, only when the engine’s run is complete the output is correctly printed. Until then, every thing is zero. How can I properly get the evaluator’s state? Thank you.

Context:
@AfonsoSalgadoSousa there are two questions here: 1) why time is not printed and 2) how to print intermediate metric value.
Q1: during the evaluation we typically run a single epoch and evaluator.state.times does not contain per-iteration times, but only epoch and total run. It would be better to use ignite.handler.Timer for iterations.
Q2: to print intermediate metric value we have to compute it per-iteration <=> call metric.compute or metric.completed, see the example below.
Here is an example of how to achieve what you would like :
import time
import torch
from ignite.engine import Engine, Events
from ignite.metrics import Accuracy
from ignite.handlers import Timer

def eval_step(engine, batch):
    return batch

evaluator = Engine(eval_step)

acc = Accuracy()
acc.attach(evaluator, "accuracy")

timer = Timer()

# Compute intermediate metrics
@evaluator.on(Events.ITERATION_COMPLETED)
def compute_and_measure():
    acc.completed(evaluator, "accuracy")
    # added sleep to emulate an amount of time taken by processing a single iteration
    time.sleep(0.1)

@evaluator.on(Events.ITERATION_COMPLETED(every=1) | Events.COMPLETED)
def log_info():
    metrics_output = "\n".join(
        [f"\t{k}: {v}" for k, v in evaluator.state.metrics.items()]
    )
    print(
        f"\nEvaluation time (seconds): {timer.value():.8f}\nValidation metrics:\n {metrics_output}"
    )

bs = 32
n_classes = 10

data = [
    [torch.rand(bs, n_classes), torch.randint(0, n_classes, size=[bs, ])]
    for _ in range(8)
]

state = evaluator.run(data)
print(state.times)

Output:

Evaluation time (seconds): 0.10191662
Validation metrics:
 	accuracy: 0.03125

Evaluation time (seconds): 0.20312049
Validation metrics:
 	accuracy: 0.0625

Evaluation time (seconds): 0.30411697
Validation metrics:
 	accuracy: 0.125

Evaluation time (seconds): 0.40528948
Validation metrics:
 	accuracy: 0.125

Evaluation time (seconds): 0.50620632
Validation metrics:
 	accuracy: 0.125

Evaluation time (seconds): 0.60733969
Validation metrics:
 	accuracy: 0.125

Evaluation time (seconds): 0.70837395
Validation metrics:
 	accuracy: 0.12053571428571429

Evaluation time (seconds): 0.80952449
Validation metrics:
 	accuracy: 0.109375

Evaluation time (seconds): 0.81010838
Validation metrics:
 	accuracy: 0.109375

{'EPOCH_COMPLETED': 0.8085811138153076, 'COMPLETED': 0.8087835311889648}

HTH


Question:
Hi, I am new to PyTorch. I am trying to train a simple ANN model for regression. The issue is my loss doesn’t seem to be changing much over ~10 epochs. The exact same model/loss/optimizer, on the same data shows very different loss in another implementation. The two implementations primarily differ in data prep. The data is in table format - each row has a target col and a bunch of feature cols. I would greatly appreciate any help (been stuck for several days). Below is my code:
First I read in the data in a pandas dataframes df and df_test. Next, I split df into train_data+val_data using stratified split. The rest of the code is below:

image480×741 123 KB

The val_loss changes between 3.84-3.85 over 10 epochs. The other implementation starts with a val_loss < 2 and by 6 epochs is < 1 (same batch size as mine). Sorry for the lengthy post. I would appreciate any help (may be an obvious mistake I am overlooking?). Thanks.

Context:
I found the bug in the code above. I needed to unsqueeze the input target tensors by the adding the line: target = target.unsqueeze(1) This changed the shape of the target tensor from torch.Size([num_of_rows]) to torch.Size([num_of_rows], 1) . The model trains fine now:

Thanks to those who looked to help.


Question:
Hi!
I am interested in possibly using Ignite to enable distributed training in CPU’s (since I am training a shallow network and have no GPU"s available). I tried using ignite.distributed with the gloo backend, but when I set nproc_per_node to more than 1, the program gets stuck and doesn’t run (it does without setting nproc_per_node). The code is practically the same as the CIFAR example 1.
My question is, is it possible to train PyTorch models on multiple CPU’s be it with or without Ignite?
Thanks in advance!

Context:
Here is how to launch the code on Jupyter.
import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""

import time
import ignite.distributed as idist

def training(local_rank, config, **kwargs):
    time.sleep(local_rank)
    print(idist.get_rank(), ': run with config:', config, '- backend=', idist.backend())
    # do the training ...

backend = 'gloo'
dist_configs = {'nproc_per_node': 4, "start_method": "fork"}
config = {'c': 12345}

with idist.Parallel(backend=backend, **dist_configs) as parallel:
    parallel.run(training, config, a=1, b=2)

You have to use start_method="fork".
If you would like to run it as a script file and spawn processes from your main.py script as you do, then you can use default start_method. Also, it could be helpful to set persistent_workers=True for the DataLoader to speed up data fetching every epoch…
If you would like to use a script file and spawn processes with torch.distributed.launch, you can simply reuse the command from my previous message (and no need to set persistent_workers=True).


Issues with Ignite Training Loop but fine with plain Pytorch

Question:
Hello, I have tabular data and am passing it to a bi-directional LSTM network defined as such:
class BiLSTM(nn.Module):
    def __init__(self, hidden_size=48, num_layers=2, inp_size=5, dropout=0.2):
        super(BiLSTM, self).__init__()

        self.hidden_size = hidden_size
        self.input_size = inp_size
        self.num_layers = num_layers
        self.lstm1 = nn.LSTM(input_size=self.input_size, hidden_size=self.hidden_size,
                             batch_first=True, bidirectional=True)
        self.lstm2 = nn.LSTM(input_size=self.hidden_size*2, hidden_size=self.hidden_size,
                             batch_first=True, bidirectional=True)
        self.dropout = nn.Dropout(dropout)
        self.dense1 = nn.Linear(96,50)
        self.flatten = nn.Flatten()
        self.dense2 = nn.Linear(50, 1)

    def forward(self, x, hidden=None):
        _ = hidden
        x, hidden = self.lstm1(x, hidden)
        x = self.dropout(x)
        x, _ = self.lstm2(x, _)
        x = self.dropout(x)
        x = self.dense1(x)
        x = self.flatten(x)
        return self.dense2(x), hidden

==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
├─LSTM: 1-1                              [100, 1, 96]              21,120
├─Dropout: 1-2                           [100, 1, 96]              --
├─LSTM: 1-3                              [100, 1, 96]              56,064
├─Dropout: 1-4                           [100, 1, 96]              --
├─Linear: 1-5                            [100, 1, 50]              4,850
├─Flatten: 1-6                           [100, 50]                 --
├─Linear: 1-7                            [100, 1]                  51
==========================================================================================
Total params: 82,085
Trainable params: 82,085
Non-trainable params: 0
Total mult-adds (M): 0.08
==========================================================================================
Input size (MB): 0.00
Forward/backward pass size (MB): 0.19
Params size (MB): 0.33
Estimated Total Size (MB): 0.52
==========================================================================================

I wanted to implement EarlyStopping from the Ignite library but there didn’t seem to be a more simple way than adopting the Ignite workflow. So, I followed the quick start and Early Stopping docs to implement this:
split_num = len(val_loaders)-2
train_loader = zip(train_loaders[split_num], train_label_loaders[split_num])
val_loader = zip(val_loaders[split_num], val_label_loaders[split_num])

net = BiLSTM()
loss_func = nn.MSELoss(reduction='mean')
optimizer = torch.optim.Adam(net.parameters(), lr=lr)

trainer = create_supervised_trainer(net, optimizer, loss_func)

val_metrics = {
    "accuracy": Accuracy(),
    "loss": Loss(loss_func)
}
evaluator = create_supervised_evaluator(net, metrics=val_metrics)

@trainer.on(Events.ITERATION_COMPLETED(every=1))
def log_training_loss(trainer):
    print(f"Epoch[{trainer.state.epoch}] Loss: {trainer.state.output:.2f}")

@trainer.on(Events.EPOCH_COMPLETED)
def log_training_results(trainer):
    evaluator.run(train_loader)
    metrics = evaluator.state.metrics
    print(f"Training Results - Epoch: {trainer.state.epoch}  Avg accuracy: {metrics['accuracy']:.2f} Avg loss: {metrics['loss']:.2f}")

@trainer.on(Events.EPOCH_COMPLETED)
def log_validation_results(trainer):
    evaluator.run(val_loader)
    metrics = evaluator.state.metrics
    print(f"Validation Results - Epoch: {trainer.state.epoch}  Avg accuracy: {metrics['accuracy']:.2f} Avg loss: {metrics['loss']:.2f}")

def score_function(engine):
    val_loss = engine.state.metrics['loss']
    return -val_loss

handler = EarlyStopping(patience=50, score_function=score_function, trainer=trainer)
# Note: the handler is attached to an *Evaluator* (runs one epoch on validation dataset).
evaluator.add_event_handler(Events.COMPLETED, handler)

model_path = PATH + str(split_num) + ".pth"
handler2 = ModelCheckpoint(model_path, filename_prefix='', score_function=score_function, n_saved=2, create_dir=True)
trainer.add_event_handler(Events.EPOCH_COMPLETED(every=2), handler2, {'mymodel': net})

trainer.run(train_loader, max_epochs=n_epochs)

I end up getting an error about input size but I have no such error when using the same objects in my pytorch training loops. Can anyone help with this?
(full error traceback)
Current run is terminating due to exception: input.size(-1) must be equal to input_size. Expected 5, got 1
Engine run is terminating due to exception: input.size(-1) must be equal to input_size. Expected 5, got 1

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-72-e8150550b1bd> in <module>
     49 trainer.add_event_handler(Events.EPOCH_COMPLETED(every=2), handler2, {'mymodel': net})
     50
---> 51 trainer.run(train_loader, max_epochs=n_epochs)

~/miniconda3/envs/py/lib/python3.8/site-packages/ignite/engine/engine.py in run(self, data, max_epochs, epoch_length, seed)
    700
    701         self.state.dataloader = data
--> 702         return self._internal_run()
    703
    704     @staticmethod

~/miniconda3/envs/py/lib/python3.8/site-packages/ignite/engine/engine.py in _internal_run(self)
    773             self._dataloader_iter = None
    774             self.logger.error(f"Engine run is terminating due to exception: {e}")
--> 775             self._handle_exception(e)
    776
    777         self._dataloader_iter = None

~/miniconda3/envs/py/lib/python3.8/site-packages/ignite/engine/engine.py in _handle_exception(self, e)
    467             self._fire_event(Events.EXCEPTION_RAISED, e)
    468         else:
--> 469             raise e
    470
    471     @property

~/miniconda3/envs/py/lib/python3.8/site-packages/ignite/engine/engine.py in _internal_run(self)
    743                     self._setup_engine()
    744
--> 745                 time_taken = self._run_once_on_dataset()
    746                 # time is available for handlers but must be update after fire
    747                 self.state.times[Events.EPOCH_COMPLETED.name] = time_taken

~/miniconda3/envs/py/lib/python3.8/site-packages/ignite/engine/engine.py in _run_once_on_dataset(self)
    848         except Exception as e:
    849             self.logger.error(f"Current run is terminating due to exception: {e}")
--> 850             self._handle_exception(e)
    851
    852         return time.time() - start_time

~/miniconda3/envs/py/lib/python3.8/site-packages/ignite/engine/engine.py in _handle_exception(self, e)
    467             self._fire_event(Events.EXCEPTION_RAISED, e)
    468         else:
--> 469             raise e
    470
    471     @property

~/miniconda3/envs/py/lib/python3.8/site-packages/ignite/engine/engine.py in _run_once_on_dataset(self)
    831                 self.state.iteration += 1
    832                 self._fire_event(Events.ITERATION_STARTED)
--> 833                 self.state.output = self._process_function(self, self.state.batch)
    834                 self._fire_event(Events.ITERATION_COMPLETED)
    835

~/miniconda3/envs/py/lib/python3.8/site-packages/ignite/engine/__init__.py in _update(engine, batch)
    100         optimizer.zero_grad()
    101         x, y = prepare_batch(batch, device=device, non_blocking=non_blocking)
--> 102         y_pred = model(x)
    103         loss = loss_fn(y_pred, y)
    104         loss.backward()

~/miniconda3/envs/py/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-31-b16c34d32b2c> in forward(self, x, hidden)
     32     def forward(self, x, hidden=None):
     33         _ = hidden
---> 34         x, hidden = self.lstm1(x, hidden)
     35         x = self.dropout(x)
     36         x, _ = self.lstm2(x, _)

~/miniconda3/envs/py/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/miniconda3/envs/py/lib/python3.8/site-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    577             hx = self.permute_hidden(hx, sorted_indices)
    578
--> 579         self.check_forward_args(input, hx, batch_sizes)
    580         if batch_sizes is None:
    581             result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,

~/miniconda3/envs/py/lib/python3.8/site-packages/torch/nn/modules/rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    528
    529     def check_forward_args(self, input: Tensor, hidden: Tuple[Tensor, Tensor], batch_sizes: Optional[Tensor]):
--> 530         self.check_input(input, batch_sizes)
    531         expected_hidden_size = self.get_expected_hidden_size(input, batch_sizes)
    532

~/miniconda3/envs/py/lib/python3.8/site-packages/torch/nn/modules/rnn.py in check_input(self, input, batch_sizes)
    176                     expected_input_dim, input.dim()))
    177         if self.input_size != input.size(-1):
--> 178             raise RuntimeError(
    179                 'input.size(-1) must be equal to input_size. Expected {}, got {}'.format(
    180                     self.input_size, input.size(-1)))

RuntimeError: input.size(-1) must be equal to input_size. Expected 5, got 1

Context:
Hi @cmdkev , thanks for using Ignite !
I think in your case, the training step is a bit different from the basic one used in create_supervised_trainer, I suggest to create the trainer and evaluator in the following and more flexible way:
from ignite.engine import Engine

model = BiLSTM()
loss_func = nn.MSELoss(reduction='mean')
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
device = "cuda"

def training_step(engine, batch):
    model.train()
    optimizer.zero_grad()

    x, y = batch[0].to(device), batch[1].to(device)
    y_pred, hidden = model(x, trainer.state.hidden)
    loss = loss_func(y_pred, y)

    loss.backward()
    optimizer.step()

    trainer.state.hidden = hidden

    return loss.item()

trainer = Engine(training_step)
trainer.state.hidden = None

@torch.no_grad()
def evaluation_step(engine, batch):
    model.eval()

    x, y = batch[0].to(device), batch[1].to(device)
    y_pred, _ = model(x)
    return y_pred, y

evaluator = Engine(evaluation_step)

Remaining code can be the same.
Hope this helps and, please, let me know if it works or not.


Question:
Hello. I found some weird situation on ignite’s evaluator.
I have a pytorch dataloader and well-trained model.
when I try to extract output from model manually like below, it works well…
logfile = open('logs/manual.log', 'w')

dataloader_iter = iter(dataloader)

for i in range(500):
    x, t = next(dataloader_iter)
    y = model(x)

    logfile.write('{0}th iteration...\n'.format(i))
    logfile.write('    x: {0}\n'.format(x))
    logfile.write('    O: {0}\n'.format(y))
    logfile.write('    o: {0}\n'.format(torch.argmax(y, dim=1)))
    logfile.write('    t: {0}\n\n'.format(t))

logfile.close()

Then the logfile shows(focus on direct output from model(O):
1th iteration...
    <I skiped other things...>
O: tensor([[-2.8684e+00, -2.4779e+00,  1.4409e+00,  4.2450e+00,  1.3716e+00,
          3.1618e+00,  2.5039e+00, -1.3585e-03, -3.0291e+00, -3.1306e+00],
        [ 3.1552e+00,  1.8194e+00, -3.0561e-01, -1.8409e+00, -2.6845e-01,
         -3.2020e+00, -3.0243e+00, -7.1415e-01,  2.4644e+00,  1.6819e+00],
        [ 8.4511e-01, -4.4308e-01,  1.1573e+00,  1.2453e-01,  5.5392e-01,
          6.7758e-03, -1.0871e+00,  2.9498e-01, -7.7099e-01, -2.3612e-01],
        [ 4.2768e+00,  1.0600e+00,  2.5328e-01, -8.9427e-01, -1.2167e+00,
         -2.5544e+00, -4.6379e+00, -9.9491e-01,  4.3370e+00,  3.3613e-01],
        [-1.5431e+00, -2.1083e+00,  2.6824e+00,  1.7168e+00,  2.7112e+00,
          9.7829e-01,  2.7005e+00,  6.7293e-01, -3.5220e+00, -3.2395e+00],
        [-1.7089e+00, -6.2691e-02, -6.5357e-01,  1.6353e+00,  3.9782e-01,
          1.3109e+00,  3.1719e-01,  2.2915e-01, -2.3173e+00,  1.5951e+00],
        [ 3.6005e-01,  6.5493e+00, -1.8465e+00,  1.6237e-01, -2.9798e+00,
         -1.6791e+00, -3.1212e+00, -6.6102e-01, -1.4621e+00,  4.4547e+00],
        [-4.2739e-02, -3.4736e+00,  2.4689e+00,  4.2563e-01,  2.9417e+00,
         -7.3102e-01,  2.5900e+00, -1.5313e-01, -6.9163e-01, -1.9232e+00],
        [ 5.4870e-01, -3.8533e+00,  1.4458e+00,  2.0452e+00,  1.9034e+00,
          2.4421e+00, -1.2948e+00,  6.9678e-01,  3.8147e-01, -3.9030e+00],
        [ 1.9710e+00,  2.5695e+00,  6.1807e-01, -3.4376e-01,  7.6892e-02,
         -2.4107e+00, -8.0755e-01, -6.9068e-01, -6.7928e-01,  3.1438e-01],
        [-2.5117e-01, -4.4562e+00,  2.7807e+00,  2.3242e+00,  3.1733e+00,
          2.7852e+00,  5.5874e-01, -1.3494e-01,  1.5951e-01, -5.1482e+00],
        [-7.8764e-01,  1.3296e+00, -1.2178e+00,  5.5745e-01, -1.5662e+00,
         -5.0908e-01, -1.1492e+00,  4.3546e-01, -1.2073e+00,  3.8453e+00],
        [-8.3729e-01, -2.7771e+00,  1.8575e+00,  2.3420e+00,  1.8659e+00,
          2.1749e+00,  1.6091e+00, -4.0015e-01, -1.2970e+00, -3.6298e+00],
        [-1.2611e+00, -2.7069e-02,  9.4248e-02,  8.1586e-01,  1.2401e+00,
          1.4915e+00, -1.5474e+00,  2.9358e+00, -2.9514e+00, -7.6922e-01],
        [ 2.9973e+00,  1.5974e+00,  3.1030e-01, -5.9278e-01, -1.3065e+00,
         -3.6284e+00, -1.7095e+00, -2.5131e+00,  2.5173e+00,  2.9853e+00],
        [ 2.8719e+00,  2.5325e-01,  1.6667e+00, -3.4530e-01,  1.8475e-01,
         -2.7671e+00,  1.9819e+00, -3.5765e+00,  2.2073e+00, -1.5772e+00],
        [-1.3318e+00, -1.1345e+00,  9.9565e-01,  3.0902e+00, -7.6009e-02,
          3.6095e+00, -1.6780e+00,  9.3089e-01, -1.7266e+00, -2.3228e+00],
        [-3.5282e-01, -2.2260e-01, -5.0407e-01, -8.8783e-02,  1.5501e+00,
         -6.1404e-01, -1.8794e+00,  2.5478e+00, -2.1103e+00,  2.4213e+00],
        [ 3.3746e+00,  1.5732e+00, -9.2370e-01, -7.0857e-01, -1.9817e+00,
         -2.9782e+00, -3.5029e+00, -2.0559e+00,  3.9011e+00,  3.1880e+00],
        [-2.7136e+00, -2.1364e+00,  2.3538e+00,  2.2541e+00,  3.3178e+00,
          2.8204e+00,  2.5427e+00,  1.8494e+00, -4.9199e+00, -4.0796e+00]],
       device='cuda:0', grad_fn=<GatherBackward>)

but like below,  when I used evaluator, I found something strange:
logfile2 = open('logs/evaluator.log', 'w')

evaluator = create_evaluator(model)

@evaluator.on(Events.ITERATION_COMPLETED)
def log_inference_output(engine):
    y_pred, y = engine.state.output

    logfile2.write('{0}th iteration...\n'.format(engine.state.iteration))
    logfile2.write('    o: {0}\n'.format(y_pred))
    logfile2.write('    t: {0}\n\n'.format(y))

evaluator.run(dataloader)

logfile2.close()

1th iteration...
    o: tensor([[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]], device='cuda:0')
    t: tensor([3, 8, 8, 0, 6, 6, 1, 6, 3, 1, 0, 9, 5, 7, 9, 8, 5, 7, 8, 6],
       device='cuda:0')

Why this situation happens? I tried to figure out this for several days but I can’t get the answer.
My environment is:

pytorch: 1.7.0
ignite: 0.4.2

and I’m using 8 GPUs with dataparalled model.
Any helps will be appreciated. thanks.

Context:
I’ve just tested your code to train resnet18 QAT 8 bits on CIFAR10 with DP on 2 GPUs and I also have nans with loss and y_pred. Then tested on a single gpu without DP and the model looks like learning something.
Maybe, you can try to wrap the model with torch DDP such that you could use all 8 GPUs:
    import ignite.distributed as idist
    # ...

    model = resnet18_QAT_8b(...)
    lrank = idist.get_local_rank()
    model = model.to(idist.device())
    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[lrank,], find_unused_parameters=True)

and you have to use distributed data sampling…
In my case with CIFAR10, resnet18 QAT 8bits model trained with nccl DDP does not throw nans and achieves test accuracy: 0.8453 on 24 epochs  (~ same as single GPU same model’s test accuracy = 0.846).
Link to the code with DDP, cifar10, resnet qat : ignite/examples/contrib/cifar10_qat at cifar10-qat · vfdev-5/ignite · GitHub 1


Question:
Hello, I am trying to train a custom database that I created and my labels are  in the following format: 10988, 10995,10995, etc.
So, I’m getting the next error
Engine run is terminating due to exception: y_pred contains less classes than y. Number of predicted classes is 12 and element in y has invalid class = 10999..

So, if I check the documentations
if y.max() + 1 > num_classes:
                raise ValueError(
                    "y_pred contains less classes than y. Number of predicted classes is {}"
                    " and element in y has invalid class = {}.".format(num_classes, y.max().item() + 1)
                )

What I think is that pytorch wants my labels to be in the format 0,1,2,3,4,5,6,7,8,9,10,11
Is there a way I can fix this without changing my database’s labels?
Thanks in advance

Context:
OK, I see.
Personally, I’d perform the mapping inside Dataset before the training/validation code:
train_dataset = ...  # your training dataset
val_dataset = ...  # your validation dataset etc

class TransformedDataset(torch.utils.data.Dataset):
    def __init__(self, dataset, transform_fn):
        self.dataset = dataset
        self.transform_fn = transform_fn

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, index):
        dp = self.dataset[index]
        return self.transform_fn(dp)

from functools import partial

labels_mapping = {
    10988: 0,
    10995: 1,
    ...
}

def map_targets_fn(dp, target_mapping):
    x, y = dp
    new_y  = target_mapping[y]
    return x, new_y

train_dataset = TransformedDataset(train_dataset, partial(map_targets_fn, target_mapping=labels_mapping))
val_dataset = TransformedDataset(val_dataset, partial(map_targets_fn, target_mapping=labels_mapping))

# same code as before without any change to metrics

HTH


Error message Ignite Engine Thread Pickle

Question:
Hi I am using pytorch ignite wrapped in the MONAI package. My code worked until 3h ago when suddenly I started receiving this code. I reverted all my changes back and the error still persists. Does anyone know more?
ERROR:ignite.engine.engine.SupervisedTrainer:Exception: cannot pickle '_thread.lock' object
Traceback (most recent call last):
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/ignite/engine/engine.py", line 728, in _internal_run
    self._setup_engine()
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/ignite/engine/engine.py", line 710, in _setup_engine
    self._dataloader_iter = iter(self.state.dataloader)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 801, in __init__
    w.start()
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object

Context: @quornmd could you please provide versions of MONAI and Ignite ?


Pytorch-ignite Training not happening on training set

Question:
I suspect there is an issue and the training is somehow picking up my validation set. Here is my sample code.
I have modified the code from here: https://github.com/HLTCHKUST/Xpersona/blob/master/multilingual/train_decoder_only.py 1
    def update(engine, batch):
        model.train()
        batch = tuple(batch[input_name].to(args.device) for input_name in MODEL_INPUTS)
        input_ids, lm_labels, token_type_ids = batch
        lm_loss, prediction_scores, *_ = model(input_ids = input_ids, token_type_ids= token_type_ids, lm_labels = lm_labels)
        #batch = tuple(input_tensor.to(args.device) for input_tensor in batch)
        loss = (lm_loss) / args.gradient_accumulation_steps
        if args.fp16:
            with amp.scale_loss(loss, optimizer) as scaled_loss:
                scaled_loss.backward()
                print("BACKWARD CALLED fp16")
            torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_norm)
        else:
            print("BACKWARD CALLED")
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_norm)
        if engine.state.iteration % args.gradient_accumulation_steps == 0:
            print("Step called")
            optimizer.step()
            optimizer.zero_grad()
        return loss.item()
    trainer = Engine(update)

    # Evaluation function and evaluator (evaluator output is the input of the metrics)
    def inference(engine, batch):
        model.eval()
        with torch.no_grad():
            batch = tuple(batch[input_name].to(args.device) for input_name in MODEL_INPUTS)
            input_ids, lm_labels, token_type_ids = batch
            logger.info(tokenizer.decode(input_ids[0, :].tolist()))
            # if we dont send labels to model, it doesnt return losses
            lm_logits, *_ = model(input_ids = input_ids, token_type_ids= token_type_ids)
            lm_logits_flat_shifted = lm_logits[..., :-1, :].contiguous().view(-1, lm_logits.size(-1))
            lm_labels_flat_shifted = lm_labels[..., 1:].contiguous().view(-1)
            return (lm_logits_flat_shifted, ), (lm_labels_flat_shifted, )
    evaluator = Engine(inference)

The way I use them is as follows:
    # Attach evaluation to trainer: we evaluate when we start the training and at the end of each epoch
    trainer.add_event_handler(Events.EPOCH_COMPLETED, lambda _: evaluator.run(val_loader))
    if args.n_epochs < 1:
        trainer.add_event_handler(Events.COMPLETED, lambda _: evaluator.run(val_loader))
    if args.eval_before_start:
        trainer.add_event_handler(Events.STARTED, lambda _: evaluator.run(val_loader))

    # Make sure distributed data samplers split the dataset nicely between the distributed processes
    if args.distributed:
        trainer.add_event_handler(Events.EPOCH_STARTED, lambda engine: train_sampler.set_epoch(engine.state.epoch))
        evaluator.add_event_handler(Events.EPOCH_STARTED, lambda engine: valid_sampler.set_epoch(engine.state.epoch))

    # Linearly decrease the learning rate from lr to zero
    scheduler = PiecewiseLinear(optimizer, "lr", [(0, args.lr), (args.n_epochs * len(train_loader), 0.0)])
    trainer.add_event_handler(Events.ITERATION_STARTED, scheduler)

    # Prepare metrics - note how we compute distributed metrics
    RunningAverage(output_transform=lambda x: x).attach(trainer, "loss")
    metrics = {"nll": Loss(torch.nn.CrossEntropyLoss(ignore_index=-1), output_transform=lambda x: (x[0][0], x[1][0]))}
    metrics.update({"average_nll": MetricsLambda(average_distributed_scalar, metrics["nll"], args)})
    metrics["average_ppl"] = MetricsLambda(math.exp, metrics["average_nll"])
    for name, metric in metrics.items():
        metric.attach(evaluator, name)

    # On the main process: add progress bar, tensorboard, checkpoints and save model, configuration and tokenizer before we start to train
    if args.local_rank in [-1, 0]:
        pbar = ProgressBar(persist=True)
        pbar.attach(trainer, metric_names=["loss"])
        evaluator.add_event_handler(Events.COMPLETED,
                                    lambda _: pbar.log_message("Validation: %s" % pformat(evaluator.state.metrics)))

        log_dir = make_logdir(args.model_checkpoint)
        log_dir += "_decoder_only"
        if args.no_lang_id:
            log_dir += "_noid"
        tb_logger = TensorboardLogger(log_dir)

        tb_logger.attach(trainer, log_handler=OutputHandler(tag="training", metric_names=["loss"]),
                         event_name=Events.ITERATION_COMPLETED)
        tb_logger.attach(trainer, log_handler=OptimizerParamsHandler(optimizer), event_name=Events.ITERATION_STARTED)
        tb_logger.attach(evaluator, log_handler=OutputHandler(tag="validation", metric_names=list(metrics.keys())),event_name=Events.EPOCH_COMPLETED)
        tb_logger.attach(evaluator, log_handler=OutputHandler(tag="validation", metric_names=list(metrics.keys()), global_step_transform=global_step_from_engine(trainer)), event_name=Events.EPOCH_COMPLETED)

        checkpoint_handler = ModelCheckpoint(log_dir, 'checkpoint', save_interval=1, n_saved=3)
        trainer.add_event_handler(Events.EPOCH_COMPLETED, checkpoint_handler, {
            'mymodel': getattr(model, 'module', model)})  # "getattr" takes care of distributed encapsulation

        torch.save(args, log_dir + '/model_training_args.bin')
        getattr(model, 'module', model).config.to_json_file(
            os.path.join(log_dir, CONFIG_NAME))  # the config for encoder and decoder should be the same
        tokenizer.save_pretrained(log_dir)

    # Run the training
    trainer.run(train_loader, max_epochs=args.n_epochs)

Now my dataset length is:

INFO:./train_decoder_only.py:Train dataset length: 112449
INFO:./train_decoder_only.py:Valid dataset length: 28113

However in the training loop:

INFO:ignite.engine.engine.Engine:Engine run starting with max_epochs=1.
[1/28113]   0%|           [00:00<?]
Epoch: [1/28113]   0%|           [00:00<00:01]
Epoch: [1/28113]   0%|          , loss=2.61 [00:00<00:09]
Epoch: [1/28113]   0%|          , loss=2.61 [00:00<1:08:23]
Epoch: [1/28113]   0%|          , loss=2.62 [00:00<1:08:35]
Epoch: [2/28113]   0%|          , loss=2.62 [00:00<1:08:37]
Epoch: [2/28113]   0%|          , loss=2.62 [00:00<1:08:37]
Epoch: [2/28113]   0%|          , loss=2.63 [00:00<1:08:37]
Epoch: [3/28113]   0%|          , loss=2.63 [00:00<1:05:06]
Epoch: [3/28113]   0%|          , loss=2.63 [00:00<1:05:06]

I do not understand what all of my training set is picked or used for training. IS there a way to fix this?

Context:
Hey thanks for the input.
I found the mistake in the understanding

INFO:./train_decoder_only.py:Train dataset length: 112449

Here we are printing the length of dataset:

len(train_dataset)

Where are

Epoch: [1/28113] 0%| [00:00<00:01]

is the length of the dataset loader with batch size 4.  so 1122449/4=28113 which is, unfortunately, the length of validation dataset  leading to all the confusion.


Question:
I have a working ignite setup - precision, recall accuracy all work. However, I am getting a bizarre error when I try to add in ConfusionMatrix(num_classes=1).  Note the batch size is 5, and this pertains to a binary classifier.
/usr/local/lib/python3.7/site-packages/ignite/metrics/confusion_matrix.py in update(self, output)
92     @reinit__is_reduced
93     def update(self, output: Sequence[torch.Tensor]) -> None:
—> 94         self._check_shape(output)
95         y_pred, y = output
96
/usr/local/lib/python3.7/site-packages/ignite/metrics/confusion_matrix.py in _check_shape(self, output)
78                 "y_pred must have shape (batch_size, num_categories, …) and y must have "
79                 "shape of (batch_size, …), "
—> 80                 “but given {} vs {}.”.format(y.shape, y_pred.shape)
81             )
82
ValueError: y_pred must have shape (batch_size, num_categories, ...) and y must have shape of (batch_size, ...), but given torch.Size([5, 1]) vs torch.Size([5, 1]).
As you can see, the y_pred and y shapes match…any ideas what might be going on here?

Context:
I agree that the error message

y_pred must have shape (batch_size, num_categories, ...) and y  must have shape of (batch_size, ...), but given torch.Size([5, 1]) vs  torch.Size([5, 1])

is not clear enough as it intends to say the following:

y_pred must have shape (batch_size, num_categories, …) and given torch.Size([5, 1]) = OK
y must have shape of (batch_size, …) and given torch.Size([5, 1]) => Wrong

y should have the shape : (5, ) without 1
Could you please open an issue to improve the docs please on the github repository ?


FastaiLRFinder does not allow running more than 1 epoch

Question:
I have been trying to use the FastaiLRFinder to find the best learning rate for my module.
If we create the trainer with the function create_supervised_trainer as below:
trainer = create_supervised_trainer(
    model, optimizer, criterion, device, output_transform=custom_output_transform
)

and run it:
with lr_finder.attach(
            trainer,
            to_save=to_save,
            num_iter=50,
            end_lr=1.,
            step_mode='exp') as lr_finder_training:
        lr_finder_training.run(train_loader)

A warning will come up say: “UserWarning: Desired num_iter 50 is unreachable with the current run setup of 15 iteration (1 epochs) from ignite.contrib.handlers.param_scheduler import (LRScheduler, PiecewiseLinear)”
My dataloader has 15 batches to iterate, which means FastaiLRFinder does not allow you to run more than 1 epoch. Why?
According to their source code, we can see here 2 that this verification limits the user to run more iterations than the maximum of epochs in their dataloader.
But why? Am I missing something important here?

Context:
Let’s discuss it here : https://github.com/pytorch/ignite/issues/1357 3
Most likely https://github.com/pytorch/ignite/issues/1357#issuecomment-703485006 1 solves the issue.


Evaluate multiple models with one evaluator results weird metrics

Question:
Hello.
I have several similar models.(with little variation of some hyperparameters)
I am trying to evaluate these models for same dataset with one evaluator, to reduce eval time.
So I made custom evaluator like below:
metrics_group = {
    experiment_name: {'Top-1': Accuracy(output_transform=lambda output: (output[0][experiment_name], output[1][experiment_name])),
                      'Top-5': TopKCategoricalAccuracy(output_transform=lambda output: (output[0][experiment_name], output[1][experiment_name]))}
    for experiment_name in experiment_settings.keys()
}

def _inference(engine, batch):
    data = {experiment_name: _prepare_batch(batch,
                                            device=experiment_settings[experiment_name]['device_config']['host'],
                                            non_blocking=True)
            for experiment_name in experiment_components.keys()}
    y_preds = {}
    with torch.no_grad():
        for experiment_name, experiment_component in experiment_components.items():
            experiment_component['model'].eval()
            y_preds[experiment_name] = experiment_component['model'](data[experiment_name][0]) # Is this work simultaneously? IDK...
        return y_preds, {experiment_name: xy[1] for experiment_name, xy in data.items()}

evaluator = Engine(_inference)

for experiment_name, metrics in metrics_group.items():
    for metric_name, metric in metrics.items():
        metric.attach(evaluator, '{0}/{1}'.format(experiment_name, metric_name))

As you can see, I attached  ‘Top-1’ and ‘Top-5’ to measure every model but the results says:
print(evaluator.state.metrics)
{'A/Top-1': 0.1328125,
 'A/Top-5': 0.6015625,
 'B/Top-1': 0.1328125,
 'B/Top-5': 0.6015625,
 'C/Top-1': 0.1328125,
 'C/Top-5': 0.6015625,
 'D/Top-1': 0.1328125,
 'D/Top-5': 0.6015625}

I confused with this result so I checked evaluator’s output, but:
print(evaluator.state.output[0]['A']
tensor([[-0.3352, -2.0896, -0.1886,  ...,  0.3233,  0.5214, -0.0945],
        [-0.3316, -2.0419, -0.1706,  ...,  0.2498,  0.5802, -0.0909],
        [-0.3056, -2.0395, -0.2438,  ...,  0.2266,  0.5328, -0.1037],
        ...,
        [-0.3001, -2.0332, -0.2573,  ...,  0.3248,  0.5248, -0.0653],
        [-0.3233, -2.0452, -0.1502,  ...,  0.2362,  0.5626, -0.0756],
        [-0.3304, -2.0787, -0.1769,  ...,  0.2427,  0.5589, -0.0379]],
       device='cuda:0')

print(evaluator.state.output[0]['B'])
tensor([[ 0.9059, -0.0701,  2.3905,  ...,  0.9909,  2.3744,  0.6785],
        [ 0.8492, -0.0840,  2.3493,  ...,  0.9192,  2.3537,  0.6840],
        [ 0.8844, -0.1049,  2.3237,  ...,  0.8993,  2.3218,  0.6488],
        ...,
        [ 0.8758, -0.1526,  2.3674,  ...,  1.0029,  2.3428,  0.6295],
        [ 0.8461, -0.0652,  2.3746,  ...,  0.9019,  2.2553,  0.6627],
        [ 0.9023, -0.0956,  2.3242,  ...,  0.9092,  2.2694,  0.6629]],
       device='cuda:0')

print(torch.eq(evaluator.state.output[0]['A'], evaluator.state.output[0]['B']))
tensor([[False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        ...,
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False]], device='cuda:0')

I think there’s no problem with  ‘predicting’ phase.
Is there a problem with attaching metrics? Any suggestion will be welcome.

Context:
@FruitVinegar I think the problem is related to output transform definition with lambdas. Lambdas can not store experiment_name internally, so all output_transform fetch the last experiment_name.
For example, take a look:
ot_list = []

for n in ["a", "b", "c", "d"]:
    ot_list.append(lambda _: print(n))

for o in ot_list:
    o(None)
> d
> d
> d
> d

In order to do what you would like I’d use functools.partial
from functools import partial

ot_list = []

def ot_func(output, exp_name):
    print(output, exp_name)

for n in ["a", "b", "c", "d"]:
    ot_list.append(partial(ot_func, exp_name=n))

for o in ot_list:
    o(1)

> 1 a
> 1 b
> 1 c
> 1 d


How access inputs in custom Ignite Metric?

Question:
I have implemented a custom Ignite Metric based on this tutorial 2.
def update(self, output):
        y_pred, y = output

How to access inputs (x values) in the update function?

Context:
Here is a colab with an example : https://colab.research.google.com/drive/1-EL_YGLPzEPIw_6jU-tWRRLXnx0lN106?usp=sharing 5
import torch
import torch.nn as nn

from ignite.metrics import Metric, Accuracy
from ignite.engine import create_supervised_evaluator

class CustomMetric(Metric):

    _required_output_keys = ("y_pred", "y", "x")

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def update(self, output):
        print("CustomMetric: output=")
        for i, o in enumerate(output):
            print(i, o.shape)

    def reset(self):
        pass

    def compute(self):
        return 0.0

model = nn.Linear(10, 3)

metrics = {
    "Accuracy": Accuracy(),
    "CustomMetric": CustomMetric()
}

evaluator = create_supervised_evaluator(
    model,
    metrics=metrics,
    output_transform=lambda x, y, y_pred: {"x": x, "y": y, "y_pred": y_pred}
)

data = [
    (torch.rand(4, 10), torch.randint(0, 3, size=(4, ))),
    (torch.rand(4, 10), torch.randint(0, 3, size=(4, ))),
    (torch.rand(4, 10), torch.randint(0, 3, size=(4, )))
]
res = evaluator.run(data)

Output:
CustomMetric: output=
0 torch.Size([4, 3])
1 torch.Size([4])
2 torch.Size([4, 10])
CustomMetric: output=
0 torch.Size([4, 3])
1 torch.Size([4])
2 torch.Size([4, 10])
CustomMetric: output=
0 torch.Size([4, 3])
1 torch.Size([4])
2 torch.Size([4, 10])


Problems with Dice Loss in Pytorch Ignite

Question:
Hi, I am having issues with Dice Loss and Pytorch Ignite. I am trying to reproduce the result of Ternausnet using dice loss but my gradients keep being zero and loss just does not improve or shows very strange results (negative, nan, etc). I am not sure where to look for a possible source of the issue. Below is the code for DiceLoss:
from torch import nn
from torch.nn import functional as F

# from catalyst.contrib.nn import DiceLoss
import torch

class DiceLoss(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, logits, targets, eps=1, threshold=None):

        # comment out if your model contains a sigmoid or
        # equivalent activation layer
        proba = torch.sigmoid(logits)
        proba = proba.view(proba.shape[0], 1, -1)
        targets = targets.view(targets.shape[0], 1, -1)
        if threshold:
            proba = (proba > threshold).float()
        # flatten label and prediction tensors

        intersection = torch.sum(proba * targets, dim=1)
        summation = torch.sum(proba, dim=1) + torch.sum(targets, dim=1)
        dice = (2.0 * intersection + eps) / (summation + eps)
        # print(intersection, summation, dice)
        return (1 - dice).mean()

and here is the model (unet11 backbone code taken from ternausnet):
import pytorch_lightning as pl
import torch
from torch import nn
from torchvision import models
from carvana_unet.utils import DiceLoss

def conv3x3(in_: int, out: int) -> nn.Module:
    return nn.Conv2d(in_, out, 3, padding=1)

class ConvRelu(nn.Module):
    def __init__(self, in_: int, out: int) -> None:
        super().__init__()
        self.conv = conv3x3(in_, out)
        self.activation = nn.ReLU(inplace=True)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.conv(x)
        x = self.activation(x)
        return x

class DecoderBlock(nn.Module):
    def __init__(
        self, in_channels: int, middle_channels: int, out_channels: int
    ) -> None:
        super().__init__()

        self.block = nn.Sequential(
            ConvRelu(in_channels, middle_channels),
            nn.ConvTranspose2d(
                middle_channels,
                out_channels,
                kernel_size=3,
                stride=2,
                padding=1,
                output_padding=1,
            ),
            nn.ReLU(inplace=True),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.block(x)

class UNet11Lightning(pl.LightningModule):
    def __init__(
        self, num_filters: int = 32, pretrained: bool = True, loss_fn=DiceLoss
    ) -> None:
        """
        Args:
            num_filters:
            pretrained:
                False - no pre-trained network is used
                True  - encoder is pre-trained with VGG11
        """
        super().__init__()
        self.loss_fn = loss_fn()
        self.pool = nn.MaxPool2d(2, 2)

        self.encoder = models.vgg11(pretrained=pretrained).features

        self.relu = self.encoder[1]
        self.conv1 = self.encoder[0]
        self.conv2 = self.encoder[3]
        self.conv3s = self.encoder[6]
        self.conv3 = self.encoder[8]
        self.conv4s = self.encoder[11]
        self.conv4 = self.encoder[13]
        self.conv5s = self.encoder[16]
        self.conv5 = self.encoder[18]

        self.center = DecoderBlock(
            num_filters * 8 * 2, num_filters * 8 * 2, num_filters * 8
        )
        self.dec5 = DecoderBlock(
            num_filters * (16 + 8), num_filters * 8 * 2, num_filters * 8
        )
        self.dec4 = DecoderBlock(
            num_filters * (16 + 8), num_filters * 8 * 2, num_filters * 4
        )
        self.dec3 = DecoderBlock(
            num_filters * (8 + 4), num_filters * 4 * 2, num_filters * 2
        )
        self.dec2 = DecoderBlock(
            num_filters * (4 + 2), num_filters * 2 * 2, num_filters
        )
        self.dec1 = ConvRelu(num_filters * (2 + 1), num_filters)

        self.final = nn.Conv2d(num_filters, 1, kernel_size=1)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        conv1 = self.relu(self.conv1(x))
        conv2 = self.relu(self.conv2(self.pool(conv1)))
        conv3s = self.relu(self.conv3s(self.pool(conv2)))
        conv3 = self.relu(self.conv3(conv3s))
        conv4s = self.relu(self.conv4s(self.pool(conv3)))
        conv4 = self.relu(self.conv4(conv4s))
        conv5s = self.relu(self.conv5s(self.pool(conv4)))
        conv5 = self.relu(self.conv5(conv5s))

        center = self.center(self.pool(conv5))

        dec5 = self.dec5(torch.cat([center, conv5], 1))
        dec4 = self.dec4(torch.cat([dec5, conv4], 1))
        dec3 = self.dec3(torch.cat([dec4, conv3], 1))
        dec2 = self.dec2(torch.cat([dec3, conv2], 1))
        dec1 = self.dec1(torch.cat([dec2, conv1], 1))
        return self.final(dec1)

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
        return optimizer

    def training_step(self, batch, batch_idx):
        x, y = batch["features"].cuda(), batch["target"].cuda()
        y_hat = self(x)
        loss = self.loss_fn(y_hat, y)
        result = pl.TrainResult(loss)
        result.log("train_loss", loss)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        x, y = batch["features"].cuda(), batch["target"].cuda()
        y_hat = self(x)
        loss = self.loss_fn(y_hat, y)
        result = pl.EvalResult(checkpoint_on=loss)
        result.log("val_loss", loss, prog_bar=True, on_step=True)
        tensorboard_log = {"val_loss": loss}
        return {"loss": loss, "log": tensorboard_log}

    def test_step(self, batch, batch_idx):
        x = batch["features"].cuda()
        y_hat = self(x)
        return torch.nn.functional.sigmoid(y_hat)

can anybody help me find the source of the issue or point me into the direction of where to look? I have tried other losses (like BCE with logits which was decreasing into -inf) and nothing seems to work.

Context: Moving to Ignite category for better visibility. CC @vfdev-5


How to pass additional parameters to forward function during training and validation?

Question:
I use Ignite to train LSTM model based on this tutorial 1.
def forward(self, source, target, teacher_force_ratio=0.5):
How to pass target and teacher_force_ratio during training (teacher_force_ratio=0.5) and validation process (teacher_force_ratio=0)?
I have checked create_supervised_trainer 2 and could not find such possibility.
Should I extend Dataset class:
return (source, target) -> return ((source, target, teacher_force_ratio), target)

Context:
Hi @odats
You can create a trainer with a custom train_step (or update_fn) as here:
https://pytorch.org/ignite/concepts.html#engine 9
HTH


How to set the same random seed for all workers?

Question:
Without setting a random seed the data loader returns the same random data for each epoch:
epoch 1: worker1->[2], worker2->[2], epoch 2:  worker1->[2], worker2->[2],...
When I set a random seed in worker_init_fn function I get random data for each worker:
epoch 1: worker1->[2], worker2->[4], epoch 2:  worker1->[7], worker2->[6],...
How to set a random seed to get a new random data each epoch but the same random data for each worker?
epoch 1: worker1->[2], worker2->[2], epoch 2:  worker1->[7], worker2->[7],...
def worker_init_fn(worker_id):
    print(torch.utils.data.get_worker_info().seed)
    print(torch.initial_seed())
    # set seed
    np.random.seed(int(torch.utils.data.get_worker_info().seed)%(2**32-1))

train_loader = DataLoader(DatasetRandom(), batch_size=2, num_workers=4, worker_init_fn=worker_init_fn)

I use Ignite for training.

Context:
@odats can you set a seed each epoch for that ?
@trainer.on(Events.EPOCH_STARTED)
def set_epoch_seed():
    set_seed(trainer.state.epoch)

If this does not work for you, please provide a minimal code snippet to see the problem.


Question:
Hi, I’m not sure why but today when I run the same code that I ran last night I get this error:
   from ignite.utils import setup_logger
  File "/home/mia/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/__init__.py", line 2, in <module>
    import ignite.distributed
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/__init__.py", line 1, in <module>
    from ignite.distributed.auto import *
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/auto.py", line 10, in <module>
    from ignite.distributed import utils as idist
  File "/home/useranaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/utils.py", line 8, in <module>
    from ignite.distributed.comp_models import (
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/comp_models/__init__.py", line 2, in <module>
    from ignite.distributed.comp_models.native import has_native_dist_support
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/comp_models/native.py", line 22, in <module>
    class _NativeDistModel(ComputationModel):
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/comp_models/native.py", line 305, in _NativeDistModel
    "AND": dist.ReduceOp.BAND,
AttributeError: type object 'torch.distributed.ReduceOp' has no attribute 'BAND'

Any idea?

Context:
@pytorchnewbie what is your pytorch version ?
It should be >= 1.3


Computing every iteration, every 20 epochs

Question:
Hi!
What if I wanted to compute something on every batch output (so every iteration), but I only wanted to do this every 20 epochs? How could I do this? Is there a sort of “detach_event_handler” as a counterpart to attach_event_handler? Or is there some other way to do this? Thanks!

Context:
I might have misunderstood your question but could you not just use:
if(epoch%20 == 0):


Question:
Can anyone suggest me good toutorial for Pytorch ignite.
I searched but can’t find anything.

Context:
@Aman_Gupta how about

https://pytorch.org/ignite/quickstart.html 2
https://pytorch.org/ignite/concepts.html

We have also various tutorials with Jupyter Notebooks:

https://pytorch.org/ignite/examples.html#notebooks 2

Let me know if it does not help you to start. Thanks


Add_event_handler arguments not passing correctly?

Question:
Hi!
So when I do:
def log_engine_output(
    engine: Engine,
    fields: Dict[LOG_OP, Union[List[str], List[VisPlot], List[VisImg]]],
    epoch_num=None,
) -> None:
    """Log numerical fields in the engine output dictionary to stdout"""

    print(epoch_num)

    for mode in fields.keys():
        if (
            mode is LOG_OP.NUMBER_TO_VISDOM
            or mode is LOG_OP.VECTOR_TO_VISDOM
            or mode is LOG_OP.IMAGE_TO_VISDOM
        ):
            mode(
                engine,
                engine.state.vis,
                fields[mode],
                engine_attr="output",
                epoch_num=epoch_num,
            )
        else:
            mode(engine, fields[mode], engine_attr="output")

trainer.add_event_handler(
            Events.EPOCH_COMPLETED(every=1),
            log_engine_output,
            {
                # Log fields as message in logfile
                LOG_OP.LOG_MESSAGE: ["loss"],
                # Log fields as separate data files
                LOG_OP.SAVE_IN_DATA_FILE: ["loss"],
                # Plot fields to Visdom
                LOG_OP.NUMBER_TO_VISDOM: [
                    # First plot, key is "p1"
                    VisPlot(
                        "loss",
                        plot_key="p1",
                        split="nll_1",
                        title="Train loss: ",
                        x_label="Iters",
                        y_label="nll",
                        env=cfg.env,
                    ),
                ],
            },
            trainer.state.epoch,
        )

The epoch_num that is printed is always 0. That’s not right since it prints every epoch - why would this happen?
I’m also confusing about the structure of doing trainer.add_event_handler() vs. the decorator of
@trainer.on(Events.EPOCH_COMPLETED(every=1)).
I thought that the only difference between the decorator and the add_event_handler(some_function, variable1, variable2) was that with add_event_handler you could pass in the variables that are the arguments to the function (variable 1, variable 2), whereas with the decorator @train you can only pass in “engine” as your argument. Is this correct? If so, why isn’t passing the arguments working for me?
Also - what is the situation with add_event_handler automatically checking to see if the first argument it finds (after the function),is  “engine”? From the tutorial, “The first argument can be optionally engine, but not necessary.”.
def log_metrics(engine, title):
    print("Epoch: {} - {} accuracy: {:.2f}"
           .format(trainer.state.epoch, title, engine.state.metrics["acc"]))

@trainer.on(Events.EPOCH_COMPLETED)
def evaluate(trainer):
    with evaluator.add_event_handler(Events.COMPLETED, log_metrics, "train"):
        evaluator.run(train_loader)

Here it automatically passes “evaluator” as the “engine” argument to log_metrics. However, here:
trainer = Engine(update_model)

trainer.add_event_handler(Events.STARTED, lambda _: print("Start training"))
# or
@trainer.on(Events.STARTED)
def on_training_started(engine):
    print("Another message of start training")
# or even simpler, use only what you need !
@trainer.on(Events.STARTED)
def on_training_started():
    print("Another message of start training")

# attach handler with args, kwargs
mydata = [1, 2, 3, 4]

def on_training_ended(data):
    print("Training is ended. mydata={}".format(data))

trainer.add_event_handler(Events.COMPLETED, on_training_ended, mydata)

The first argument is not “engine”. Does this mean that somehow ignite checks the function passed into add_event_handler to see if the first argument is called “engine”? And if so, does the nomenclature have to be exactly called “engine” in order for it to pass the engine that add_event_handler is attached to to the function?
Thanks!

Context:
Hi @pytorchnewbie

The epoch_num that is printed is always 0. That’s not right since it prints every epoch - why would this happen?

This a correct behaviour as passing trainer.state.epoch it is passed as value and not reference. I agree that it would be nice to have it as reference, but I have no idea how to do that in python.
You need to fetch epoch inside the handler with trainer.state.epoch.


Question:
Hi,
In one of ignite’s issues (https://github.com/pytorch/ignite/pull/48/files 1) I saw that the _inference was passed in to the trainer engine. Why is this option not in the docs? Is it not recommended to do this? Why not?
Thanks

Context: Moving to Ignite category for better visibility and CC’ing @vfdev-5


Storing values in metric or returning them in engine’s dict for plotting purposes

Question:
Hi! I have a question about how to best plot things. Suppose I have a model and that during inference I am getting the error per example in a batch. I then want to correlate the error of the predicted y with a property of the input x. So ultimately I’m plotting a huge amount of property(x) and error(y,y_pred) on a scatterplot, and then calculating their correlation. I assume that the best way to do this would be to log every batch x properties and y errors using a custom metric, and then outputing the correlation between the property(x) and error(y,y_pred) for all x,y pairs at the end through the compute() function.
However, what if I also wanted a scatterplot of all the property(x) and error(y,y_pred) for all pairs x,y? I understand that metric stores the history, but would it be too convoluted to also plot all the property(x) and error(y,y_pred) from within Metric’s compute() function? It seems complicated because I create my visdom Visualizer in a file called “train.py” (where I also run my evaluator) and the custom Metric is in its own file called metric.py.
The only other option I can think of is to store all the property(x) and error(y,y_pred) outside of the Metric.py file by returning the property(x) and error(y,y_pred) in the dict returned by the evaluator engine in train.py, and then creating an event handler that will save those values every iteration until the end of the validation set. After this, all the property(x) and error(y,y_pred) are saved in a file, and I just repull them all to do a scatter plot.
I was wondering what is the best/more ignite/most elegant way to do this.
Thanks!

Context:
Hi @pytorchnewbie,
IMO, it would be better to separate the logic of metrics computation and logging for visualization.
def inference_step(e, batch):
    # ...
    x = batch["x"]
    y_pred = model(x)
    return {
        "y_pred": y_pred,
    }

infer_engine = Engine(inference_step)
infer_engine.state.viz_data = {}  # data storage for visualization

@infer_engine.on(Events.ITERATION_COMPLETED)
def compute_iterationwise_error(engine):
    batch = engine.state.batch
    x = batch["x"]
    y = batch["y"]
    y_pred = engine.state.output["y_pred"]

    err = compute_error(y,y_pred)
    engine.state.viz_data["err"] = err
    engine.state.viz_data["x_properties"] = get_properties(x)

plot_every = 1 # 2 or 10

@infer_engine.on(Events.ITERATION_COMPLETED(every=plot_every))
def plot_infer_data(engine):
    viz_data = engine.state.viz_data
    visdom_plot_data(viz_data["x_properties"], viz_data["err"])

For me, it is not clear from the description, how often you would like to compute a metric:

epoch-wise (1 per the dataset)
batch-wise (1 per batch)


Question: Suppose I have two metrics, mse and correlation. Every 2 epochs of the training, I want to run the evaluator and collect the MSE metric. Also, at the end of training (800 epochs later), I want to run the evaluator and collect and the correlation metric. However, the metrics are attached to the evaluator engine. Therefore, every time evaluator is run, both metrics are calculated. This seems inefficient, since I only need to calculate correlation on the evaluator run at the end of the 800 training epochs, not 400 times (800/2) during training. Thoughts?

Context:
You can attach/detach metrics, so I think you can just attach it at the end of the training:
@trainer.on(Events.EPOCH_COMPLETED(every=2))
def compute_mse():
    evaluator.run(validation)

@trainer.on(Events.COMPLETED)
def compute_correlation():
    correlation_metric.attach(evaluator, "corr")
    evaluator.run(validation)
    # optionally
    correlation_metric.detach(evaluator)


Question:
In research, a lot of the time we will be comparing machine learning approaches to non-machine learning approaches. For the latter, there isn’t usually a notion of “iterations” or “epochs”. The model does what it does and outputs an error. Since the idea would be to code all the experiments in the same framework, I was wondering how to use ignite for non-learning based approaches/if it makes sense to do that. For instance, does it make sense to use ignite’s “metrics” class for a non-learning based approach? Does using any of ignite make sense for non-learning approaches?
Thanks!

Context:
Hi @pytorchnewbie

The model does what it does and outputs an error. Since the idea would be to code all the experiments in the same framework, I was wondering how to use ignite for non-learning based approaches/if it makes sense to do that. For instance, does it make sense to use ignite’s “metrics” class for a non-learning based approach? Does using any of ignite make sense for non-learning approaches?

Probably, it depends on details etc (if you could provide some, it would help), but I think this can be just considered as inference phase: a model (any kind of) is used to compute predictions over the batches of the test dataset and we wish to save the outputs
def inference_step(engine, batch):
    sample, meta_data, what_ever_data = batch
    result = non_ml_model(sample)
    return {
        "result": result,
    }

inference_engine = Engine(inference_step)

@inference_engine.on(Events.ITERATION_COMPLETED)
def save_result(engine):
    sample, meta_data, what_ever_data = engine.state.batch
    result = engine.state.output["result"]
    # do something to save the result

Another example of using Engine can be to compute mean/std of the dataset (no ML-model at all):
from ignite.metrics import Average

def compute_mean_std(engine, batch):
    b, c, *_ = batch['image'].shape
    data = batch['image'].reshape(b, c, -1).to(dtype=torch.float64)
    mean = torch.mean(data, dim=-1).sum(dim=0)
    mean2 = torch.mean(data ** 2, dim=-1).sum(dim=0)
    return {"mean": mean, "mean^2": mean2}

compute_engine = Engine(compute_mean_std)
img_mean = Average(output_transform=lambda output: output['mean'])
img_mean.attach(compute_engine, 'mean')
img_mean2 = Average(output_transform=lambda output: output['mean^2'])
img_mean2.attach(compute_engine, 'mean2')
state = compute_engine.run(train_loader)
state.metrics['std'] = torch.sqrt(state.metrics['mean2'] - state.metrics['mean'] ** 2)
mean = state.metrics['mean'].tolist()
std = state.metrics['std'].tolist()

https://pytorch.org/ignite/engine.html#ignite.engine.engine.Engine 1
Let me know if this helps. Otherwise, please provide a minimal code snippet to understand better the question…


Difference between MeanSquaredError & Loss (where loss = mse)

Question:
Hi!
So I am trying to calculate the average mean squared error over my validation dataset. I’ve done this in two ways: using Ignite’s Loss metric, where the loss_fn = nn.MSELoss() and then using Ignite’s MeanSquaredError metric, as can be seen in the code snippets below:

loss_fn = torch.nn.MSELoss()

metrics = {
        "mse": Loss(
            loss_fn,
            output_transform=lambda infer_dict: (infer_dict["y_pred"], infer_dict["y"]),
        ),
    }

    for name, metric in metrics.items():
        metric.attach(engine, name)

vs.
metrics = {
        "mse": MeanSquaredError(
            output_transform=lambda infer_dict: (infer_dict["y_pred"], infer_dict["y"]),
        ),
    }

    for name, metric in metrics.items():
        metric.attach(engine, name)

I obtain two different results, as can be sceen from the two images below.
MeanSquaredError (the top left is evaluation MSE error, and top right is training MSE error):
meansquarederror1568×1482 246 KB
Loss:
loss1532×1460 243 KB
We can see that MSE has an error of the order of “M” whereas Loss has an error of the order of “K”. What accounts for this difference?
Initially I thought that its probably because Loss() probably calculates the average mean squared error per batch, and then takes the average of the averages, whereas MeanSquaredError (from what I saw in the source code) keeps track of all squared errors, and takes the average of all the squared errors across all the batches (so it only does one average, not average of averages). However, since the batch size is constant, the two results should be numerically equivalent. For example:
( (((7^2) + (8^2) + (13^2))/3) + (((3^2) + (6^2) + (11^2))/3)  )  / 2
is the same as:
(7^2) + (8^2) + (13^2) + (3^2) + (6^2) + (11^2)/6
because both groups in the first example have 3 elements in them.
Therefore, what accounts for the difference in magnitudes of the error functions?
Thanks so much!

Context:
Ignite’s Metric API allows to inspect what happens batch by batch. All metrics allow to do the following:
mse_metric = MeanSquaredError()

mse_metric.reset()

mse_metric.update((y_pred1, y1))
# check result of the 1st batch
print(mse_metric.compute())

mse_metric.update((y_pred2, y2))
# check result of the 1st and 2nd batch
print(mse_metric.compute())

In this way you can compare both on your predictions and targets and see where comes from the difference.
HTH
PS.
Shouldn’t loss_fn = torch.nn.MSELoss() have reduction “sum” ?


Question:
What if I want to run my evaluation engine (which itself has event handlers) with my evaluation dataloader every 20 epochs of the training engine? Right now this isn’t possible, so I made a wrapper function that allows for this below. However, I was wondering if there was an ignite way of doing this? I couldn’t find any.
def _lf_two(
    log_fn: Callable[[Engine, List[str]], None],
    inner_engine: Engine,
    loader: DataLoader,
    **kwargs
) -> Callable[[Engine], Any]:
    """Returns a lambda calling custom log function with two engines (e.g. the training loop and validation loop)"""
    return lambda outer_engine: log_fn(
        inner_engine, loader, epoch_num=outer_engine.state.epoch, **kwargs
    )

def run_engine_and_log_output(
    engine: Engine, loader: DataLoader, fields: List[str], epoch_num=None,
) -> None:

    engine.run(loader,max_epochs=1, epoch_length=10)
    ###Here i grab things from engine.state.output and do what i want###

trainer.add_event_handler(
        Events.EPOCH_COMPLETED(every=20),
        _lf_two(
            run_engine_and_log_output,
            evaluator,
            val_loader))

Also, I was wondering if I then wanted to run the evaluation outside of the training loop, and this time with different event handlers, would best practice be to create two different evaluation engines? One to pass into the training loop event handler into (__lf_two()), and one to run after my train.run ?
Thanks!

Context:
Maybe I do not correctly understand the issue, but if you would like to run model evaluation every 20-th epochs and in the end of the training it is simply done like :

trainer = ...
evaluator = ...

@trainer.on(Events.EPOCH_COMPLETED(every=20) | Events.COMPLETED)
def validate_model():
    state = evaluator.run(val_loader)
    print(trainer.state.epoch, state.metrics)

@pytorchnewbie, what do you think? Does it answer your question ?


Question:
Hi! This is a simple question but for some reason I’m having a hard time finding the answer. It says here (https://pytorch.org/ignite/metrics.html 14) that metrics doesn’t store the memory of the entire output history of the model.
However, the evaluation loop predicts y values (a regressive MLP), and the MSE is computed with the ground truth y values per batch. Therefore, the “mse” returned by the evaluation metrics thing gets overriden every batch. I want to have the “mse” of the entire validation set - this seems not compatible with the current setup.
def create_evaluation_loop(
    model: nn.Module, cfg: DictConfig, name: str, device="cpu",
) -> Engine:

    # Loss
    if cfg.sum:
        loss_fn = torch.nn.MSELoss(reduction="sum")
    else:
        loss_fn = torch.nn.MSELoss()

    def _inference(engine, batch):
        model.eval()
        with torch.no_grad():
            x, y = batch
            x, y = x.to(device), y.to(device)
            y_pred = model(x)

            if cfg.house:
                factor = 6.0
            else:
                factor = 1777.0

            # pdb.set_trace()

            y = y * factor
            y_pred = y_pred * factor

            if cfg.envelope:
                y_hat_env = torch.tensor(np.abs(hilbert(y.cpu().detach().numpy())))
                y_pred_hat_env = torch.tensor(
                    np.abs(hilbert(y_pred.cpu().detach().numpy()))
                )
                mse_val = loss_fn(y_hat_env, y_pred_hat_env).item()
            else:
                mse_val = loss_fn(y, y_pred).item()

        # Anything you want to log must be returned in this dictionary
        # pdb.set_trace()
        infer_dict = {
            "loss": mse_val,
            "y_pred": y_pred,
            "y": y,
            "ypred_first": [y[0], y_pred[0]],  # * mean + stdv,  # * mean + stdv,
        }

        return infer_dict

    engine = Engine(_inference)

    engine.logger = setup_logger(name=name)

    metrics = {
        "mse": Loss(
            loss_fn,
            output_transform=lambda infer_dict: (infer_dict["y_pred"], infer_dict["y"]),
        ),
    }

    for name, metric in metrics.items():
        metric.attach(engine, name)

    return engine

Also,  what is the  difference between the trainer which returns an “output” and the evaluator which returns the infer_dict? The difference comes when you do “metrics = {
“mse”: Loss(
loss_fn,
output_transform=lambda infer_dict: (infer_dict[“y_pred”], infer_dict[“y”]),
),
}” and then “attach” the engine and name together… but im not really quite clear on what that does. I know for fact that with the trainer, the output dictionary is very online in that it stores no history. So  I’d assume the “attach” stores the history, but then the link above specifically says that it doesn’t?
Thanks for your time!

Context:
@pytorchnewbie sorry I saw the question but forgot to answer.
If I understand correctly the problem, you would like to compute MSE metric (https://pytorch.org/ignite/metrics.html#ignite.metrics.MeanSquaredError 10).
To do that create a validation engine and attach MSE metric to that:
from ignite.engine import create_supervised_evaluator
from ignite.metrics import MeanSquaredError

val_metrics = {
    "mse": MeanSquaredError(),
}
evaluator = create_supervised_evaluator(model, metrics=val_metrics)

res = evaluator.run(val_loader)
print(res.metrics["mse"])

Also, what is the difference between the trainer which returns an “output” and the evaluator which returns the infer_dict?

Output of an engine is not restricted at all, but should be coherent with how we would like to use it after inside attached handlers/metrics. See concepts 1 for more details.
If we would like to log batch loss during the training, then we need logically to return its value. But in addition if we would like to do something with predictions or compute another error metric etc we can also put everything into output and work with the output inside attached handler.
Same for validation engine. If we use it to compute metrics, we need to output predictions and target. If we set it to do inference=compute prediction and write the prediction after inside a save_handler. We output only predictions…
Hope it is more clear.

So I’d assume the “attach” stores the history, but then the link above specifically says that it doesn’t?

It depends on metric. Some metric can compute in online manner: storing only certain internal variables. For example, MSE = 1/N sum ( sample_err^2 ). It can be computed as accumulation of all sample_err^2 and finally divided by number of seen samples. So, practically, we do not store a list of [y_pred1, y_pred2, ...] and [y1, y2, ...] to compute sample_err etc.
But there are metrics, e.g. median “something” error that can not be computed like that and we need to store two lists and execute computation on stored history.
You can also compute any metric on the training dataset during the training (see here how to do this). But, in this case you should be aware of the fact that model is also changing. So, it means that first predictions in the begining of an epoch will be worse than the last ones, but everything will be accounted to compute final overall training dataset metric. For example, accuracy on an epoch of 3 batches (bs=16) can give the following: num_correct_per_batch = [0, 4, 8]. Final accuracy is (0 + 4 + 8) / (16 + 16 + 16). However, on validation it could give (8 + 8 + 8)/(16 + 16 + 16).
Please, let me know if it answers your questions.


Change model’s attribute during training

Question:
Hello.
I’d like to change my model’s attribute during training.
In order to that, now I’m calling my model as global variable.
For example:
trainer = create_supervised_trainer(~)

@trainer.on(Events.EPOCH_STARTED(once=10)
def change_attribute(engine):
    global model
    model.my_attribute = new_attribute

Is there more clear way using trainer?
I investigated trainer.__dict__ and trainer.state.__dict__, but there’s no attributes that keeps model.

Context:
@FruitVinegar
Trainer defined as
trainer = create_supervised_trainer(~)

is just a configured Engine. Engine, by default, does not store any info on model, optimizer, loss etc.
This can be done if needed like that:
trainer = ...
# trainer.state is not None in recent nightly releases: 0.4.0.dev202005XX
trainer.state.model = model
trainer.state.optimizer = optimizer
# otherwise, in stable v0.3.0, you need to set attributes in a handler attached to `Events.STARTED`

@trainer.on(Events.EPOCH_STARTED(once=10)
def change_attribute(engine):
    trainer.state.model.my_attribute = new_attribute

About using Engine, see https://pytorch.org/ignite/concepts.html#engine 1 .
About State, please, see the documentation on it : https://pytorch.org/ignite/engine.html#ignite.engine.State 1 and https://pytorch.org/ignite/concepts.html#state

A good practice is to use State also as a storage of user data created in update or handler functions. For example, we would like to save new_attribute in the state:

def user_handler_function(engine):
    engine.state.new_attribute = 12345

HTH


RuntimeError: multi-target not supported custom dataset in ignite

Question:
Hello! I am trying to train a neural network with a preprocessed CIFAR10 : I managed to pass them from a .npy extension to a .pt extension and load them on the code
train_images=torch.from_numpy(torch.load('train_images.pt')).type(torch.LongTensor)
test_images=torch.from_numpy(torch.load('test_images.pt')).type(torch.LongTensor)
train_labels=torch.from_numpy(torch.load('train_labels.pt')).type(torch.LongTensor)
test_labels=torch.from_numpy(torch.load('test_labels.pt')).type(torch.LongTensor)

train_transform = Compose([
    ToPILImage(mode='RGB'),
    Resize(image_size, BICUBIC),
    RandomAffine(degrees=2, translate=(0.02, 0.02), scale=(0.98, 1.02), shear=2, fillcolor=(124,117,104)),
    RandomHorizontalFlip(),
    Pad(4),
    ToTensor(),
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

test_transform = Compose([
    ToPILImage(mode='RGB'),
    Resize(image_size, BICUBIC),
    Pad(4),
    ToTensor(),
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

class CIFAR100Dataset(Dataset):

    def __init__(self, images, labels=None, transforms=None):
        self.X = images
        self.y = labels
        self.transforms=train_transform

    def __len__(self):
        return(len(self.X))

    def __getitem__(self, i):
        data = self.X[i]
        data = np.asarray(data).astype(np.uint8).reshape(32, 32, 3)
        if self.transforms:
            data = self.transforms(data)
        if self.y is not None:
            return (data, self.y[i])
        else:
            return data

train_dataset = CIFAR100Dataset(train_images, train_labels, train_transform)
test_dataset = CIFAR100Dataset(test_images, test_labels, test_transform)

The problem is…when I try to train I get this message:
RuntimeError: multi-target not supported at /opt/conda/conda-bld/pytorch_1587428270644/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu:18

So, I take a look into my target data and the size is [100,100], so I change it to [100] with
y=(torch.max(y, 1)[1]).type(torch.LongTensor).cuda()

in this part of the code
def update_fn(engine, batch):
    model.train()

    x = convert_tensor(batch[0], device=device, non_blocking=True)
    y = convert_tensor(batch[1], device=device, non_blocking=True)
    y=(torch.max(y, 1)[1]).type(torch.LongTensor).cuda()
    y_pred = model(x)
    # Compute loss
    loss = criterion(y_pred,y)

    optimizer.zero_grad()
    if use_amp:
        with amp.scale_loss(loss, optimizer, loss_id=0) as scaled_loss:
            scaled_loss.backward()
    else:
        loss.backward()
    optimizer.step()

    return {
        "batchloss": loss.item(),

If I print y (my target)
tensor([27, 40, 90, 26, 50, 41, 27, 93, 84, 82, 76, 85, 93, 57, 68, 89, 25, 18,
         9, 18, 40,  7, 26, 84, 64, 73, 43, 74, 49, 18, 22, 26, 31,  7, 67, 42,
         3, 96, 53, 38, 47, 99, 26, 55, 64, 22, 29, 81, 90, 19, 16, 79, 93, 17,
        95, 42, 34, 25, 29, 47,  2, 43, 32, 94, 13,  9, 14, 45, 90, 92,  2,  9,
        11, 62, 75, 54,  7, 45, 68,  5, 24, 66, 36, 72, 43, 68,  4, 56, 57, 31,
        52, 27, 49, 19,  2, 88, 33, 11, 48, 59], device='cuda:0')

The shape is now fine
torch.Size([100])

Note: I used this same change in another code that is not ignite and it works:
    model.eval()
    for data, target in test_loader:
        # move tensors to GPU if CUDA is available

       # if contador ==35000
        if train_on_gpu:
            data, target = data.cuda(), target.cuda()

        # forward pass: compute predicted outputs by passing inputs to the model

        output = model(data)
        values,indices=torch.max(target,1)
        target=indices

So, my question here is…Why is not working on ignite? I am missing something on the update_fn function?
Regards,

Context:
The shape and the values of y look correct, so Ignite shouldn’t raise an issue.
Are you sure you’ve rerun the complete code (or cells)?
If so, could you directly print the shapes of y_pred and y before passing them to the criterion?


What is the difference of `call_function` and `call_method`?

Question:
From the FX documentation, I learned that there are several node types:

placeholder
getattr
output
call_method
call_function
call_module

I have difficulty to tell the last three concepts apart. Could someone explain them using more examples?

Context:
Here is an example:
import torch
import torch.fx as fx
import torch.nn as nn

def myadd(x, y):
    return x + y

class Adder(nn.Module):
    def forward(self, x, y):
        return x + y

class M(nn.Module):
    def __init__(self):
        super(M, self).__init__()
        self.adder = Adder()

    def forward(self, x, y):
        return x + y, myadd(x, y), torch.add(x, y), x.add(y), self.adder(x, y)

fx.symbolic_trace(M()).graph.print_tabular()

#### output
opcode         name    target                                               args                                  kwargs
-------------  ------  ---------------------------------------------------  ------------------------------------  --------
placeholder    x       x                                                    ()                                    {}
placeholder    y       y                                                    ()                                    {}
call_function  add     <built-in function add>                              (x, y)                                {}
call_function  add_1   <built-in function add>                              (x, y)                                {}
call_function  add_2   <built-in method add of type object at 0x10310d390>  (x, y)                                {}
call_method    add_3   add                                                  (x, y)                                {}
call_function  add_4   <built-in function add>                              (x, y)                                {}
output         output  output                                               ((add, add_1, add_2, add_3, add_4),)  {}

Why the Adder module not traced as call_module？
What is the difference between Tensor.add and myadd? Because Tensor.add is bound with a Tensor object and thus it is not free?


Torch.fx cannot trace torch.Size() properly

Question:
torch.fx cannot trace the model properly if there are manipulations of torch.Size() types.
This is the example to try to reconstruct a size type object from the input tensor’s first and second dimensions.
import torch
import torch.nn as nn
import torch.fx as fx

class Example (nn.Module):
    def __init__(self) -> None:
        super(Example, self).__init__()

    def forward(self, input):

        batch_size = input.size(0)
        ch_size = input.size(1)

        return torch.Size([batch_size, ch_size])

if __name__ == "__main__":

    model = Example()
    y = model(torch.randn(16, 5, 3, 3))
    print(y)

    fx_model = fx.symbolic_trace(model)
    fx_graph = fx_model.graph

    fx_graph.print_tabular()
    print(fx_model.code)

The terminal output here:
torch.Size([16, 5])
Traceback (most recent call last):
  File "trace_bug.py", line 23, in <module>
    fx_model = fx.symbolic_trace(model)
  File "/home/shengcn/anaconda3/envs/torch110/lib/python3.7/site-packages/torch/fx/_symbolic_trace.py", line 907, in symbolic_trace
    graph = tracer.trace(root, concrete_args)
  File "/home/shengcn/anaconda3/envs/torch110/lib/python3.7/site-packages/torch/fx/_symbolic_trace.py", line 615, in trace
    self.create_node('output', 'output', (self.create_arg(fn(*args)),), {},
  File "trace_bug.py", line 14, in forward
    return torch.Size([batch_size, ch_size])
TypeError: torch.Size() takes an iterable of 'int' (item 0 is 'Proxy')

Context:
I am a freshman to FX. Maybe fx.wrap can help you.
@fx.wrap
def get_size(x, y):
    return torch.Size([x, y])

class Example (nn.Module):
    def __init__(self) -> None:
        super(Example, self).__init__()

    def forward(self, input):

        batch_size = input.size(0)
        ch_size = input.size(1)
        # return torch.Size([batch_size, ch_size])
        return get_size(batch_size, ch_size)


“self.graph.owning_module not set for purity check” error when trying to remove a node from torch.fx.graph

Question:
“self.graph.owning_module not set for purity check” error when trying to remove nodes from torch.fx.graph

I have tried to remove nodes from torch.fx.graph but it does not work properly.
Here is the code example. This example uses a small model of 2 conv layers and finally sum outputs of these two layers. I changed conv2 and add layers into rogue nodes, and then let the output node connect directly to conv1.
The purity check error stops the program when I try to cleanup rogue nodes using eliminate_dead_code().
torch.Size([64, 8, 28, 28])
opcode         name    target                   args            kwargs
-------------  ------  -----------------------  --------------  --------
placeholder    x       x                        ()              {}
call_module    conv1   conv1                    (x,)            {}
call_module    conv2   conv2                    (x,)            {}
call_function  add     <built-in function add>  (conv1, conv2)  {}
output         output  output                   (add,)          {}

==== Predecessors and successors of each node: ====
node: x
Predecessors:
Successors:
	 conv1 <class 'torch.fx.node.Node'>
	 conv2 <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv1
Predecessors:
	 x <class 'torch.fx.node.Node'>
Successors:
	 add <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv2
Predecessors:
	 x <class 'torch.fx.node.Node'>
Successors:
	 add <class 'torch.fx.node.Node'>
----------------------------------------------
node: add
Predecessors:
	 conv1 <class 'torch.fx.node.Node'>
	 conv2 <class 'torch.fx.node.Node'>
Successors:
	 output <class 'torch.fx.node.Node'>
----------------------------------------------
node: output
Predecessors:
	 add <class 'torch.fx.node.Node'>
Successors:
----------------------------------------------

opcode         name    target                   args      kwargs
-------------  ------  -----------------------  --------  --------
placeholder    x       x                        ()        {}
call_module    conv1   conv1                    (x,)      {}
call_module    conv2   conv2                    ()        {}
call_function  add     <built-in function add>  ()        {}
output         output  output                   (conv1,)  {}

==== Predecessors and successors of each node: ====
node: x
Predecessors:
Successors:
	 conv1 <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv1
Predecessors:
	 x <class 'torch.fx.node.Node'>
Successors:
	 output <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv2
Predecessors:
Successors:
----------------------------------------------
node: add
Predecessors:
Successors:
----------------------------------------------
node: output
Predecessors:
	 conv1 <class 'torch.fx.node.Node'>
Successors:
----------------------------------------------

Traceback (most recent call last):
  File "github_question.py", line 77, in <module>
    graph.eliminate_dead_code()
  File "/home/shengcn/anaconda3/envs/torch110/lib/python3.7/site-packages/torch/fx/graph.py", line 1170, in eliminate_dead_code
    if not node.is_impure() and len(node.users) == 0:
  File "/home/shengcn/anaconda3/envs/torch110/lib/python3.7/site-packages/torch/fx/node.py", line 511, in is_impure
    ), "self.graph.owning_module not set for purity check"
AssertionError: self.graph.owning_module not set for purity check
Reply

created
21h
last reply
1m
1
reply
8
views
1
user

jfix
Jordan Fix
6h
You need to set graph.owning_module on the graph module. This is because for call_module nodes, we need to fetch the original module object instance to check if it is set as pure (i.e. if it has no side effects) before removing it.

BTW, this is done usually by default when calling torch.fx.symbolic_trace(). Is there a reason you don’t want to call that helper instead?

Solution
Reply

shengchunnan
1m
I updated the code a little and now it works.
Many thanks for your kind suggestion.

import torch
import torch.nn as nn
import torch.fx as fx

class Network(nn.Module):
    def __init__(self, num_classes=10):
        super(Network, self).__init__()

        self.conv1 = nn.Conv2d(1, 8, kernel_size=5, stride=1, padding=2, bias=False)
        self.conv2 = nn.Conv2d(1, 8, kernel_size=3, stride=1, padding=1, bias=False)

    def forward(self, x):

        out1 = self.conv1(x)
        out2 = self.conv2(x)

        return out1 + out2

def delete_conv2(graph):

    preserved_conv_node = None
    output_node = None

    for node in graph.nodes:
        if 'call_module' == node.op:
            if 'conv1' in node.name:
                preserved_conv_node = node
            # Remove conv2 operation from graph
            elif 'conv2' in node.name:
                node.args = ()
        # Remove the add operation from graph
        elif 'call_function' == node.op and 'add' == node.name:
            node.args = ()
        elif 'output' == node.op:
            output_node = node

    if preserved_conv_node and output_node:
        output_node.args = (preserved_conv_node,)

def print_pre_and_suc(graph):

    print('\n==== Predecessors and successors of each node: ====')
    for node in graph.nodes:
        print('node: {}'.format(node.name))

        print('Predecessors:')
        for item in node.args:
            print('\t', item, type(item))

        print('Successors:')
        for key, val in node.users.items():
            print('\t', key, type(key))
        print('----------------------------------------------')
    print()

if __name__ == "__main__":

    model = Network()

    y = model(torch.randn(64, 1, 28, 28))
    print(y.size())

    fx_model = fx.symbolic_trace(model)
    graph = fx_model.graph

    graph.print_tabular()
    print_pre_and_suc(graph)

    delete_conv2(graph)

    graph.print_tabular()
    print_pre_and_suc(graph)

    graph.eliminate_dead_code()
    fx_model.recompile()
    print(fx_model.code)

    y = fx_model(torch.randn(64, 1, 28, 28))
    print(y.size())

You can see from the log that conv2 and add are successfully changed into rogue nodes (no parents, no children), but graph.eliminate_dead_code() fails to run properly due to assertion of self.graph.owning_module.
I am wondering, why do we need this kind of assertion? And how can I make sure that owning_module is not none.

torch.Size([64, 8, 28, 28])
opcode         name    target                   args            kwargs
-------------  ------  -----------------------  --------------  --------
placeholder    x       x                        ()              {}
call_module    conv1   conv1                    (x,)            {}
call_module    conv2   conv2                    (x,)            {}
call_function  add     <built-in function add>  (conv1, conv2)  {}
output         output  output                   (add,)          {}

==== Predecessors and successors of each node: ====
node: x
Predecessors:
Successors:
	 conv1 <class 'torch.fx.node.Node'>
	 conv2 <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv1
Predecessors:
	 x <class 'torch.fx.node.Node'>
Successors:
	 add <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv2
Predecessors:
	 x <class 'torch.fx.node.Node'>
Successors:
	 add <class 'torch.fx.node.Node'>
----------------------------------------------
node: add
Predecessors:
	 conv1 <class 'torch.fx.node.Node'>
	 conv2 <class 'torch.fx.node.Node'>
Successors:
	 output <class 'torch.fx.node.Node'>
----------------------------------------------
node: output
Predecessors:
	 add <class 'torch.fx.node.Node'>
Successors:
----------------------------------------------

opcode         name    target                   args      kwargs
-------------  ------  -----------------------  --------  --------
placeholder    x       x                        ()        {}
call_module    conv1   conv1                    (x,)      {}
call_module    conv2   conv2                    ()        {}
call_function  add     <built-in function add>  ()        {}
output         output  output                   (conv1,)  {}

==== Predecessors and successors of each node: ====
node: x
Predecessors:
Successors:
	 conv1 <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv1
Predecessors:
	 x <class 'torch.fx.node.Node'>
Successors:
	 output <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv2
Predecessors:
Successors:
----------------------------------------------
node: add
Predecessors:
Successors:
----------------------------------------------
node: output
Predecessors:
	 conv1 <class 'torch.fx.node.Node'>
Successors:
----------------------------------------------

Traceback (most recent call last):
  File "github_question.py", line 77, in <module>
    graph.eliminate_dead_code()
  File "/home/shengcn/anaconda3/envs/torch110/lib/python3.7/site-packages/torch/fx/graph.py", line 1170, in eliminate_dead_code
    if not node.is_impure() and len(node.users) == 0:
  File "/home/shengcn/anaconda3/envs/torch110/lib/python3.7/site-packages/torch/fx/node.py", line 511, in is_impure
    ), "self.graph.owning_module not set for purity check"
AssertionError: self.graph.owning_module not set for purity check

Context:
I updated the code a little and now it works.
Many thanks for your kind suggestion.
import torch
import torch.nn as nn
import torch.fx as fx

class Network(nn.Module):
    def __init__(self, num_classes=10):
        super(Network, self).__init__()

        self.conv1 = nn.Conv2d(1, 8, kernel_size=5, stride=1, padding=2, bias=False)
        self.conv2 = nn.Conv2d(1, 8, kernel_size=3, stride=1, padding=1, bias=False)

    def forward(self, x):

        out1 = self.conv1(x)
        out2 = self.conv2(x)

        return out1 + out2

def delete_conv2(graph):

    preserved_conv_node = None
    output_node = None

    for node in graph.nodes:
        if 'call_module' == node.op:
            if 'conv1' in node.name:
                preserved_conv_node = node
            # Remove conv2 operation from graph
            elif 'conv2' in node.name:
                node.args = ()
        # Remove the add operation from graph
        elif 'call_function' == node.op and 'add' == node.name:
            node.args = ()
        elif 'output' == node.op:
            output_node = node

    if preserved_conv_node and output_node:
        output_node.args = (preserved_conv_node,)

def print_pre_and_suc(graph):

    print('\n==== Predecessors and successors of each node: ====')
    for node in graph.nodes:
        print('node: {}'.format(node.name))

        print('Predecessors:')
        for item in node.args:
            print('\t', item, type(item))

        print('Successors:')
        for key, val in node.users.items():
            print('\t', key, type(key))
        print('----------------------------------------------')
    print()

if __name__ == "__main__":

    model = Network()

    y = model(torch.randn(64, 1, 28, 28))
    print(y.size())

    fx_model = fx.symbolic_trace(model)
    graph = fx_model.graph

    graph.print_tabular()
    print_pre_and_suc(graph)

    delete_conv2(graph)

    graph.print_tabular()
    print_pre_and_suc(graph)

    graph.eliminate_dead_code()
    fx_model.recompile()
    print(fx_model.code)

    y = fx_model(torch.randn(64, 1, 28, 28))
    print(y.size())

torch.Size([64, 8, 28, 28])
opcode         name    target                   args            kwargs
-------------  ------  -----------------------  --------------  --------
placeholder    x       x                        ()              {}
call_module    conv1   conv1                    (x,)            {}
call_module    conv2   conv2                    (x,)            {}
call_function  add     <built-in function add>  (conv1, conv2)  {}
output         output  output                   (add,)          {}

==== Predecessors and successors of each node: ====
node: x
Predecessors:
Successors:
	 conv1 <class 'torch.fx.node.Node'>
	 conv2 <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv1
Predecessors:
	 x <class 'torch.fx.node.Node'>
Successors:
	 add <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv2
Predecessors:
	 x <class 'torch.fx.node.Node'>
Successors:
	 add <class 'torch.fx.node.Node'>
----------------------------------------------
node: add
Predecessors:
	 conv1 <class 'torch.fx.node.Node'>
	 conv2 <class 'torch.fx.node.Node'>
Successors:
	 output <class 'torch.fx.node.Node'>
----------------------------------------------
node: output
Predecessors:
	 add <class 'torch.fx.node.Node'>
Successors:
----------------------------------------------

opcode         name    target                   args      kwargs
-------------  ------  -----------------------  --------  --------
placeholder    x       x                        ()        {}
call_module    conv1   conv1                    (x,)      {}
call_module    conv2   conv2                    ()        {}
call_function  add     <built-in function add>  ()        {}
output         output  output                   (conv1,)  {}

==== Predecessors and successors of each node: ====
node: x
Predecessors:
Successors:
	 conv1 <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv1
Predecessors:
	 x <class 'torch.fx.node.Node'>
Successors:
	 output <class 'torch.fx.node.Node'>
----------------------------------------------
node: conv2
Predecessors:
Successors:
----------------------------------------------
node: add
Predecessors:
Successors:
----------------------------------------------
node: output
Predecessors:
	 conv1 <class 'torch.fx.node.Node'>
Successors:
----------------------------------------------

def forward(self, x):
    conv1 = self.conv1(x);  x = None
    return conv1

torch.Size([64, 8, 28, 28])