Repository: pytorch/ignite

Issue:
Bug in binary precision
Basic tests of binary precision seem to fail:
```python
precision = Precision(average=True)

y_pred = torch.rand(10, 1)
y = torch.randint(0, 2, size=(10,)).type(torch.LongTensor)

precision.update((y_pred, y))

np_y = y.numpy().ravel()
np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')

precision_score(np_y, np_y_pred), precision.compute()
```

@jasonkriss could you please confirm this ?


EDIT: Another failing test: https://github.com/pytorch/ignite/pull/333#issuecomment-442643530


Probably, the error is at binary to categorical mapping and counting class-0 similarly to class-1. But in binary case we should ignore class-0.

There is also input "binary or categorical" checking is missing if user tries to mix both in several updates.

Discussion:
anmolsjoshi: @vfdev-5 please see the following PR. If we stick with this method, it'll be easier to incorporate multilabel as well. 
vfdev-5: @anmolsjoshi thanks for the PR !
Refactoring huge `update` into `_check_shape`, `_check_type` definitly makes sense. I have an impression that `_check_shape`, `_check_type` are the same for Precision and Recall ? Maybe even Accuracy could benefit from one of them ?
anmolsjoshi: @vfdev-5 

> Refactoring huge update into _check_shape, _check_type definitly makes sense. I have an impression that _check_shape, _check_type are the same for Precision and Recall ? 

You are correct, they are the same for Precision and Recall. I took a note from how sklearn calculates precision and recall. 

> Maybe even Accuracy could benefit from one of them ?

Definitely! I can work on that later tonight. 
vfdev-5: Could you please post a reference on sklearn here ?
vfdev-5: Tell me when you are done with the code, I'll update the tests in the same way as for Accuracy.

anmolsjoshi: @vfdev-5 not sure what went wrong with the pytorch-nightly-cpu 2.7
anmolsjoshi: Here is the [sklearn reference](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py#L1094-L1316)

vfdev-5: @anmolsjoshi seems like that pytorch nightly is broken for 2.7
anmolsjoshi: > Maybe we could create a common class for Precision, Recall in order to not to copy these methods ?

That's an interesting idea. So a base class with a common `_check_type`, `_check_shape`? looks like we can make `compute` the same too. 
anmolsjoshi: @vfdev-5 I believe that I'm done with all the tests and code. Feel free to incorporate this into Accuracy.

> @anmolsjoshi seems like that pytorch nightly is broken for 2.7

what are next steps?
vfdev-5: Let's ignore this failing test for a moment, if this is pytorch problem, it will be solved soon. We can ask to be sure.

Next step is to refactor classes to avoid code copying.
anmolsjoshi: @vfdev-5 ok I'll start refactoring the classes. Did you want me to work on Accuracy as well?
vfdev-5: > Did you want me to work on Accuracy as well? 

@anmolsjoshi yes please :)
anmolsjoshi: @vfdev-5 I refactored the code, let me know if this works for you! 

_classification_support.py
- ClassificationSupport - this will be used in Accuracy, Precision, Recall - contains type check and shape check
- PrecisionRecallSupport - this will be used to calculate Precision/Recall - contains common update, reset, compute

precision.py
- Precision - uses PrecisionRecallSupport as a base class with self._precision_vs_recall True

recall.py
- Recall - uses PrecisionRecallSuport as a base class with self._precision_vs_recall False. 
vfdev-5: @anmolsjoshi I think we can create these classes directly in `accuracy.py`, `precision.py` files.
`ClassificationSupport` -> ~ `_BaseClassification`, leave it abstract (do not define `update`, we should not instantiate this class). 
`PrecisionRecallSupport` -> ~ `_(Base)PrecisionRecall(Support)`
I'm not fan of `_precision_vs_recall` as true/false. Maybe better a function to override ?
Initialization may be specific to Accurary, Precision, Recall, IMO should not be done in `ClassificationSupport` ?

anmolsjoshi: @vfdev-5 

> ClassificationSupport -> ~ _BaseClassification, leave it abstract (do not define update, we should not instantiate this class).

Agree, will update. 

>  I'm not fan of _precision_vs_recall as true/false. Maybe better a function to override ?
Could you explain that further? The calculation is entirely the same, the only difference is what type of positives to calculate i.e. positives of y_pred or y. precision_vs_recall might not be the cleanest way of doing this. 

> Initialization may be specific to Accurary, Precision, Recall, IMO should not be done in ClassificationSupport
Referring to [this comment](https://github.com/pytorch/ignite/pull/333#issuecomment-442970894), it might be best stick with threshold_function for binary calculation. I think initialization of _BaseClassification should stay the same, only thing that is added for Precision and Recall is the term average, which is introduced in _BasePrecisionRecallSupport (child of _BaseClassification).

>  think we can create these classes directly in accuracy.py, precision.py files. No need of _classification_support.py.

I'll move _BaseClassification to accuracy.py, and _BasePrecisionRecallSupport to precision.py, and will call it in recall.py as well.

Thoughts?

anmolsjoshi: @vfdev-5 Here is a summary of the most up to date commit:

- accuracy.py - contains _BaseClassification (still abstract) with improved binary shape check (last few commits had incorrect logic)

- recall.py - deleted

- precision.py - renamed to precision_recall.py

- precision_recall.py - contains _BasePrecisionRecallSupport (child of _BaseClassification and still abstract), Precision and Recall. _Base..Support contains two functions _calculate_correct and _sum_positives that both Precision and Recall use. Here's how:

```python
class Precision(_BasePrecisionRecallSupport):

...

    def update(self, output):
        correct, y_pred, y = self._calculate_correct(output)
        all_positives = y_pred.sum(dim=0)
        self._sum_positives(correct, all_positives)
``` 

- __init__.py - changed for updated Precision and Recall calls

- tests_accuracy.py - Contains your modified tests from Multilabel PR and include tests similar to test_precision/recall testing threshold_function, _check_type.

Please note that Binary Accuracy is now being calculated using torch.round, I think it is the preferred method due to inconsistent results in between PyTorch versions. Discussed in this [comment](https://github.com/pytorch/ignite/pull/333#issuecomment-442970894).
anmolsjoshi: @vfdev-5 all tests are now passing. I updated precision and recall so that they have there own unique updates. Let me know if you have any other issues! 
vfdev-5: @anmolsjoshi I'm analysing some corner cases: Precision binary input with average=False
```python
precision = Precision(average=False)

y_pred = torch.rand(10, 1)
y = torch.randint(0, 2, size=(10,))
precision.update((y_pred, y))

np_y = y.numpy()
np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
print(np_y)
print(np_y_pred)
precision_score(np_y, np_y_pred, average=None), precision.compute().numpy()
```
and it gives
```
[1 0 0 0 1 0 0 1 0 1]
[1 1 0 0 0 0 1 0 1 1]
(array([0.6, 0.4]), array(0.4, dtype=float32))
```
In `precision_score` when `average=None` it interprets this example as 2-classes case and the score is not averaged in the end.


vfdev-5: I left some comments in the code to make some changes concerning threshold function. I'm hesitating on the usage of threshold function as it is specific for binary case only.

Again, why it was decided to introduce `threshold_function`. There was a problem with binary to 2-classes mapping: `y_pred -> [1.0 - y_pred, y_pred]` which have implicit assumption that `y_pred` is between 0 and 1 and `torch.max` used as argmax gives different results for 0.4.1 and 1.0-preview.
Using `threshold_function` we handles the first limitation on `y_pred` range and we do not use torch max to compute the number of TP for the class 1 -> no more different results between torch versions. 

I have another objection agains `threshold_function`, the binarization of `y_pred` can be also done by `output_transform`. And as it is specific for binary case only, we can update the docs and require binary `y_pred` instead of logits, probas or whatever and apply a custom threshold operation. 

@jasonkriss any ideas on this ?
anmolsjoshi: @vfdev-5 the use of threshold_function is for binary at this time, but in the future will be used for multilabel cases. So it might be useful to keep it. 

You are correct, we'll have these issues in multiclass cases also, eg: [0.333, 0.333, 0.333]. Although its not incorrect what argmax selects, our tests can't reflect tests that involve such cases because these tests are used for multiple versions of PyTorch, and will fail dependent on what version it is tested on. 

I don't mind switching to torch.max method. But if we want people to rely on the output_transform, when we try to add multilabel cases, are we going to ask people to use output_transform as well? My thoughts are that we should use the same method for both binary and multilabel cases. Although I think we can use torch.max for multilabel cases as well. 

We have had a user point out issues with writing their own rounding code before, see [this comment](https://github.com/pytorch/ignite/issues/262#issuecomment-421625931).

Regarding this [comment](https://github.com/pytorch/ignite/pull/349#issuecomment-443457033), problem will be solved if we used torch.max method. We'll need to modify to include an if else statement in compute for binary cases (we can think of better methods). 

```python
if self._average:
    if self._type == 'binary':
        return results[1].item()
    else:
        return result.mean().item()
else:
        return result
```

It might be helpful to include a note about `1` being the primary class in binary cases. Just so that users don't have to look into the code to figure this out. 
anmolsjoshi: > sorry for the delay, haven't much time these days :(

@vfdev-5 no worries!

I agree with your approach and have removed threshold_function entirely. The following changes have been made. 

- Removed threshold_function from all metrics, all metrics now include Binary to Categorical Mapping
- All tests have been updated accordingly
- Binary case outputs Precision/Recall for 2 classes, when it times to compute average it only outputs the last class, if average False it returns score for both classes
- Added test for when average=None in sklearn scores, binary case
- Added test for binary cases when average=None
- I updated Precision/Recall tests similar to Accuracy to include random numbers instead of hard code values
- Removed constructor from all final metrics, Base classes handles everything
- Renamed _BasePrecisionRecallSupport to _BasePrecisionRecall

It seems that tests are passing! 

I wanted to remove [this test[(https://github.com/anmolsjoshi/ignite/blob/precisionbug/tests/ignite/metrics/test_precision.py#L75-L80) from both precision and recall. I don't think there is a case where one would average the results of Binary classes. Given that, it might be useful to include a line of code like this:

```python
if y_pred.shape[1] == 2:
    self._type = 'binary'
```

Let me know if you have any comments! 

vfdev-5: @anmolsjoshi I just looked over the PR, thanks for working on this, really good work especially with the tests !

A minor suggestion in `_Base` classes you propose two attributes
```
self._type
self._updated
```
I think we can have one `self._type`, at initialization it can be equal `None` and once specified it plays the same role as updated.
anmolsjoshi: > I think we can have only one self._type, at initialization it can be equal None and once specified it plays the same role as updated.

@vfdev-5 I agree, thanks for a thorough review!

I have pushed this change. 
anmolsjoshi: @vfdev-5 Made some additional changes.

- Updated warning on binary accuracy and categorical accuracy to DepreciationWarning
- Updated binary accuracy and categorical accuracy tests to catch warning on said metric instantiation 
- Added assert of self._type in accuracy, binary accuracy, categorical accuracy tests
anmolsjoshi: cc @jasonkriss 
anmolsjoshi: Updated docstring to mention that y_pred has to be probabilities, user should use output_transform. 
vfdev-5: @anmolsjoshi I think I badly explained the idea when I talked about to remove `threshold_function` and force user to binarize output with `output_transform`. Idea was to move `threshold_function` into `output_transform` and directly compute the metric without `binary -> categorical` mapping.

Another thing, I'm trying to list all cases and write all possible tests:

Case | y shape | y_pred | y**2 = y | y_pred**2=y_pred 
---|---|---|---|---
Binary a) | (N,) or (N, 1) | (N,) or (N, 1) | Yes | Yes 
Binary b) | (N, L) | (N, L) | Yes | Yes 
Binary c) | (N, H, W, ...) | (N, H, W, ...) | Yes | Yes 
Multiclass a) | (N,) or (N, 1) | (N, C) | No | No
Multiclass b) | (N, L) | (N, C, L) | No | No
Multiclass c) | (N, H, W, ...) | (N, C, H, W, ...) | No | No

The current implementation makes the following code ambigious:
```python
acc.update((torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor),
                    torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
```
so, possible interpretations:
- bad shapes in binary case
- correct data in multiclass case: target data has only 0, 1 classes among 0-4 classes

anmolsjoshi: @vfdev-5 could you explain why the code is ambiguous? 

Also could you give an example of the threshold function with output transform? I don’t think I understands. 
vfdev-5: > could you explain why the code is ambiguous?

the code 
```python
acc.update((torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor),
             torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
```
is now interpreted as multiclass. However I supposed to test it as wrong shapes in binary. But OK probably in a sequence of binary data, when this case is encountered it will raise the problem as multiclass instead of binary.

> Also could you give an example of the threshold function with output transform? I don’t think I understand

Something like this
```python
        def thresholded_output_transform(output):
            y_pred, y = output
            y_pred = torch.round(torch.softmax(y_pred, dim=1))
            return y_pred, y

        binary_accuracy = Accuracy(thresholded_output_transform)
```

anmolsjoshi: @vfdev-5 

> acc.update((torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor),
             torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))

Looking at this problem, y_pred.ndimension() == y.ndimension() + 1, currently that's how we check is something is binary or multiclass. And in the case shape[1] == 1, we squeeze y and y_pred to better decide if the data is multiclass or binary. 

Given the docs, if you want to pass an example that is a sequence of binary data it should be of the form, y=(batch_size, seq_len) and y_pred = (batch_size, num_classes=1, seq_len) or (batch_size, seq_len). The current example assumes each example of y_pred is of shape (1,5), while y is of shape (1,), thus is not setup of sequence classification of binary data. 

Given the current example you wrote up, for sequence classification if you're classifying each element of the sequence, each example of both y and y_pred should be of shape (1, 5). If the entire sequence has to be classified, then both y and y_pred should of shape (1,). 

Sorry if I'm not understanding the issue correctly.
anmolsjoshi: @vfdev-5 as for threshold, if you want to move that to output_transform, will we provide that as the default for Accuracy, Precision and Recall? 
vfdev-5: @anmolsjoshi actually I'm not sure that it is an issue with the previous example.

> We are not using the values of y and y_pred to decide if a problem is binary or multiclass. I think we shouldn't do that because not every batch of data will have the same number of classes, meaning you could get a batch of just 1 class. 

Correct

When I have more time I'll continue with other tests on `Accuracy` and others according to the table from the previous message. Important thing is `y_pred**2=y_pred` for binary case.


vfdev-5: > will we provide that as the default for Accuracy, Precision and Recall?

Did not get your point on this.


anmolsjoshi: How do you think we should include thresholded_output_transform in the current API. 

The code you provided above, will we include that as an example and remove the binary to categorical mapping? 
vfdev-5: > How do you think we should include thresholded_output_transform in the current API.

As in the previous example:
```python
def thresholded_output_transform(output):
     y_pred, y = output
     y_pred = torch.round(torch.softmax(y_pred, dim=1))
     return y_pred, y

binary_accuracy = Accuracy(output_transform=thresholded_output_transform)
```
As it is shown thresholding is integrated into `output_transform`.

> The code you provided above, will we include that as an example and remove the binary to categorical mapping?

Yes
anmolsjoshi: What are your thoughts on adding `thresholded_output_transform` to ignite._utils.py? So that users can access it. 
vfdev-5: > What are your thoughts on adding thresholded_output_transform to ignite._utils.py? So that users can access it.

For instance I'm not sure that this is necessary... Let's see
anmolsjoshi: @vfdev-5 the original binary_accuracy had torch.round implemented within update, see [here](https://github.com/jasonkriss/ignite/blob/5db45c2427322dfd63f5fe0865aed6a81c9bc178/ignite/metrics/binary_accuracy.py#L24). 

Is there a chance we might inconvenience people by having them use output_transform? I guess if the plan is to cut a release after this is merged, we can point this out. 
vfdev-5: @anmolsjoshi this is a good point ! Actually, just remarked that in the docs it was directly stated that `y_pred` should be between 0 and 1. I think we lost this when merged binary and categorical accuracies.
anmolsjoshi: Yes, that's probably my bad when I sent in PR #275, I missed including that docstring. 

How do you think we should proceed? Still include binary to categorical mapping? Or introduce this change in the release notes?
vfdev-5: @anmolsjoshi no problems with that. We'll take care of it this time :) I'm readjusting the tests now.

> How do you think we should proceed? Still include binary to categorical mapping? Or introduce this change in the release notes?

Thanks for asking! So, as we discussed #348 in we need to fix the bug and cut the release `0.1.2` that should be compatible with `0.1.1`. Thus, we should include `torch.round` in binary case and update the docs  saying that `y_pred` is between 0 and 1 (in binary case).

In another PR we can remove `torch.round` and propose user to apply binarization inside `output_transform`.
vfdev-5: I'll commit my modifications on `Accuracy` and I let you review it.
vfdev-5: @anmolsjoshi I did yet not finished the code and tests on precision. Seems binary case of (N, L) is not well coded. If you would like to continue, feel free (wont touch it at least for 7-8h :)
anmolsjoshi: @vfdev-5 thanks, this looks fantastic! I'll work on it
anmolsjoshi: @vfdev-5 all tests are passing, let me know what you think! 

Currently, there are warnings for Precision and Recall using sklearn for cases where number of predicted or actual positives is 0 for a specific class. If we want to test average, it might be best to ignore these warnings. 
vfdev-5: @anmolsjoshi thanks a lot ! That starts look much better than the previous version :) 
I'll check out the new version and make some local tests (I had a doubt on some combinations)
vfdev-5: @anmolsjoshi I updated the doc.

@alykhantejani @jasonkriss any comments, review please

Patch:
diff --git a/examples/contrib/mnist_with_tqdm_logger.py b/examples/contrib/mnist_with_tqdm_logger.py
new file mode 100644
index 00000000..eb557f83
--- /dev/null
+++ b/examples/contrib/mnist_with_tqdm_logger.py
@@ -0,0 +1,108 @@
+from argparse import ArgumentParser
+
+from torch import nn
+from torch.optim import SGD
+from torch.utils.data import DataLoader
+import torch
+import torch.nn.functional as F
+from torchvision.transforms import Compose, ToTensor, Normalize
+from torchvision.datasets import MNIST
+
+from ignite.engine import Events, create_supervised_trainer, create_supervised_evaluator
+from ignite.metrics import Accuracy, Loss, RunningAverage
+
+from ignite.contrib.handlers import ProgressBar
+
+
+class Net(nn.Module):
+    def __init__(self):
+        super(Net, self).__init__()
+        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
+        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
+        self.conv2_drop = nn.Dropout2d()
+        self.fc1 = nn.Linear(320, 50)
+        self.fc2 = nn.Linear(50, 10)
+
+    def forward(self, x):
+        x = F.relu(F.max_pool2d(self.conv1(x), 2))
+        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
+        x = x.view(-1, 320)
+        x = F.relu(self.fc1(x))
+        x = F.dropout(x, training=self.training)
+        x = self.fc2(x)
+        return F.log_softmax(x, dim=-1)
+
+
+def get_data_loaders(train_batch_size, val_batch_size):
+    data_transform = Compose([ToTensor(), Normalize((0.1307,), (0.3081,))])
+
+    train_loader = DataLoader(MNIST(download=True, root=".", transform=data_transform, train=True),
+                              batch_size=train_batch_size, shuffle=True)
+
+    val_loader = DataLoader(MNIST(download=False, root=".", transform=data_transform, train=False),
+                            batch_size=val_batch_size, shuffle=False)
+    return train_loader, val_loader
+
+
+def run(train_batch_size, val_batch_size, epochs, lr, momentum):
+    train_loader, val_loader = get_data_loaders(train_batch_size, val_batch_size)
+    model = Net()
+    device = 'cpu'
+
+    if torch.cuda.is_available():
+        device = 'cuda'
+
+    optimizer = SGD(model.parameters(), lr=lr, momentum=momentum)
+    trainer = create_supervised_trainer(model, optimizer, F.nll_loss, device=device)
+    evaluator = create_supervised_evaluator(model,
+                                            metrics={'accuracy': Accuracy(),
+                                                     'nll': Loss(F.nll_loss)},
+                                            device=device)
+
+    RunningAverage(output_transform=lambda x: x).attach(trainer, 'loss')
+
+    pbar = ProgressBar(persist=True)
+    pbar.attach(trainer, ['loss'])
+
+    @trainer.on(Events.EPOCH_COMPLETED)
+    def log_training_results(engine):
+        evaluator.run(train_loader)
+        metrics = evaluator.state.metrics
+        avg_accuracy = metrics['accuracy']
+        avg_nll = metrics['nll']
+        pbar.log_message(
+            "Training Results - Epoch: {}  Avg accuracy: {:.2f} Avg loss: {:.2f}"
+            .format(engine.state.epoch, avg_accuracy, avg_nll)
+        )
+
+    @trainer.on(Events.EPOCH_COMPLETED)
+    def log_validation_results(engine):
+        evaluator.run(val_loader)
+        metrics = evaluator.state.metrics
+        avg_accuracy = metrics['accuracy']
+        avg_nll = metrics['nll']
+        pbar.log_message(
+            "Validation Results - Epoch: {}  Avg accuracy: {:.2f} Avg loss: {:.2f}"
+            .format(engine.state.epoch, avg_accuracy, avg_nll))
+
+        pbar.n = pbar.last_print_n = 0
+
+    trainer.run(train_loader, max_epochs=epochs)
+
+
+if __name__ == "__main__":
+    parser = ArgumentParser()
+    parser.add_argument('--batch_size', type=int, default=64,
+                        help='input batch size for training (default: 64)')
+    parser.add_argument('--val_batch_size', type=int, default=1000,
+                        help='input batch size for validation (default: 1000)')
+    parser.add_argument('--epochs', type=int, default=10,
+                        help='number of epochs to train (default: 10)')
+    parser.add_argument('--lr', type=float, default=0.01,
+                        help='learning rate (default: 0.01)')
+    parser.add_argument('--momentum', type=float, default=0.5,
+                        help='SGD momentum (default: 0.5)')
+
+    args = parser.parse_args()
+
+    run(args.batch_size, args.val_batch_size, args.epochs, args.lr, args.momentum)
diff --git a/examples/mnist/README.md b/examples/mnist/README.md
index db421989..f9fc0c80 100644
--- a/examples/mnist/README.md
+++ b/examples/mnist/README.md
@@ -2,6 +2,14 @@
 
 ported from [pytorch-examples](https://github.com/pytorch/examples/tree/master/mnist)
 
+#### Requirements:
+
+- [torchvision](https://github.com/pytorch/vision/): `pip install torchvision`
+- [tqdm](https://github.com/tqdm/tqdm/): `pip install tqdm`
+
+#### Usage:
+
+Run the example:
 ```
 python mnist.py
 ```
@@ -12,6 +20,7 @@ MNIST example with training and validation monitoring using TensorboardX and Ten
 
 #### Requirements:
 
+- [torchvision](https://github.com/pytorch/vision/): `pip install torchvision`
 - [TensorboardX](https://github.com/lanpa/tensorboard-pytorch): `pip install tensorboardX`
 - Tensorboard: `pip install tensorflow` (or just install tensorboard without the rest of tensorflow)
 
@@ -33,6 +42,7 @@ MNIST example with training and validation monitoring using Visdom
 
 #### Requirements:
 
+- [torchvision](https://github.com/pytorch/vision/): `pip install torchvision`
 - [Visdom](https://github.com/facebookresearch/visdom): `pip install visdom`
 
 #### Usage:
diff --git a/ignite/contrib/handlers/tqdm_logger.py b/ignite/contrib/handlers/tqdm_logger.py
index 4116efd4..4ff10c95 100644
--- a/ignite/contrib/handlers/tqdm_logger.py
+++ b/ignite/contrib/handlers/tqdm_logger.py
@@ -23,6 +23,7 @@ class ProgressBar:
               rate_inv, rate_inv_fmt, elapsed, remaining, desc, postfix.
             Note that a trailing ": " is automatically removed after {desc}
             if the latter is empty.
+        **tqdm_kwargs: kwargs passed to tqdm progress bar
 
     Examples:
 
@@ -62,16 +63,20 @@ class ProgressBar:
     """
 
     def __init__(self, persist=False,
-                 bar_format='{desc}[{n_fmt}/{total_fmt}] {percentage:3.0f}%|{bar}{postfix} [{elapsed}<{remaining}]'):
+                 bar_format='{desc}[{n_fmt}/{total_fmt}] {percentage:3.0f}%|{bar}{postfix} [{elapsed}<{remaining}]',
+                 **tqdm_kwargs):
         self.pbar = None
         self.persist = persist
         self.bar_format = bar_format
+        self.tqdm_kwargs = tqdm_kwargs
 
     def _reset(self, engine):
         self.pbar = tqdm(
             total=len(engine.state.dataloader),
             leave=self.persist,
-            bar_format=self.bar_format)
+            bar_format=self.bar_format,
+            **self.tqdm_kwargs
+        )
 
     def _close(self, engine):
         self.pbar.close()
@@ -81,7 +86,8 @@ class ProgressBar:
         if self.pbar is None:
             self._reset(engine)
 
-        self.pbar.set_description('Epoch [{}/{}]'.format(engine.state.epoch, engine.state.max_epochs))
+        if 'desc' not in self.tqdm_kwargs:
+            self.pbar.set_description('Epoch [{}/{}]'.format(engine.state.epoch, engine.state.max_epochs))
 
         metrics = {}
         if metric_names is not None:
@@ -132,5 +138,5 @@ class ProgressBar:
             raise TypeError("output_transform should be a function, got {} instead"
                             .format(type(output_transform)))
 
-        engine.add_event_handler(Events.EPOCH_COMPLETED, self._close)
         engine.add_event_handler(Events.ITERATION_COMPLETED, self._update, metric_names, output_transform)
+        engine.add_event_handler(Events.EPOCH_COMPLETED, self._close)
diff --git a/ignite/contrib/metrics/__init__.py b/ignite/contrib/metrics/__init__.py
index 199c0f7f..a701ad57 100644
--- a/ignite/contrib/metrics/__init__.py
+++ b/ignite/contrib/metrics/__init__.py
@@ -1,2 +1,3 @@
 from ignite.contrib.metrics.average_precision import AveragePrecision
 from ignite.contrib.metrics.roc_auc import ROC_AUC
+import ignite.contrib.metrics.regression
diff --git a/ignite/contrib/metrics/average_precision.py b/ignite/contrib/metrics/average_precision.py
index c5556b00..6d63c0ee 100644
--- a/ignite/contrib/metrics/average_precision.py
+++ b/ignite/contrib/metrics/average_precision.py
@@ -1,14 +1,13 @@
 from functools import partial
-
-try:
-    from sklearn.metrics import average_precision_score
-except ImportError:
-    raise RuntimeError("This contrib module requires sklearn to be installed")
-
 from ignite.metrics import EpochMetric
 
 
 def average_precision_compute_fn(y_preds, y_targets, activation=None):
+    try:
+        from sklearn.metrics import average_precision_score
+    except ImportError:
+        raise RuntimeError("This contrib module requires sklearn to be installed")
+
     y_true = y_targets.numpy()
     if activation is not None:
         y_preds = activation(y_preds)
diff --git a/ignite/contrib/metrics/roc_auc.py b/ignite/contrib/metrics/roc_auc.py
index 8bb6d661..23cbb35c 100644
--- a/ignite/contrib/metrics/roc_auc.py
+++ b/ignite/contrib/metrics/roc_auc.py
@@ -1,14 +1,13 @@
 from functools import partial
-
-try:
-    from sklearn.metrics import roc_auc_score
-except ImportError:
-    raise RuntimeError("This contrib module requires sklearn to be installed")
-
 from ignite.metrics import EpochMetric
 
 
 def roc_auc_compute_fn(y_preds, y_targets, activation=None):
+    try:
+        from sklearn.metrics import roc_auc_score
+    except ImportError:
+        raise RuntimeError("This contrib module requires sklearn to be installed")
+
     y_true = y_targets.numpy()
     if activation is not None:
         y_preds = activation(y_preds)
diff --git a/ignite/metrics/accuracy.py b/ignite/metrics/accuracy.py
index cd5eddad..13dd7fd4 100644
--- a/ignite/metrics/accuracy.py
+++ b/ignite/metrics/accuracy.py
@@ -6,48 +6,95 @@ from ignite.metrics.metric import Metric
 from ignite.exceptions import NotComputableError
 
 
-class Accuracy(Metric):
-    """
-    Calculates the accuracy.
+class _BaseClassification(Metric):
 
-    - `update` must receive output of the form `(y_pred, y)`.
-    - `y_pred` must be in the following shape (batch_size, num_categories, ...) or (batch_size, ...)
-    - `y` must be in the following shape (batch_size, ...)
-    """
-    def reset(self):
-        self._num_correct = 0
-        self._num_examples = 0
+    def __init__(self, output_transform=lambda x: x):
+        self._type = None
+        super(_BaseClassification, self).__init__(output_transform=output_transform)
 
-    def update(self, output):
+    def _check_shape(self, output):
         y_pred, y = output
 
-        if not (y.ndimension() == y_pred.ndimension() or y.ndimension() + 1 == y_pred.ndimension()):
-            raise ValueError("y must have shape of (batch_size, ...) and y_pred "
-                             "must have shape of (batch_size, num_classes, ...) or (batch_size, ...).")
-
         if y.ndimension() > 1 and y.shape[1] == 1:
+            # (N, 1, ...) -> (N, ...)
             y = y.squeeze(dim=1)
 
         if y_pred.ndimension() > 1 and y_pred.shape[1] == 1:
+            # (N, 1, ...) -> (N, ...)
             y_pred = y_pred.squeeze(dim=1)
 
+        if not (y.ndimension() == y_pred.ndimension() or y.ndimension() + 1 == y_pred.ndimension()):
+            raise ValueError("y must have shape of (batch_size, ...) and y_pred must have "
+                             "shape of (batch_size, num_categories, ...) or (batch_size, ...), "
+                             "but given {} vs {}".format(y.shape, y_pred.shape))
+
         y_shape = y.shape
         y_pred_shape = y_pred.shape
 
         if y.ndimension() + 1 == y_pred.ndimension():
-            y_pred_shape = (y_pred_shape[0], ) + y_pred_shape[2:]
+            y_pred_shape = (y_pred_shape[0],) + y_pred_shape[2:]
 
         if not (y_shape == y_pred_shape):
             raise ValueError("y and y_pred must have compatible shapes.")
 
-        if y_pred.ndimension() == y.ndimension():
-            # Maps Binary Case to Categorical Case with 2 classes
-            y_pred = y_pred.unsqueeze(dim=1)
-            y_pred = torch.cat([1.0 - y_pred, y_pred], dim=1)
+        return y_pred, y
 
-        indices = torch.max(y_pred, dim=1)[1]
-        correct = torch.eq(indices, y).view(-1)
+    def _check_type(self, output):
+        y_pred, y = output
+
+        if y.ndimension() + 1 == y_pred.ndimension():
+            update_type = "multiclass"
+        elif y.ndimension() == y_pred.ndimension():
+            update_type = "binary"
+            if not torch.equal(y, y ** 2):
+                raise ValueError("For binary cases, y must be comprised of 0's and 1's.")
+            # TODO: Uncomment the following after 0.1.2 release
+            # if not torch.equal(y_pred, y_pred ** 2):
+            #     raise ValueError("For binary cases, y_pred must be comprised of 0's and 1's.")
+        else:
+            raise RuntimeError("Invalid shapes of y (shape={}) and y_pred (shape={}), check documentation"
+                               " for expected shapes of y and y_pred.".format(y.shape, y_pred.shape))
+        if self._type is None:
+            self._type = update_type
+        else:
+            if self._type != update_type:
+                raise RuntimeError("update_type has changed from {} to {}.".format(self._type, update_type))
+
+
+class Accuracy(_BaseClassification):
+    """
+    Calculates the accuracy for binary and multiclass data
+    - `update` must receive output of the form `(y_pred, y)`.
+    - `y_pred` must be in the following shape (batch_size, num_categories, ...) or (batch_size, ...)
+    - `y` must be in the following shape (batch_size, ...)
+
+    In binary case, when `y` has 0 or 1 values, the elements of `y_pred` must be between 0 and 1.
+    """
+    # TODO: Include the following into the docstring after 0.1.2 release
+    # .. code-block:: python
+    #
+    #     def thresholded_output_transform(output):
+    #         y_pred, y = output
+    #         y_pred = torch.round(y_pred)
+    #         return y_pred, y
+    #
+    #     binary_accuracy = Accuracy(thresholded_output_transform)
+
+    def reset(self):
+        self._num_correct = 0
+        self._num_examples = 0
+
+    def update(self, output):
+
+        y_pred, y = self._check_shape(output)
+        self._check_type((y_pred, y))
 
+        if self._type == "binary":
+            indices = torch.round(y_pred).type(y.type())
+        elif self._type == "multiclass":
+            indices = torch.max(y_pred, dim=1)[1]
+
+        correct = torch.eq(indices, y).view(-1)
         self._num_correct += torch.sum(correct).item()
         self._num_examples += correct.shape[0]
 
diff --git a/ignite/metrics/binary_accuracy.py b/ignite/metrics/binary_accuracy.py
index d0086371..8196a7fc 100644
--- a/ignite/metrics/binary_accuracy.py
+++ b/ignite/metrics/binary_accuracy.py
@@ -10,5 +10,5 @@ class BinaryAccuracy(Accuracy):
     """
     def __init__(self, *args, **kwargs):
         warnings.warn("The use of ignite.metrics.BinaryAccuracy is deprecated, it will be "
-                      "removed in 0.1.2. Please use ignite.metrics.Accuracy instead.")
+                      "removed in 0.2.0. Please use ignite.metrics.Accuracy instead.", DeprecationWarning)
         super(Accuracy, self).__init__(*args, **kwargs)
diff --git a/ignite/metrics/categorical_accuracy.py b/ignite/metrics/categorical_accuracy.py
index 184176c1..2e381fd3 100644
--- a/ignite/metrics/categorical_accuracy.py
+++ b/ignite/metrics/categorical_accuracy.py
@@ -10,5 +10,5 @@ class CategoricalAccuracy(Accuracy):
     """
     def __init__(self, *args, **kwargs):
         warnings.warn("The use of ignite.metrics.CategoricalAccuracy is deprecated, it will be "
-                      "removed in 0.1.2. Please use ignite.metrics.Accuracy instead.")
+                      "removed in 0.2.0. Please use ignite.metrics.Accuracy instead.", DeprecationWarning)
         super(Accuracy, self).__init__(*args, **kwargs)
diff --git a/ignite/metrics/precision.py b/ignite/metrics/precision.py
index a1bbdbe6..acb3d4cb 100644
--- a/ignite/metrics/precision.py
+++ b/ignite/metrics/precision.py
@@ -2,63 +2,66 @@ from __future__ import division
 
 import torch
 
-from ignite.metrics.metric import Metric
+from ignite.metrics.accuracy import _BaseClassification
 from ignite.exceptions import NotComputableError
 from ignite._utils import to_onehot
 
 
-class Precision(Metric):
-    """
-    Calculates precision.
-
-    - `update` must receive output of the form `(y_pred, y)`.
+class _BasePrecisionRecall(_BaseClassification):
 
-    If `average` is True, returns the unweighted average across all classes.
-    Otherwise, returns a tensor with the precision for each class.
-    """
-    def __init__(self, average=False, output_transform=lambda x: x):
-        super(Precision, self).__init__(output_transform)
+    def __init__(self, output_transform=lambda x: x, average=False):
         self._average = average
+        super(_BasePrecisionRecall, self).__init__(output_transform=output_transform)
 
     def reset(self):
-        self._all_positives = None
-        self._true_positives = None
+        self._true_positives = 0
+        self._positives = 0
 
-    def update(self, output):
-        y_pred, y = output
-        dtype = y_pred.type()
+    def compute(self):
+        if not isinstance(self._positives, torch.Tensor):
+            raise NotComputableError("{} must have at least one example before"
+                                     " it can be computed".format(self.__class__.__name__))
 
-        if not (y.ndimension() == y_pred.ndimension() or y.ndimension() + 1 == y_pred.ndimension()):
-            raise ValueError("y must have shape of (batch_size, ...) and y_pred "
-                             "must have shape of (batch_size, num_classes, ...) or (batch_size, ...).")
+        result = self._true_positives / self._positives
+        result[result != result] = 0.0
+        if self._average:
+            return result.mean().item()
+        else:
+            return result
 
-        if y.ndimension() > 1 and y.shape[1] == 1:
-            y = y.squeeze(dim=1)
 
-        if y_pred.ndimension() > 1 and y_pred.shape[1] == 1:
-            y_pred = y_pred.squeeze(dim=1)
+class Precision(_BasePrecisionRecall):
+    """
+    Calculates precision for binary and multiclass data
+    - `update` must receive output of the form `(y_pred, y)`.
+    - `y_pred` must be in the following shape (batch_size, num_categories, ...) or (batch_size, ...)
+    - `y` must be in the following shape (batch_size, ...)
 
-        y_shape = y.shape
-        y_pred_shape = y_pred.shape
+    In binary case, when `y` has 0 or 1 values, the elements of `y_pred` must be between 0 and 1. Precision is
+    computed over positive class, assumed to be 1.
 
-        if y.ndimension() + 1 == y_pred.ndimension():
-            y_pred_shape = (y_pred_shape[0],) + y_pred_shape[2:]
+    Args:
+        average (bool, optional): if True, precision is computed as the unweighted average (across all classes
+            in multiclass case), otherwise, returns a tensor with the precision (for each class in multiclass case).
 
-        if not (y_shape == y_pred_shape):
-            raise ValueError("y and y_pred must have compatible shapes.")
+    """
+    def update(self, output):
+        y_pred, y = self._check_shape(output)
+        self._check_type((y_pred, y))
 
-        if y_pred.ndimension() == y.ndimension():
-            # Maps Binary Case to Categorical Case with 2 classes
-            y_pred = y_pred.unsqueeze(dim=1)
-            y_pred = torch.cat([1.0 - y_pred, y_pred], dim=1)
+        dtype = y_pred.type()
 
-        y = to_onehot(y.view(-1), num_classes=y_pred.size(1))
-        indices = torch.max(y_pred, dim=1)[1].view(-1)
-        y_pred = to_onehot(indices, num_classes=y_pred.size(1))
+        if self._type == "binary":
+            y_pred = torch.round(y_pred).view(-1)
+            y = y.view(-1)
+        elif self._type == "multiclass":
+            num_classes = y_pred.size(1)
+            y = to_onehot(y.view(-1), num_classes=num_classes)
+            indices = torch.max(y_pred, dim=1)[1].view(-1)
+            y_pred = to_onehot(indices, num_classes=num_classes)
 
         y_pred = y_pred.type(dtype)
         y = y.type(dtype)
-
         correct = y * y_pred
         all_positives = y_pred.sum(dim=0)
 
@@ -66,19 +69,6 @@ class Precision(Metric):
             true_positives = torch.zeros_like(all_positives)
         else:
             true_positives = correct.sum(dim=0)
-        if self._all_positives is None:
-            self._all_positives = all_positives
-            self._true_positives = true_positives
-        else:
-            self._all_positives += all_positives
-            self._true_positives += true_positives
 
-    def compute(self):
-        if self._all_positives is None:
-            raise NotComputableError('Precision must have at least one example before it can be computed')
-        result = self._true_positives / self._all_positives
-        result[result != result] = 0.0
-        if self._average:
-            return result.mean().item()
-        else:
-            return result
+        self._true_positives += true_positives
+        self._positives += all_positives
diff --git a/ignite/metrics/recall.py b/ignite/metrics/recall.py
index a8d1d640..b0dd6fa6 100644
--- a/ignite/metrics/recall.py
+++ b/ignite/metrics/recall.py
@@ -1,85 +1,48 @@
-from __future__ import division
-
 import torch
-
-from ignite.metrics.metric import Metric
-from ignite.exceptions import NotComputableError
+from ignite.metrics.precision import _BasePrecisionRecall
 from ignite._utils import to_onehot
 
 
-class Recall(Metric):
+class Recall(_BasePrecisionRecall):
     """
-    Calculates recall.
-
+    Calculates recall for binary and multiclass data
     - `update` must receive output of the form `(y_pred, y)`.
+    - `y_pred` must be in the following shape (batch_size, num_categories, ...) or (batch_size, ...)
+    - `y` must be in the following shape (batch_size, ...)
 
-    If `average` is True, returns the unweighted average across all classes.
-    Otherwise, returns a tensor with the recall for each class.
-    """
+    In binary case, when `y` has 0 or 1 values, the elements of `y_pred` must be between 0 and 1. Recall is
+    computed over positive class, assumed to be 1.
 
-    def __init__(self, average=False, output_transform=lambda x: x):
-        super(Recall, self).__init__(output_transform)
-        self._average = average
+    Args:
+        average (bool, optional): if True, recall is computed as the unweighted average (across all classes
+            in multiclass case), otherwise, returns a tensor with the recall (for each class in multiclass case).
 
-    def reset(self):
-        self._actual = None
-        self._true_positives = None
+    """
 
     def update(self, output):
-        y_pred, y = output
-        dtype = y_pred.type()
-
-        if not (y.ndimension() == y_pred.ndimension() or y.ndimension() + 1 == y_pred.ndimension()):
-            raise ValueError("y must have shape of (batch_size, ...) and y_pred "
-                             "must have shape of (batch_size, num_classes, ...) or (batch_size, ...).")
-
-        if y.ndimension() > 1 and y.shape[1] == 1:
-            y = y.squeeze(dim=1)
+        y_pred, y = self._check_shape(output)
+        self._check_type((y_pred, y))
 
-        if y_pred.ndimension() > 1 and y_pred.shape[1] == 1:
-            y_pred = y_pred.squeeze(dim=1)
-
-        y_shape = y.shape
-        y_pred_shape = y_pred.shape
-
-        if y.ndimension() + 1 == y_pred.ndimension():
-            y_pred_shape = (y_pred_shape[0],) + y_pred_shape[2:]
-
-        if not (y_shape == y_pred_shape):
-            raise ValueError("y and y_pred must have compatible shapes.")
-
-        if y_pred.ndimension() == y.ndimension():
-            # Maps Binary Case to Categorical Case with 2 classes
-            y_pred = y_pred.unsqueeze(dim=1)
-            y_pred = torch.cat([1.0 - y_pred, y_pred], dim=1)
+        dtype = y_pred.type()
 
-        y = to_onehot(y.view(-1), num_classes=y_pred.size(1))
-        indices = torch.max(y_pred, dim=1)[1].view(-1)
-        y_pred = to_onehot(indices, num_classes=y_pred.size(1))
+        if self._type == "binary":
+            y_pred = torch.round(y_pred).view(-1)
+            y = y.view(-1)
+        elif self._type == "multiclass":
+            num_classes = y_pred.size(1)
+            y = to_onehot(y.view(-1), num_classes=num_classes)
+            indices = torch.max(y_pred, dim=1)[1].view(-1)
+            y_pred = to_onehot(indices, num_classes=num_classes)
 
         y_pred = y_pred.type(dtype)
         y = y.type(dtype)
-
         correct = y * y_pred
-        actual = y.sum(dim=0)
+        actual_positives = y.sum(dim=0)
 
         if correct.sum() == 0:
-            true_positives = torch.zeros_like(actual)
+            true_positives = torch.zeros_like(actual_positives)
         else:
             true_positives = correct.sum(dim=0)
-        if self._actual is None:
-            self._actual = actual
-            self._true_positives = true_positives
-        else:
-            self._actual += actual
-            self._true_positives += true_positives
 
-    def compute(self):
-        if self._actual is None:
-            raise NotComputableError('Recall must have at least one example before it can be computed')
-        result = self._true_positives / self._actual
-        result[result != result] = 0.0
-        if self._average:
-            return result.mean().item()
-        else:
-            return result
+        self._true_positives += true_positives
+        self._positives += actual_positives


Test patch:
diff --git a/tests/ignite/contrib/handlers/test_pbar.py b/tests/ignite/contrib/handlers/test_pbar.py
index 8ba55060..980663a6 100644
--- a/tests/ignite/contrib/handlers/test_pbar.py
+++ b/tests/ignite/contrib/handlers/test_pbar.py
@@ -4,7 +4,7 @@ import pytest
 import torch
 
 from ignite.contrib.handlers import ProgressBar
-from ignite.engine import Engine
+from ignite.engine import Engine, Events
 from ignite.metrics import Accuracy
 
 
@@ -131,3 +131,20 @@ def test_pbar_with_scalar_output(capsys):
     err = list(filter(None, err))
     expected = u'Epoch [2/2]: [1/2]  50%|█████     , output=1.00e+00 [00:00<00:00]'
     assert err[-1] == expected
+
+
+def test_pbar_with_tqdm_kwargs(capsys):
+    n_epochs = 10
+    loader = [1, 2, 3, 4, 5]
+    engine = Engine(update_fn)
+
+    pbar = ProgressBar(desc="My description: ")
+    pbar.attach(engine, output_transform=lambda x: x)
+    engine.run(loader, max_epochs=n_epochs)
+
+    captured = capsys.readouterr()
+    err = captured.err.split('\r')
+    err = list(map(lambda x: x.strip(), err))
+    err = list(filter(None, err))
+    expected = u'My description: [4/5]  80%|████████  , output=1.00e+00 [00:00<00:00]'.format()
+    assert err[-1] == expected
diff --git a/tests/ignite/metrics/test_accuracy.py b/tests/ignite/metrics/test_accuracy.py
index af85a4a0..515c9a40 100644
--- a/tests/ignite/metrics/test_accuracy.py
+++ b/tests/ignite/metrics/test_accuracy.py
@@ -5,115 +5,277 @@ import torch
 from sklearn.metrics import accuracy_score
 
 
-def test_zero_div():
+torch.manual_seed(12)
+
+
+def test_no_update():
     acc = Accuracy()
     with pytest.raises(NotComputableError):
         acc.compute()
 
 
-def test_binary_compute():
+def test__check_shape():
     acc = Accuracy()
 
-    y_pred = torch.sigmoid(torch.rand(4, 1))
-    y = torch.ones(4).type(torch.LongTensor)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
+    # Check squeezed dimensions
+    y_pred, y = acc._check_shape((torch.randint(0, 2, size=(10, 1, 5, 6)).type(torch.LongTensor),
+                                  torch.randint(0, 2, size=(10, 5, 6)).type(torch.LongTensor)))
+    assert y_pred.shape == (10, 5, 6)
+    assert y.shape == (10, 5, 6)
+
+    y_pred, y = acc._check_shape((torch.randint(0, 2, size=(10, 5, 6)).type(torch.LongTensor),
+                                  torch.randint(0, 2, size=(10, 1, 5, 6)).type(torch.LongTensor)))
+    assert y_pred.shape == (10, 5, 6)
+    assert y.shape == (10, 5, 6)
+
+
+def test_binary_wrong_inputs():
+    acc = Accuracy()
+
+    with pytest.raises(ValueError):
+        # y has not only 0 or 1 values
+        acc.update((torch.randint(0, 2, size=(10,)).type(torch.LongTensor),
+                    torch.arange(0, 10).type(torch.LongTensor)))
+
+    # TODO: Uncomment the following after 0.1.2 release
+    # with pytest.raises(ValueError):
+    #     # y_pred values are not thresholded to 0, 1 values
+    #     acc.update((torch.rand(10, 1),
+    #                 torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
+
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        acc.update((torch.randint(0, 2, size=(10,)).type(torch.LongTensor),
+                    torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor)))
+
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        acc.update((torch.randint(0, 2, size=(10, 5, 6)).type(torch.LongTensor),
+                    torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
+
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        acc.update((torch.randint(0, 2, size=(10,)).type(torch.LongTensor),
+                    torch.randint(0, 2, size=(10, 5, 6)).type(torch.LongTensor)))
+
+
+def test_binary_input_N():
+    # Binary accuracy on input of shape (N, 1) or (N, )
+    acc = Accuracy()
+
+    # TODO: y_pred should be binary after 0.1.2 release
+    # y_pred = torch.randint(0, 2, size=(10, 1)).type(torch.LongTensor)
+    y_pred = torch.rand(10, 1)
+    y = torch.randint(0, 2, size=(10,)).type(torch.LongTensor)
     acc.update((y_pred, y))
+    np_y = y.numpy().ravel()
+    # np_y_pred = y_pred.numpy().ravel()
+    np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+    assert acc._type == 'binary'
     assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.data.numpy(), indices.data.numpy()) == pytest.approx(acc.compute())
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
     acc.reset()
-    y_pred = torch.sigmoid(torch.rand(4))
-    y = torch.ones(4).type(torch.LongTensor)
-    y_pred = y_pred.unsqueeze(1)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
+    # TODO: y_pred should be binary after 0.1.2 release
+    # y_pred = torch.randint(0, 2, size=(10, )).type(torch.LongTensor)
+    y_pred = torch.rand(10)
+    y = torch.randint(0, 2, size=(10,)).type(torch.LongTensor)
+    acc.update((y_pred, y))
+    np_y = y.numpy().ravel()
+    # np_y_pred = y_pred.numpy().ravel()
+    np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+    assert acc._type == 'binary'
+    assert isinstance(acc.compute(), float)
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
+
+
+def test_binary_input_NL():
+    # Binary accuracy on input of shape (N, L)
+    acc = Accuracy()
+
+    # TODO: y_pred should be binary after 0.1.2 release
+    # y_pred = torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor)
+    y_pred = torch.rand(10, 5)
+    y = torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor)
+    acc.update((y_pred, y))
+    np_y = y.numpy().ravel()
+    # np_y_pred = y_pred.numpy().ravel()
+    np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+    assert acc._type == 'binary'
+    assert isinstance(acc.compute(), float)
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
+
+    acc.reset()
+    # TODO: y_pred should be binary after 0.1.2 release
+    # y_pred = torch.randint(0, 2, size=(10, 1, 5)).type(torch.LongTensor)
+    y_pred = torch.rand(10, 1, 5)
+    y = torch.randint(0, 2, size=(10, 1, 5)).type(torch.LongTensor)
     acc.update((y_pred, y))
+    np_y = y.numpy().ravel()
+    # np_y_pred = y_pred.numpy().ravel()
+    np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+    assert acc._type == 'binary'
     assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.data.numpy(), indices.data.numpy()) == pytest.approx(acc.compute())
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
 
-def test_compute_batch_images():
+def test_binary_input_NHW():
+    # Binary accuracy on input of shape (N, H, W, ...)
     acc = Accuracy()
 
-    y_pred = torch.sigmoid(torch.rand(1, 2, 2))
-    y = torch.ones(1, 2, 2).type(torch.LongTensor)
-    y_pred = y_pred.unsqueeze(1)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
+    # TODO: y_pred should be binary after 0.1.2 release
+    # y_pred = torch.randint(0, 2, size=(4, 12, 10)).type(torch.LongTensor)
+    y_pred = torch.rand(4, 12, 10)
+    y = torch.randint(0, 2, size=(4, 12, 10)).type(torch.LongTensor)
     acc.update((y_pred, y))
+    np_y = y.numpy().ravel()
+    # np_y_pred = y_pred.numpy().ravel()
+    np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+    assert acc._type == 'binary'
     assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
     acc.reset()
-    y_pred = torch.sigmoid(torch.rand(2, 1, 2, 2))
-    y = torch.ones(2, 2, 2).type(torch.LongTensor)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
+    # TODO: y_pred should be binary after 0.1.2 release
+    # y_pred = torch.randint(0, 2, size=(4, 1, 12, 10)).type(torch.LongTensor)
+    y_pred = torch.rand(4, 1, 12, 10)
+    y = torch.randint(0, 2, size=(4, 1, 12, 10)).type(torch.LongTensor)
     acc.update((y_pred, y))
+    np_y = y.numpy().ravel()
+    # np_y_pred = y_pred.numpy().ravel()
+    np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+    assert acc._type == 'binary'
     assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
     acc.reset()
-    y_pred = torch.sigmoid(torch.rand(2, 1, 2, 2))
-    y = torch.ones(2, 1, 2, 2).type(torch.LongTensor)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
+    # TODO: y_pred should be binary after 0.1.2 release
+    # y_pred = torch.randint(0, 2, size=(4, 12, 10, 8)).type(torch.LongTensor)
+    y_pred = torch.rand(4, 12, 10, 8)
+    y = torch.randint(0, 2, size=(4, 12, 10, 8)).type(torch.LongTensor)
     acc.update((y_pred, y))
+    np_y = y.numpy().ravel()
+    # np_y_pred = y_pred.numpy().ravel()
+    np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+    assert acc._type == 'binary'
     assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
 
-def test_categorical_compute():
+def test_multiclass_wrong_inputs():
     acc = Accuracy()
 
-    y_pred = torch.softmax(torch.rand(4, 4), dim=1)
-    y = torch.ones(4).type(torch.LongTensor)
-    indices = torch.max(y_pred, dim=1)[1]
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        acc.update((torch.rand(10, 5, 4),
+                    torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
+
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        acc.update((torch.rand(10, 5, 6),
+                    torch.randint(0, 5, size=(10, 5)).type(torch.LongTensor)))
+
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        acc.update((torch.rand(10),
+                    torch.randint(0, 5, size=(10, 5, 6)).type(torch.LongTensor)))
+
+
+def test_multiclass_input_N():
+    # Multiclass input data of shape (N, ) and (N, C)
+
+    acc = Accuracy()
+
+    y_pred = torch.rand(10, 4)
+    y = torch.randint(0, 4, size=(10,)).type(torch.LongTensor)
+    acc.update((y_pred, y))
+    np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+    np_y = y.numpy().ravel()
+    assert acc._type == 'multiclass'
+    assert isinstance(acc.compute(), float)
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
+
+    acc.reset()
+    y_pred = torch.rand(4, 10)
+    y = torch.randint(0, 10, size=(4, 1)).type(torch.LongTensor)
     acc.update((y_pred, y))
+    np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+    np_y = y.numpy().ravel()
+    assert acc._type == 'multiclass'
     assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
+    # 2-classes
     acc.reset()
-    y_pred = torch.softmax(torch.rand(2, 2), dim=1)
-    y = torch.ones(2).type(torch.LongTensor)
-    indices = torch.max(y_pred, dim=1)[1]
+    y_pred = torch.rand(4, 2)
+    y = torch.randint(0, 2, size=(4, 1)).type(torch.LongTensor)
     acc.update((y_pred, y))
+    np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+    np_y = y.numpy().ravel()
+    assert acc._type == 'multiclass'
     assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
 
-def test_categorical_compute_batch_images():
+def test_multiclass_input_NL():
+    # Multiclass input data of shape (N, L) and (N, C, L)
     acc = Accuracy()
 
-    y_pred = torch.softmax(torch.rand(2, 3, 2, 2), dim=1)
-    y = torch.LongTensor([[[0, 1],
-                           [0, 1]],
-                          [[0, 2],
-                           [0, 2]]])
-    indices = torch.max(y_pred, dim=1)[1]
+    y_pred = torch.rand(10, 4, 5)
+    y = torch.randint(0, 4, size=(10, 5)).type(torch.LongTensor)
     acc.update((y_pred, y))
+    np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+    np_y = y.numpy().ravel()
+    assert acc._type == 'multiclass'
     assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
+    acc.reset()
+    y_pred = torch.rand(4, 10, 5)
+    y = torch.randint(0, 10, size=(4, 5)).type(torch.LongTensor)
+    acc.update((y_pred, y))
+    np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+    np_y = y.numpy().ravel()
+    assert acc._type == 'multiclass'
+    assert isinstance(acc.compute(), float)
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
-def test_ner_example():
+
+def test_multiclass_input_NHW():
+    # Multiclass input data of shape (N, H, W, ...) and (N, C, H, W, ...)
     acc = Accuracy()
 
-    y_pred = torch.softmax(torch.rand(2, 3, 8), dim=1)
-    y = torch.Tensor([[1, 1, 1, 1, 1, 1, 1, 1],
-                      [2, 2, 2, 2, 2, 2, 2, 2]]).type(torch.LongTensor)
-    indices = torch.max(y_pred, dim=1)[1]
+    y_pred = torch.rand(4, 5, 12, 10)
+    y = torch.randint(0, 5, size=(4, 12, 10)).type(torch.LongTensor)
     acc.update((y_pred, y))
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
+    np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+    np_y = y.numpy().ravel()
+    assert acc._type == 'multiclass'
+    assert isinstance(acc.compute(), float)
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
+    acc.reset()
+    y_pred = torch.rand(4, 5, 10, 12, 8)
+    y = torch.randint(0, 5, size=(4, 10, 12, 8)).type(torch.LongTensor)
+    acc.update((y_pred, y))
+    np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+    np_y = y.numpy().ravel()
+    assert acc._type == 'multiclass'
+    assert isinstance(acc.compute(), float)
+    assert accuracy_score(np_y, np_y_pred) == pytest.approx(acc.compute())
 
-def test_incorrect_shape():
-    acc = Accuracy()
 
-    y_pred = torch.zeros(2, 3, 2, 2)
-    y = torch.zeros(2, 3)
+def test_incorrect_type():
+    acc = Accuracy()
 
-    with pytest.raises(ValueError):
-        acc.update((y_pred, y))
+    # Start as binary data
+    y_pred = torch.rand(4,)
+    y = torch.ones(4).type(torch.LongTensor)
+    acc.update((y_pred, y))
 
-    y_pred = torch.zeros(2, 3, 2, 2)
-    y = torch.zeros(2, 3, 4, 4)
+    # And add a multiclass data
+    y_pred = torch.rand(4, 4)
+    y = torch.ones(4).type(torch.LongTensor)
 
-    with pytest.raises(ValueError):
+    with pytest.raises(RuntimeError):
         acc.update((y_pred, y))
diff --git a/tests/ignite/metrics/test_binary_accuracy.py b/tests/ignite/metrics/test_binary_accuracy.py
index 37d91aed..5bb9c79d 100644
--- a/tests/ignite/metrics/test_binary_accuracy.py
+++ b/tests/ignite/metrics/test_binary_accuracy.py
@@ -1,59 +1,7 @@
-from ignite.exceptions import NotComputableError
 from ignite.metrics import BinaryAccuracy
 import pytest
-import torch
-from sklearn.metrics import accuracy_score
 
 
-def test_zero_div():
-    acc = BinaryAccuracy()
-    with pytest.raises(NotComputableError):
-        acc.compute()
-
-
-def test_compute():
-    acc = BinaryAccuracy()
-
-    y_pred = torch.sigmoid(torch.rand(4, 1))
-    y = torch.ones(4).type(torch.LongTensor)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    acc.update((y_pred, y))
-    assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.data.numpy(), indices.data.numpy()) == pytest.approx(acc.compute())
-
-    acc.reset()
-    y_pred = torch.sigmoid(torch.rand(4))
-    y = torch.ones(4).type(torch.LongTensor)
-    y_pred = y_pred.unsqueeze(1)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    acc.update((y_pred, y))
-    assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.data.numpy(), indices.data.numpy()) == pytest.approx(acc.compute())
-
-
-def test_compute_batch_images():
-    acc = BinaryAccuracy()
-
-    y_pred = torch.sigmoid(torch.rand(1, 2, 2))
-    y = torch.ones(1, 2, 2).type(torch.LongTensor)
-    y_pred = y_pred.unsqueeze(1)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    acc.update((y_pred, y))
-    assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
-
-    acc.reset()
-    y_pred = torch.sigmoid(torch.rand(2, 1, 2, 2))
-    y = torch.ones(2, 2, 2).type(torch.LongTensor)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    acc.update((y_pred, y))
-    assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
-
-    acc.reset()
-    y_pred = torch.sigmoid(torch.rand(2, 1, 2, 2))
-    y = torch.ones(2, 1, 2, 2).type(torch.LongTensor)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    acc.update((y_pred, y))
-    assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
+def test_warning():
+    with pytest.warns(DeprecationWarning):
+        BinaryAccuracy()
diff --git a/tests/ignite/metrics/test_categorical_accuracy.py b/tests/ignite/metrics/test_categorical_accuracy.py
index df2131a7..30a3bc52 100644
--- a/tests/ignite/metrics/test_categorical_accuracy.py
+++ b/tests/ignite/metrics/test_categorical_accuracy.py
@@ -1,44 +1,7 @@
-from ignite.exceptions import NotComputableError
 from ignite.metrics import CategoricalAccuracy
 import pytest
-import torch
-from sklearn.metrics import accuracy_score
 
 
-def test_zero_div():
-    acc = CategoricalAccuracy()
-    with pytest.raises(NotComputableError):
-        acc.compute()
-
-
-def test_compute():
-    acc = CategoricalAccuracy()
-
-    y_pred = torch.softmax(torch.rand(4, 4), dim=1)
-    y = torch.ones(4).type(torch.LongTensor)
-    indices = torch.max(y_pred, dim=1)[1]
-    acc.update((y_pred, y))
-    assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
-
-    acc.reset()
-    y_pred = torch.softmax(torch.rand(2, 2), dim=1)
-    y = torch.ones(2).type(torch.LongTensor)
-    indices = torch.max(y_pred, dim=1)[1]
-    acc.update((y_pred, y))
-    assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
-
-
-def test_compute_batch_images():
-    acc = CategoricalAccuracy()
-
-    y_pred = torch.softmax(torch.rand(2, 3, 2, 2), dim=1)
-    y = torch.LongTensor([[[0, 1],
-                           [0, 1]],
-                          [[0, 2],
-                           [0, 2]]])
-    indices = torch.max(y_pred, dim=1)[1]
-    acc.update((y_pred, y))
-    assert isinstance(acc.compute(), float)
-    assert accuracy_score(y.view(-1).data.numpy(), indices.view(-1).data.numpy()) == pytest.approx(acc.compute())
+def test_warning():
+    with pytest.warns(DeprecationWarning):
+        CategoricalAccuracy()
diff --git a/tests/ignite/metrics/test_precision.py b/tests/ignite/metrics/test_precision.py
index 63d55a4d..fd203915 100644
--- a/tests/ignite/metrics/test_precision.py
+++ b/tests/ignite/metrics/test_precision.py
@@ -1,8 +1,15 @@
+import pytest
+import warnings
+
+from sklearn.metrics import precision_score
+from sklearn.exceptions import UndefinedMetricWarning
+
 from ignite.exceptions import NotComputableError
 from ignite.metrics import Precision
-import pytest
+
 import torch
-from sklearn.metrics import precision_score
+
+torch.manual_seed(12)
 
 
 def test_no_update():
@@ -11,147 +18,313 @@ def test_no_update():
         precision.compute()
 
 
-def test_compute():
-    precision = Precision()
-
-    y_pred = torch.eye(4)
-    y = torch.ones(4).type(torch.LongTensor)
-    precision.update((y_pred, y))
-    results = list(precision.compute())
-    assert results[0] == 0.0
-    assert results[1] == 1.0
-    assert results[2] == 0.0
-    assert results[3] == 0.0
-
-    precision.reset()
-    y_pred = torch.eye(2)
-    y = torch.ones(2).type(torch.LongTensor)
-    precision.update((y_pred, y))
-    y = torch.zeros(2).type(torch.LongTensor)
-    precision.update((y_pred, y))
-
-    results = list(precision.compute())
-
-    assert results[0] == 0.5
-    assert results[1] == 0.5
-
-
-def test_compute_average():
-    precision = Precision(average=True)
-
-    y_pred = torch.eye(4)
-    y = torch.ones(4).type(torch.LongTensor)
-    precision.update((y_pred, y))
-    assert isinstance(precision.compute(), float)
-    assert precision.compute() == 0.25
-
-
-def test_compute_all_wrong():
-    precision = Precision()
-
-    y_pred = torch.FloatTensor([[1.0, 0.0], [1.0, 0.0]])
-    y = torch.ones(2).type(torch.LongTensor)
-    precision.update((y_pred, y))
-
-    results = list(precision.compute())
-
-    assert results[0] == 0.0
-    assert results[1] == 0.0
-
-
-def test_binary_vs_categorical():
-    precision = Precision(average=True)
-
-    y_pred = torch.FloatTensor([0.9, 0.2])
-    y = torch.LongTensor([1, 0])
-    y_pred = y_pred.unsqueeze(1)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    precision.update((y_pred, y))
-    assert precision.compute() == pytest.approx(precision_score(y.data.numpy(), indices.data.numpy(), average='macro'))
-    assert precision.compute() == 1.0
-
-    precision.reset()
-    y_pred = torch.FloatTensor([[0.1, 0.9], [0.8, 0.2]])
-    y = torch.LongTensor([1, 0])
-    indices = torch.max(y_pred, dim=1)[1]
-    precision.update((y_pred, y))
-    assert precision.compute() == pytest.approx(precision_score(y.data.numpy(), indices.data.numpy(), average='macro'))
-    assert precision.compute() == 1.0
-
-
-def test_binary_shapes():
-    precision = Precision(average=True)
-
-    y = torch.LongTensor([1, 0])
-    y_pred = torch.FloatTensor([0.9, 0.2])
-    y_pred = y_pred.unsqueeze(1)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    precision.update((y_pred, y))
-    assert precision.compute() == pytest.approx(precision_score(y.data.numpy(), indices.data.numpy(), average='macro'))
-    assert precision.compute() == 1.0
-
-    y = torch.LongTensor([[1], [0]])
-    y_pred = torch.FloatTensor([[0.9], [0.2]])
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    precision.reset()
-    precision.update((y_pred, y))
-    assert precision.compute() == pytest.approx(precision_score(y.data.numpy(), indices.data.numpy(), average='macro'))
-    assert precision.compute() == 1.0
-
-
-def test_ner_example():
-    precision = Precision()
-
-    y = torch.Tensor([[1, 1, 1, 1, 1, 1, 1, 1],
-                      [2, 2, 2, 2, 2, 2, 2, 2]]).type(torch.LongTensor)
-    y_pred = torch.softmax(torch.rand(2, 3, 8), dim=1)
-    indices = torch.max(y_pred, dim=1)[1]
-    y_pred_labels = list(set(indices.view(-1).tolist()))
-
-    precision_sk = precision_score(y.view(-1).data.numpy(),
-                                   indices.view(-1).data.numpy(),
-                                   labels=y_pred_labels,
-                                   average=None)
-    precision.update((y_pred, y))
-    precision_ig = precision.compute().tolist()
-    precision_ig = [precision_ig[i] for i in y_pred_labels]
-
-    assert all([a == pytest.approx(b) for a, b in zip(precision_sk, precision_ig)])
-
-
-def test_incorrect_shape():
-    precision = Precision()
-
-    y_pred = torch.zeros(2, 3, 2, 2)
-    y = torch.zeros(2, 3)
+def test_binary_wrong_inputs():
+    pr = Precision()
 
     with pytest.raises(ValueError):
-        precision.update((y_pred, y))
+        # y has not only 0 or 1 values
+        pr.update((torch.randint(0, 2, size=(10,)).type(torch.LongTensor),
+                   torch.arange(0, 10).type(torch.LongTensor)))
 
-    y_pred = torch.zeros(2, 3, 2, 2)
-    y = torch.zeros(2, 3, 4, 4)
+    # TODO: Uncomment the following after 0.1.2 release
+    # with pytest.raises(ValueError):
+    #     # y_pred values are not thresholded to 0, 1 values
+    #     pr.update((torch.rand(10, 1),
+    #                torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
 
     with pytest.raises(ValueError):
-        precision.update((y_pred, y))
-
-
-def test_sklearn_compute():
-    precision = Precision(average=False)
+        # incompatible shapes
+        pr.update((torch.randint(0, 2, size=(10,)).type(torch.LongTensor),
+                   torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor)))
 
-    y = torch.Tensor(range(5)).type(torch.LongTensor)
-    y_pred = torch.softmax(torch.rand(5, 5), dim=1)
-
-    indices = torch.max(y_pred, dim=1)[1]
-    precision.update((y_pred, y))
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        pr.update((torch.randint(0, 2, size=(10, 5, 6)).type(torch.LongTensor),
+                   torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
 
-    y_pred_labels = list(set(indices.tolist()))
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        pr.update((torch.randint(0, 2, size=(10,)).type(torch.LongTensor),
+                   torch.randint(0, 2, size=(10, 5, 6)).type(torch.LongTensor)))
+
+
+def test_binary_input_N():
+    # Binary accuracy on input of shape (N, 1) or (N, )
+
+    def _test(average):
+        pr = Precision(average=average)
+        y_pred = torch.rand(10, 1)
+        y = torch.randint(0, 2, size=(10,)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert pr._type == 'binary'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        assert precision_score(np_y, np_y_pred, average='binary') == pytest.approx(pr_compute)
+
+        pr.reset()
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, )).type(torch.LongTensor)
+        y_pred = torch.rand(10)
+        y = torch.randint(0, 2, size=(10,)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert pr._type == 'binary'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        assert precision_score(np_y, np_y_pred, average='binary') == pytest.approx(pr_compute)
+
+        pr.reset()
+        # TODO: y_pred should be binary after 0.1.2 release
+        y_pred = torch.Tensor([0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.51])
+        y = torch.randint(0, 2, size=(10,)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert pr._type == 'binary'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        assert precision_score(np_y, np_y_pred, average='binary') == pytest.approx(pr_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_binary_input_NL():
+    # Binary accuracy on input of shape (N, L)
+
+    def _test(average):
+        pr = Precision(average=average)
+
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor)
+        y_pred = torch.rand(10, 5)
+        y = torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert pr._type == 'binary'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        assert precision_score(np_y, np_y_pred, average='binary') == pytest.approx(pr_compute)
+
+        pr.reset()
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, 1, 5)).type(torch.LongTensor)
+        y_pred = torch.rand(10, 1, 5)
+        y = torch.randint(0, 2, size=(10, 1, 5)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert pr._type == 'binary'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        assert precision_score(np_y, np_y_pred, average='binary') == pytest.approx(pr_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_binary_input_NHW():
+    # Binary accuracy on input of shape (N, H, W)
+
+    def _test(average):
+        pr = Precision(average=average)
+
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, 12, 10)).type(torch.LongTensor)
+        y_pred = torch.rand(10, 12, 10)
+        y = torch.randint(0, 2, size=(10, 12, 10)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert pr._type == 'binary'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        assert precision_score(np_y, np_y_pred, average='binary') == pytest.approx(pr_compute)
+
+        pr.reset()
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, 1, 12, 10)).type(torch.LongTensor)
+        y_pred = torch.rand(10, 1, 12, 10)
+        y = torch.randint(0, 2, size=(10, 1, 12, 10)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert pr._type == 'binary'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        assert precision_score(np_y, np_y_pred, average='binary') == pytest.approx(pr_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_multiclass_wrong_inputs():
+    pr = Precision()
 
-    precision_sk = precision_score(y.data.numpy(),
-                                   indices.data.numpy(),
-                                   labels=y_pred_labels,
-                                   average=None)
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        pr.update((torch.rand(10, 5, 4), torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
 
-    precision_ig = precision.compute().tolist()
-    precision_ig = [precision_ig[i] for i in y_pred_labels]
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        pr.update((torch.rand(10, 5, 6), torch.randint(0, 5, size=(10, 5)).type(torch.LongTensor)))
 
-    assert all([a == pytest.approx(b) for a, b in zip(precision_sk, precision_ig)])
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        pr.update((torch.rand(10), torch.randint(0, 5, size=(10, 5, 6)).type(torch.LongTensor)))
+
+
+def test_multiclass_input_N():
+    # Multiclass input data of shape (N, ) and (N, C)
+
+    def _test(average):
+        pr = Precision(average=average)
+        y_pred = torch.rand(20, 6)
+        y = torch.randint(0, 5, size=(20,)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert pr._type == 'multiclass'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert precision_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(pr_compute)
+
+        pr.reset()
+        y_pred = torch.rand(10, 4)
+        y = torch.randint(0, 3, size=(10, 1)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert pr._type == 'multiclass'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert precision_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(pr_compute)
+
+        # 2-classes
+        pr.reset()
+        y_pred = torch.rand(10, 2)
+        y = torch.randint(0, 2, size=(10, 1)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert pr._type == 'multiclass'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert precision_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(pr_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_multiclass_input_NL():
+    # Multiclass input data of shape (N, L) and (N, C, L)
+
+    def _test(average):
+        pr = Precision(average=average)
+
+        y_pred = torch.rand(10, 5, 8)
+        y = torch.randint(0, 4, size=(10, 8)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert pr._type == 'multiclass'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert precision_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(pr_compute)
+
+        pr.reset()
+        y_pred = torch.rand(15, 10, 8)
+        y = torch.randint(0, 9, size=(15, 8)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert pr._type == 'multiclass'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert precision_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(pr_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_multiclass_input_NHW():
+    # Multiclass input data of shape (N, H, W, ...) and (N, C, H, W, ...)
+
+    def _test(average):
+        pr = Precision(average=average)
+
+        y_pred = torch.rand(10, 5, 18, 16)
+        y = torch.randint(0, 4, size=(10, 18, 16)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert pr._type == 'multiclass'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert precision_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(pr_compute)
+
+        pr.reset()
+        y_pred = torch.rand(10, 7, 20, 12)
+        y = torch.randint(0, 6, size=(10, 20, 12)).type(torch.LongTensor)
+        pr.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert pr._type == 'multiclass'
+        assert isinstance(pr.compute(), float if average else torch.Tensor)
+        pr_compute = pr.compute() if average else pr.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert precision_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(pr_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_incorrect_type():
+    # Tests changing of type during training
+
+    def _test(average):
+        pr = Precision(average=average)
+
+        y_pred = torch.softmax(torch.rand(4, 4), dim=1)
+        y = torch.ones(4).type(torch.LongTensor)
+        pr.update((y_pred, y))
+
+        y_pred = torch.rand(4, 1)
+        y = torch.ones(4).type(torch.LongTensor)
+
+        with pytest.raises(RuntimeError):
+            pr.update((y_pred, y))
+
+    _test(average=True)
+    _test(average=False)
diff --git a/tests/ignite/metrics/test_recall.py b/tests/ignite/metrics/test_recall.py
index 0d3c610d..885698a5 100644
--- a/tests/ignite/metrics/test_recall.py
+++ b/tests/ignite/metrics/test_recall.py
@@ -1,8 +1,15 @@
+import pytest
+import warnings
+
+from sklearn.metrics import recall_score
+from sklearn.exceptions import UndefinedMetricWarning
+
 from ignite.exceptions import NotComputableError
 from ignite.metrics import Recall
-import pytest
+
 import torch
-from sklearn.metrics import recall_score
+
+torch.manual_seed(12)
 
 
 def test_no_update():
@@ -11,150 +18,313 @@ def test_no_update():
         recall.compute()
 
 
-def test_compute():
-    recall = Recall()
-
-    y_pred = torch.eye(4)
-    y = torch.ones(4).type(torch.LongTensor)
-    recall.update((y_pred, y))
-
-    result = list(recall.compute())
-
-    assert result[0] == 0.0
-    assert result[1] == 0.25
-    assert result[2] == 0.0
-    assert result[3] == 0.0
-
-    recall.reset()
-    y_pred = torch.eye(2)
-    y = torch.ones(2).type(torch.LongTensor)
-    recall.update((y_pred, y))
-    y = torch.zeros(2).type(torch.LongTensor)
-    recall.update((y_pred, y))
-
-    result = list(recall.compute())
-
-    assert result[0] == 0.5
-    assert result[1] == 0.5
-
-
-def test_compute_average():
-    recall = Recall(average=True)
-
-    y_pred = torch.eye(4)
-    y = torch.ones(4).type(torch.LongTensor)
-    recall.update((y_pred, y))
-
-    assert isinstance(recall.compute(), float)
-    assert recall.compute() == 0.0625
-
-
-def test_compute_all_wrong():
-    recall = Recall()
-
-    y_pred = torch.FloatTensor([[1.0, 0.0], [1.0, 0.0]])
-    y = torch.ones(2).type(torch.LongTensor)
-    recall.update((y_pred, y))
-
-    result = list(recall.compute())
-
-    assert result[0] == 0.0
-    assert result[1] == 0.0
-
-
-def test_binary_vs_categorical():
-    recall = Recall(average=True)
-
-    y_pred = torch.FloatTensor([0.9, 0.2])
-    y = torch.LongTensor([1, 0])
-    y_pred = y_pred.unsqueeze(1)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    recall.update((y_pred, y))
-    assert recall.compute() == pytest.approx(recall_score(y.data.numpy(), indices.data.numpy(), average='macro'))
-    assert recall.compute() == 1.0
-
-    recall.reset()
-    y_pred = torch.FloatTensor([[0.1, 0.9], [0.8, 0.2]])
-    y = torch.LongTensor([1, 0])
-    indices = torch.max(y_pred, dim=1)[1]
-    recall.update((y_pred, y))
-    assert recall.compute() == pytest.approx(recall_score(y.data.numpy(), indices.data.numpy(), average='macro'))
-    assert recall.compute() == 1.0
-
-
-def test_binary_shapes():
-    recall = Recall(average=True)
-
-    y = torch.LongTensor([1, 0])
-    y_pred = torch.FloatTensor([0.9, 0.2])
-    y_pred = y_pred.unsqueeze(1)
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    recall.update((y_pred, y))
-    assert recall.compute() == pytest.approx(recall_score(y.data.numpy(), indices.data.numpy(), average='macro'))
-    assert recall.compute() == 1.0
-
-    y = torch.LongTensor([[1], [0]])
-    y_pred = torch.FloatTensor([[0.9], [0.2]])
-    indices = torch.max(torch.cat([1.0 - y_pred, y_pred], dim=1), dim=1)[1]
-    recall.reset()
-    recall.update((y_pred, y))
-    assert recall.compute() == pytest.approx(recall_score(y.data.numpy(), indices.data.numpy(), average='macro'))
-    assert recall.compute() == 1.0
-
-
-def test_ner_example():
-    recall = Recall()
-
-    y = torch.Tensor([[0, 1, 1, 1, 1, 1, 1, 1],
-                      [2, 2, 2, 2, 2, 2, 2, 2]]).type(torch.LongTensor)
-    y_pred = torch.softmax(torch.rand(2, 3, 8), dim=1)
-    indices = torch.max(y_pred, dim=1)[1]
-    y_pred_labels = list(set(indices.view(-1).tolist()))
-
-    recall_sk = recall_score(y.view(-1).data.numpy(),
-                             indices.view(-1).data.numpy(),
-                             labels=y_pred_labels,
-                             average=None)
-    recall.update((y_pred, y))
-    recall_ig = recall.compute().tolist()
-    recall_ig = [recall_ig[i] for i in y_pred_labels]
-
-    assert all([a == pytest.approx(b) for a, b in zip(recall_sk, recall_ig)])
-
-
-def test_incorrect_shape():
-    recall = Recall()
-
-    y_pred = torch.zeros(2, 3, 2, 2)
-    y = torch.zeros(2, 3)
+def test_binary_wrong_inputs():
+    re = Recall()
 
     with pytest.raises(ValueError):
-        recall.update((y_pred, y))
+        # y has not only 0 or 1 values
+        re.update((torch.randint(0, 2, size=(10,)).type(torch.LongTensor),
+                   torch.arange(0, 10).type(torch.LongTensor)))
 
-    y_pred = torch.zeros(2, 3, 2, 2)
-    y = torch.zeros(2, 3, 4, 4)
+    # TODO: Uncomment the following after 0.1.2 release
+    # with pytest.raises(ValueError):
+    #     # y_pred values are not thresholded to 0, 1 values
+    #     pr.update((torch.rand(10, 1),
+    #                torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
 
     with pytest.raises(ValueError):
-        recall.update((y_pred, y))
-
-
-def test_sklearn_compute():
-    recall = Recall(average=False)
+        # incompatible shapes
+        re.update((torch.randint(0, 2, size=(10,)).type(torch.LongTensor),
+                   torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor)))
 
-    y = torch.Tensor(range(5)).type(torch.LongTensor)
-    y_pred = torch.softmax(torch.rand(5, 5), dim=1)
-
-    indices = torch.max(y_pred, dim=1)[1]
-    recall.update((y_pred, y))
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        re.update((torch.randint(0, 2, size=(10, 5, 6)).type(torch.LongTensor),
+                   torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
 
-    y_pred_labels = list(set(indices.tolist()))
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        re.update((torch.randint(0, 2, size=(10,)).type(torch.LongTensor),
+                   torch.randint(0, 2, size=(10, 5, 6)).type(torch.LongTensor)))
+
+
+def test_binary_input_N():
+    # Binary accuracy on input of shape (N, 1) or (N, )
+
+    def _test(average):
+        re = Recall(average=average)
+        y_pred = torch.rand(10, 1)
+        y = torch.randint(0, 2, size=(10,)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert re._type == 'binary'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        assert recall_score(np_y, np_y_pred, average='binary') == pytest.approx(re_compute)
+
+        re.reset()
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, )).type(torch.LongTensor)
+        y_pred = torch.rand(10)
+        y = torch.randint(0, 2, size=(10,)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert re._type == 'binary'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        assert recall_score(np_y, np_y_pred, average='binary') == pytest.approx(re_compute)
+
+        re.reset()
+        # TODO: y_pred should be binary after 0.1.2 release
+        y_pred = torch.Tensor([0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.51])
+        y = torch.randint(0, 2, size=(10,)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert re._type == 'binary'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        assert recall_score(np_y, np_y_pred, average='binary') == pytest.approx(re_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_binary_input_NL():
+    # Binary accuracy on input of shape (N, L)
+
+    def _test(average):
+        re = Recall(average=average)
+
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor)
+        y_pred = torch.rand(10, 5)
+        y = torch.randint(0, 2, size=(10, 5)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert re._type == 'binary'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        pr_compute = re.compute() if average else re.compute().numpy()
+        assert recall_score(np_y, np_y_pred, average='binary') == pytest.approx(pr_compute)
+
+        re.reset()
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, 1, 5)).type(torch.LongTensor)
+        y_pred = torch.rand(10, 1, 5)
+        y = torch.randint(0, 2, size=(10, 1, 5)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert re._type == 'binary'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        pr_compute = re.compute() if average else re.compute().numpy()
+        assert recall_score(np_y, np_y_pred, average='binary') == pytest.approx(pr_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_binary_input_NHW():
+    # Binary accuracy on input of shape (N, H, W)
+
+    def _test(average):
+        re = Recall(average=average)
+
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, 12, 10)).type(torch.LongTensor)
+        y_pred = torch.rand(10, 12, 10)
+        y = torch.randint(0, 2, size=(10, 12, 10)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert re._type == 'binary'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        assert recall_score(np_y, np_y_pred, average='binary') == pytest.approx(re_compute)
+
+        re.reset()
+        # TODO: y_pred should be binary after 0.1.2 release
+        # y_pred = torch.randint(0, 2, size=(10, 1, 12, 10)).type(torch.LongTensor)
+        y_pred = torch.rand(10, 1, 12, 10)
+        y = torch.randint(0, 2, size=(10, 1, 12, 10)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y = y.numpy().ravel()
+        # np_y_pred = y_pred.numpy().ravel()
+        np_y_pred = (y_pred.numpy().ravel() > 0.5).astype('int')
+        assert re._type == 'binary'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        assert recall_score(np_y, np_y_pred, average='binary') == pytest.approx(re_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_multiclass_wrong_inputs():
+    re = Recall()
 
-    recall_sk = recall_score(y.data.numpy(),
-                             indices.data.numpy(),
-                             labels=y_pred_labels,
-                             average=None)
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        re.update((torch.rand(10, 5, 4), torch.randint(0, 2, size=(10,)).type(torch.LongTensor)))
 
-    recall_ig = recall.compute().tolist()
-    recall_ig = [recall_ig[i] for i in y_pred_labels]
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        re.update((torch.rand(10, 5, 6), torch.randint(0, 5, size=(10, 5)).type(torch.LongTensor)))
 
-    assert all([a == pytest.approx(b) for a, b in zip(recall_sk, recall_ig)])
+    with pytest.raises(ValueError):
+        # incompatible shapes
+        re.update((torch.rand(10), torch.randint(0, 5, size=(10, 5, 6)).type(torch.LongTensor)))
+
+
+def test_multiclass_input_N():
+    # Multiclass input data of shape (N, ) and (N, C)
+
+    def _test(average):
+        re = Recall(average=average)
+        y_pred = torch.rand(20, 6)
+        y = torch.randint(0, 5, size=(20,)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert re._type == 'multiclass'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert recall_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(re_compute)
+
+        re.reset()
+        y_pred = torch.rand(10, 4)
+        y = torch.randint(0, 3, size=(10, 1)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert re._type == 'multiclass'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert recall_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(re_compute)
+
+        # 2-classes
+        re.reset()
+        y_pred = torch.rand(10, 2)
+        y = torch.randint(0, 2, size=(10, 1)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert re._type == 'multiclass'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert recall_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(re_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_multiclass_input_NL():
+    # Multiclass input data of shape (N, L) and (N, C, L)
+
+    def _test(average):
+        re = Recall(average=average)
+
+        y_pred = torch.rand(10, 5, 8)
+        y = torch.randint(0, 4, size=(10, 8)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert re._type == 'multiclass'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert recall_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(re_compute)
+
+        re.reset()
+        y_pred = torch.rand(15, 10, 8)
+        y = torch.randint(0, 9, size=(15, 8)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert re._type == 'multiclass'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert recall_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(re_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_multiclass_input_NHW():
+    # Multiclass input data of shape (N, H, W, ...) and (N, C, H, W, ...)
+
+    def _test(average):
+        re = Recall(average=average)
+
+        y_pred = torch.rand(10, 5, 18, 16)
+        y = torch.randint(0, 4, size=(10, 18, 16)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert re._type == 'multiclass'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert recall_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(re_compute)
+
+        re.reset()
+        y_pred = torch.rand(10, 7, 20, 12)
+        y = torch.randint(0, 6, size=(10, 20, 12)).type(torch.LongTensor)
+        re.update((y_pred, y))
+        np_y_pred = y_pred.numpy().argmax(axis=1).ravel()
+        np_y = y.numpy().ravel()
+        assert re._type == 'multiclass'
+        assert isinstance(re.compute(), float if average else torch.Tensor)
+        re_compute = re.compute() if average else re.compute().numpy()
+        sklearn_average_parameter = 'macro' if average else None
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", category=UndefinedMetricWarning)
+            assert recall_score(np_y, np_y_pred, average=sklearn_average_parameter) == pytest.approx(re_compute)
+
+    _test(average=True)
+    _test(average=False)
+
+
+def test_incorrect_type():
+    # Tests changing of type during training
+
+    def _test(average):
+        re = Recall(average=average)
+
+        y_pred = torch.softmax(torch.rand(4, 4), dim=1)
+        y = torch.ones(4).type(torch.LongTensor)
+        re.update((y_pred, y))
+
+        y_pred = torch.rand(4, 1)
+        y = torch.ones(4).type(torch.LongTensor)
+
+        with pytest.raises(RuntimeError):
+            re.update((y_pred, y))
+
+    _test(average=True)
+    _test(average=False)
diff --git a/tests/ignite/metrics/test_running_average.py b/tests/ignite/metrics/test_running_average.py
index 0ec6a8d1..6106ee4c 100644
--- a/tests/ignite/metrics/test_running_average.py
+++ b/tests/ignite/metrics/test_running_average.py
@@ -4,7 +4,7 @@ import numpy as np
 import torch
 
 from ignite.engine import Engine, Events
-from ignite.metrics import CategoricalAccuracy, RunningAverage
+from ignite.metrics import Accuracy, RunningAverage
 
 import pytest
 
@@ -17,7 +17,7 @@ def test_wrong_input_args():
         _ = RunningAverage(alpha=-1.0)
 
     with pytest.raises(ValueError):
-        _ = RunningAverage(CategoricalAccuracy(), output_transform=lambda x: x[0])
+        _ = RunningAverage(Accuracy(), output_transform=lambda x: x[0])
 
     with pytest.raises(ValueError):
         _ = RunningAverage()
@@ -41,7 +41,7 @@ def test_integration():
     trainer = Engine(update_fn)
     alpha = 0.98
 
-    acc_metric = RunningAverage(CategoricalAccuracy(output_transform=lambda x: [x[1], x[2]]), alpha=alpha)
+    acc_metric = RunningAverage(Accuracy(output_transform=lambda x: [x[1], x[2]]), alpha=alpha)
     acc_metric.attach(trainer, 'running_avg_accuracy')
 
     avg_output = RunningAverage(output_transform=lambda x: x[0], alpha=alpha)