How to plot ROC Curve using PyTorch model

username: ahmediqbal
cooked: <p>Hello,</p>
<p>I have semantic segmentation code, this code help me to test 25 images results (using confusion matrix). But I want to plot ROC Curve of testing datasets. But I am unable to do this job. Please check my shared code, and let me know, how I properly draw ROC curve by using this code.</p>
<blockquote>
<p>import os<br>
import cv2<br>
import torch<br>
import numpy as np<br>
from glob import glob<br>
from model import AI_Net<br>
from operator import add<br>
from crf import apply_crf<br>
import matplotlib.pyplot as plt<br>
from sklearn.metrics import roc_curve<br>
from sklearn.metrics import roc_auc_score</p>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)</p>
<p>“”" Load the checkpoint “”"<br>
model = AI_Net()<br>
model = model.to(device)<br>
model.load_state_dict(torch.load(‘datasets/models/A_Net/Fold_1_Model.pth’, map_location=device))<br>
model.eval()</p>
<p>def calculate_metrics(y_true, y_pred):<br>
y_true = y_true.cpu().numpy()<br>
y_pred = y_pred.cpu().numpy()</p>
<pre><code>y_pred = y_pred &gt; 0.5
y_pred = y_pred.reshape(-1)
y_pred = y_pred.astype(np.uint8)

y_true = y_true &gt; 0.5
y_true = y_true.reshape(-1)
y_true = y_true.astype(np.uint8)

## Score
score_fpr, score_tpr, _ = roc_curve(y_true, y_pred)
score_roc_auc = roc_auc_score(y_true, y_pred)

return [score_fpr, score_tpr, score_roc_auc]

""" Load dataset """
</code></pre>
<p>root_path  =  ‘datasets/UDIAT/’<br>
test_x = sorted(glob(os.path.join(root_path, “test/images”, “<em>.png")))<br>
test_y = sorted(glob(os.path.join(root_path, “test/masks”, "</em>.png”)))</p>
<p>metrics_score = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]</p>
<p>for i, (x, y) in enumerate(zip(test_x, test_y)):<br>
name = y.split(“/”)[-1].split(“.”)[0]<br>
## Image<br>
image = cv2.imread(x, cv2.IMREAD_GRAYSCALE)<br>
image = cv2.resize(image, (128, 128))<br>
ori_img1 = image<br>
image1 = np.expand_dims(image, axis=0)<br>
image1 = image1/255.0<br>
image1 = np.expand_dims(image1, axis=0)<br>
image1 = image1.astype(np.float32)<br>
image1 = torch.from_numpy(image1)<br>
image1 = image1.to(device)</p>
<pre><code>  ## Mask
  mask = cv2.imread(y, cv2.IMREAD_GRAYSCALE)
  mask = cv2.resize(mask, (128, 128))
  ori_mask1 = mask
  mask1 = np.expand_dims(mask, axis=0)
  mask1 = mask1/255.0
  mask1 = np.expand_dims(mask1, axis=0)
  mask1 = mask1.astype(np.float32)
  mask1 = torch.from_numpy(mask1)
  mask1 = mask1.to(device)

  # Shape
  #print('image shape:', image1.shape, 'roi shape:', mask1.shape)
          
  with torch.no_grad():
      pred_y1 = torch.sigmoid(model(image1))
      score = calculate_metrics(mask1, pred_y1)
      metrics_score = list(map(add, metrics_score, score))

    tpr     = metrics_score[0]/len(test_x)
    fpr     = metrics_score[1]/len(test_x)
    roc_auc = metrics_score[2]/len(test_x)
</code></pre>
<p>algorithm = ‘CNN’<br>
dataset = ‘BUS’<br>
rocCurve = plt.figure()<br>
plt.plot(fpr, tpr, ‘-’, label=algorithm + ‘_’ + dataset + ‘(AUC = %0.4f)’ % roc_auc)<br>
plt.title(‘ROC curve’, fontsize=14)<br>
plt.xlabel(“FPR (False Positive Rate)”, fontsize=15)<br>
plt.ylabel(“TPR (True Positive Rate)”, fontsize=15)<br>
plt.legend(loc=“lower right”)<br>
plt.xticks(fontsize=15)<br>
plt.yticks(fontsize=15)<br>
plt.show()</p>
</blockquote>

username: ptrblck
cooked: <p>You could use the ROC implementations from other libraries such as <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html"><code>sklearn.metrics.roc_curve</code></a>. To do so you could transform the predictions and targets to numpy arrays via <code>tensor.numpy()</code> and apply the mentioned method.</p>

username: ahmediqbal
cooked: <p>Hello,</p>
<p>To store all iterations results of <strong>y_true</strong>, and <strong>y_pred</strong>, i added <strong>all_y_true</strong>, <strong>all_y_pred</strong>. But when i try to plot ROC Curve, it shows <strong>ValueError: continuous format is not supported</strong>,  at line → 1<strong>1 fpr, tpr, _ = roc_curve(y_true, y_pred)</strong></p>
<p>Here is my code:</p>
<blockquote>
<p>all_y_true = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
all_y_pred = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>for i, (x, y) in enumerate(zip(test_x, test_y)):<br>
name = y.split(“/”)[-1].split(“.”)[0]<br>
## Image<br>
image = cv2.imread(x, cv2.IMREAD_GRAYSCALE)<br>
image = cv2.resize(image, (128, 128))<br>
image = np.expand_dims(image, axis=0)<br>
image = image/255.0<br>
image = np.expand_dims(image, axis=0)<br>
image = image.astype(np.float32)<br>
image = torch.from_numpy(image)<br>
image = image.to(device)</p>
<pre><code>## Mask
mask = cv2.imread(y, cv2.IMREAD_GRAYSCALE)
mask = cv2.resize(mask, (128, 128))
mask = np.expand_dims(mask, axis=0)
mask = mask/255.0
mask = np.expand_dims(mask, axis=0)
mask = mask.astype(np.float32)
mask = torch.from_numpy(mask)
#mask = mask.to(device)
all_y_true = np.append(all_y_true, mask)

# Shape
#print('image shape:', image1.shape, 'roi shape:', mask1.shape)

with torch.no_grad():
    pred_y1 = torch.sigmoid(model(image))
    y_pred = pred_y1.cpu().numpy()
    y_pred = y_pred &gt; 0.5
    y_pred = y_pred.reshape(-1)
    y_pred = y_pred.astype(np.uint8)
    #y_pred.flatten()
    all_y_pred = np.append(all_y_pred, y_pred)
</code></pre>
<h1>Calculate image-level ROC AUC score</h1>
<p>y_true = all_y_true<br>
y_pred = all_y_pred</p>
<p>fpr, tpr, _ = roc_curve(y_true, y_pred)<br>
roc_auc = roc_auc_score(y_true, y_pred)</p>
<p>plt.figure(1)<br>
plt.plot([0, 1], [0, 1], ‘k–’)<br>
plt.plot(fpr, tpr, label=‘CNN(area = {:.3f})’.format(roc_auc))<br>
plt.xlabel(‘False positive rate’)<br>
plt.ylabel(‘True positive rate’)<br>
plt.title(‘ROC curve’)<br>
plt.legend(loc=‘best’)<br>
plt.show()</p>
</blockquote>

username: ptrblck
cooked: <p>I guess the inputs to <code>roc_curve</code> are wrong, so you would have to make sure they fit the expected arrays as described in the docs:</p>
<blockquote>
<p><strong>y_true</strong> ndarray of shape (n_samples,)<br>
True binary labels. If labels are not either {-1, 1} or {0, 1}, then pos_label should be explicitly given.</p>
</blockquote>
<blockquote>
<p><strong>y_score</strong> ndarray of shape (n_samples,)<br>
Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).</p>
</blockquote>

username: ahmediqbal
cooked: <p>I resolved error, but now i am getting this error</p>
<blockquote>
<p>ValueError: multiclass format is not supported → Line 12 fpr, tpr, _ = roc_curve(y_true, y_pred)</p>
</blockquote>

username: ptrblck
cooked: <p>This indicates a wrong shape of one of the inputs, so you would have to make sure to use the described shapes from my previous post.</p>

username: ahmediqbal
cooked: <p>Thanks very much, I transform my <strong>y_true</strong>, <strong>y_score</strong> into acceptable shapes, and issue is resolved.</p>