BrokenPipeError: [Errno 32] Broken pipe on Windows and CUDA 10

username: Neda
cooked: <p>I am able to run my program on system with <code>CUDA 9.2, Quadro M5000 (5), and python 3.7.0 (anaconda)</code> while the same program giving me an error on another system with the <code>CUDA 10.0, GeForce GTX 1080 Ti (6), and python 3.7.1</code></p>
<pre><code class="lang-auto"> ForkingPickler(file, protocol).dump(obj)

BrokenPipeError: [Errno 32] Broken pipe
</code></pre>
<p>the whole code wrap in a function named  <code>train_valid_model()</code>  and just add the  <code>main</code>  guard in the  script:</p>
<pre><code class="lang-auto">def train_valid_model():
    # complete code here

if __name__=='__main__':
    train_valid_model()
</code></pre>
<p>In Anaconda Prompt to make sure about is CUDA available I’ve done this <code>import torch torch.cuda.is_available()</code> and it’s returning True while it’s not able to do this <code>t_image = t_image.to(device)</code></p>
<p>It only works when num_workers=0 and if I set it to 1 or 2 I will get this error. Any idea why I am  getting this error while the program is same just the CUDA version and python version are different?</p>

username: SimonW
cooked: <p>Which pytorch version is this?</p>

username: Neda
cooked: <p><a class="mention" href="/u/simonw">@SimonW</a> It’s  <code>Torch version: 1.0.0</code></p>
<p>I also tried with the cuda 9.2 and I am getting the same error.</p>

username: marcin
cooked: <p>Any update on this issue?<br>
I am experiencing similar problem. Below whole error stack.</p>
<pre><code class="lang-auto">
(pytorch) C:\Users\dzidu\Documents\Python Scripts\pytorch-cifar-master&gt;python main.py
==&gt; Preparing data..
Files already downloaded and verified
Files already downloaded and verified
==&gt; Building model..

Epoch: 0
==&gt; Preparing data..
Files already downloaded and verified
Files already downloaded and verified
==&gt; Building model..

Epoch: 0
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\dzidu\Documents\Python Scripts\pytorch-cifar-master\main.py", line 142, in &lt;module&gt;
    train(epoch)
  File "C:\Users\dzidu\Documents\Python Scripts\pytorch-cifar-master\main.py", line 90, in train
    for batch_idx, (inputs, targets) in enumerate(trainloader):
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 193, in __iter__
    return _DataLoaderIter(self)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 469, in __init__
    w.start()
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "main.py", line 142, in &lt;module&gt;
    train(epoch)
  File "main.py", line 90, in train
    for batch_idx, (inputs, targets) in enumerate(trainloader):
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 193, in __iter__
    return _DataLoaderIter(self)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 469, in __init__
    w.start()
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\dzidu\Anaconda3\envs\pytorch\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe
</code></pre>
<p>Thanks for any info.</p>
<p>system info:</p>
<pre><code class="lang-auto">Microsoft Windows [Version 10.0.17134.706]
(c) 2018 Microsoft Corporation. All rights reserved.
</code></pre>
<p>PyTorch:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.__version__
'1.1.0'
</code></pre>
<p>Thanks for any update on this.</p>

username: ptrblck
cooked: <p>Did you add the if-clause protection to your script?<br>
This is necessary if you are using Windows and multiple workers in your <code>DataLoader</code>:</p>
<pre><code class="lang-python">def main():
   # your code

if __name__ == '__main__':
    main()
</code></pre>

username: marcin
cooked: <p>Of course I didn’t and that is why it didn’t work ;).</p>
<p>Thanks for that, now it works!</p>

username: Eric1012
cooked: <p>I use if-clause to protect my dataloader and training process, but it still told me “[Errno 32] Broken pipe”. Please tell me how to solve it.</p>

username: ptrblck
cooked: <p>Try to use <code>num_workers=0</code> and rerun the code.<br>
If your <code>Dataset</code> raises errors, the multiple processes of the <code>DataLoader</code> might crash without giving you a good error message.</p>

username: Eric1012
cooked: <p>Thanks a lot. I use num_workers=0  and it works. Maybe my Dataset has some problem, I will check it later.</p>

username: Anasz
cooked: <p>Dear Sir, I got the error"File “C:\Users\max\anaconda3\lib\multiprocessing\reduction.py”, line 60, in dump ForkingPickler(file, protocol).dump(obj)<br>
BrokenPipeError: [Errno 32] Broken pipe. not i use win7 what the modifications that should i made here my codes</p>
<p>import os<br>
import itertools<br>
import time<br>
import random</p>
<p>import numpy as np # linear algebra<br>
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)<br>
import matplotlib.pyplot as plt<br>
import matplotlib.colors as mcolors<br>
import seaborn as sns<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
from torch.utils.data import Dataset, DataLoader<br>
from torch.optim import AdamW<br>
from torch.optim.lr_scheduler import (CosineAnnealingLR,<br>
CosineAnnealingWarmRestarts,<br>
StepLR,<br>
ExponentialLR)</p>
<p>from sklearn.model_selection import train_test_split<br>
from sklearn.metrics import precision_recall_curve<br>
from sklearn.metrics import classification_report, confusion_matrix<br>
from sklearn.metrics import accuracy_score, auc, f1_score, precision_score, recall_score<br>
class Config:<br>
csv_path = ‘’<br>
seed = 2021<br>
device = ‘cuda:0’ if torch.cuda.is_available() else ‘cpu’<br>
attn_state_path = ‘C:/Users/max/iinput/mitbih-with-synthetic/attn.pth’<br>
lstm_state_path = ‘C:/Users/max/input/mitbih-with-synthetic/lstm.pth’<br>
cnn_state_path = ‘C:/Users/max/iinput/mitbih-with-synthetic/cnn.pth’</p>
<pre><code>attn_logs = 'C:/Users/max/iinput/mitbih-with-synthetic/attn.csv'
lstm_logs = 'C:/Users/max/iinput/mitbih-with-synthetic/lstm.csv'
cnn_logs = 'C:/Users/max/iinput/mitbih-with-synthetic/cnn.csv'

train_csv_path = 'C:/Users/max/iinput/mitbih_with_syntetic_train.csv'
test_csv_path = 'C:/Users/max/iinput/mitbih_with_syntetic_test.csv'
</code></pre>
<p>def seed_everything(seed: int):<br>
random.seed(seed)<br>
np.random.seed(seed)<br>
torch.manual_seed(seed)<br>
if torch.cuda.is_available():<br>
torch.cuda.manual_seed(seed)<br>
config = Config()<br>
seed_everything(config.seed)<br>
df_ptbdb = pd.read_csv(‘C:/Users/max/iinput/ptbdb_abnormal.csv’)<br>
df_mitbih = pd.read_csv(‘C:/Users/max/iinput/mitbih_train.csv’)<br>
df_ptbdb<br>
df_mitbih_train = pd.read_csv(‘C:/Users/max/iinput/mitbih_train.csv’, header=None)<br>
df_mitbih_test = pd.read_csv(‘C:/Users/max/iinput/mitbih_test.csv’, header=None)<br>
df_mitbih = pd.concat([df_mitbih_train, df_mitbih_test], axis=0)<br>
df_mitbih.rename(columns={187: ‘class’}, inplace=True)</p>
<p>id_to_label = {<br>
0: “Normal”,<br>
1: “Artial Premature”,<br>
2: “Premature ventricular contraction”,<br>
3: “Fusion of ventricular and normal”,<br>
4: “Fusion of paced and normal”<br>
}<br>
df_mitbih[‘label’] = df_mitbih.iloc[:, -1].map(id_to_label)<br>
print(df_mitbih.info())<br>
df_mitbih.to_csv(‘data.csv’, index=False)<br>
config.csv_path = ‘data.csv’<br>
df_mitbih = pd.read_csv(config.csv_path)<br>
df_mitbih[‘label’].value_counts()<br>
percentages = [count / df_mitbih.shape[0] * 100 for count in df_mitbih[‘label’].value_counts()]</p>
<p>fig, ax = plt.subplots(figsize=(12, 6))<br>
sns.countplot(<br>
x=df_mitbih[‘label’],<br>
ax=ax,<br>
palette=“bright”,<br>
order=df_mitbih[‘label’].value_counts().index<br>
)<br>
ax.set_xticklabels(ax.get_xticklabels(), rotation=15);</p>
<p>for percentage, count, p in zip(<br>
percentages,<br>
df_mitbih[‘label’].value_counts(sort=True).values,<br>
ax.patches):<br>
percentage = f’{np.round(percentage, 2)}%’<br>
x = p.get_x() + p.get_width() / 2 - 0.4<br>
y = p.get_y() + p.get_height()<br>
ax.annotate(str(percentage)+" / "+str(count), (x, y), fontsize=12, fontweight=‘bold’)</p>
<p>plt.savefig(‘data_dist.png’, facecolor=‘w’, edgecolor=‘w’, format=‘png’,<br>
transparent=False, bbox_inches=‘tight’, pad_inches=0.1)<br>
plt.savefig(‘data_dist.svg’, facecolor=‘w’, edgecolor=‘w’, format=‘svg’,<br>
transparent=False, bbox_inches=‘tight’, pad_inches=0.1)<br>
config.csv_path = ‘C:/Users/max/iinput/mitbih_with_syntetic.csv’<br>
df_mitbih_new = pd.read_csv(config.csv_path)<br>
percentages1 = [count / df_mitbih.shape[0] * 100 for count in df_mitbih[‘label’].value_counts()]<br>
percentages2 = [count / df_mitbih_new.shape[0] * 100 for count in df_mitbih_new[‘label’].value_counts()]</p>
<p>fig, axs = plt.subplots(1,2, figsize=(18, 4))</p>
<h1>
<a name="origin-1" class="anchor" href="#origin-1"></a>origin</h1>
<p>sns.countplot(<br>
x=df_mitbih[‘label’],<br>
ax=axs[0],<br>
palette=“bright”,<br>
order=df_mitbih[‘label’].value_counts().index<br>
)<br>
axs[0].set_xticklabels(axs[0].get_xticklabels(), rotation=15);<br>
axs[0].set_title(“Before”, fontsize=15)<br>
for percentage, count, p in zip(<br>
percentages1,<br>
df_mitbih[‘label’].value_counts(sort=True).values,<br>
axs[0].patches):</p>
<pre><code>percentage = f'{np.round(percentage, 2)}%'
x = p.get_x() + p.get_width() / 2 - 0.4
y = p.get_y() + p.get_height()
axs[0].annotate(str(percentage)+" / "+str(count), (x, y), fontsize=10, fontweight='bold')
</code></pre>
<h1>
<a name="with-synthetic-2" class="anchor" href="#with-synthetic-2"></a>with synthetic</h1>
<p>sns.countplot(<br>
x=df_mitbih_new[‘label’],<br>
ax=axs[1],<br>
palette=“bright”,<br>
order=df_mitbih_new[‘label’].value_counts().index<br>
)<br>
axs[1].set_xticklabels(axs[1].get_xticklabels(), rotation=15);<br>
axs[1].set_title(“After”, fontsize=15)<br>
for percentage, count, p in zip(<br>
percentages2,<br>
df_mitbih_new[‘label’].value_counts(sort=True).values,<br>
axs[1].patches):</p>
<pre><code>percentage = f'{np.round(percentage, 2)}%'
x = p.get_x() + p.get_width() / 2 - 0.4
y = p.get_y() + p.get_height()
axs[1].annotate(str(percentage)+" / "+str(count), (x, y), fontsize=10, fontweight='bold')
</code></pre>
<p><span class="hashtag">#plt</span>.suptitle(“Balanced Sampling between classes”, fontsize=20, weight=“bold”, y=1.01)<br>
plt.savefig(‘data_dist.png’, facecolor=‘w’, edgecolor=‘w’, format=‘png’,<br>
transparent=False, bbox_inches=‘tight’, pad_inches=0.1)<br>
plt.savefig(‘data_dist.svg’, facecolor=‘w’, edgecolor=‘w’, format=‘svg’,<br>
transparent=False, bbox_inches=‘tight’, pad_inches=0.1)<br>
N = 5<br>
samples = [df_mitbih.loc[df_mitbih[‘class’] == cls].sample(N) for cls in range(N)]<br>
titles = [id_to_label[cls] for cls in range(5)]</p>
<p>with plt.style.context(“seaborn-white”):<br>
fig, axs = plt.subplots(3, 2, figsize=(20, 7))<br>
for i in range(5):<br>
ax = axs.flat[i]<br>
ax.plot(samples[i].values[:,:-2].transpose())<br>
ax.set_title(titles[i])<br>
<span class="hashtag">#plt</span>.ylabel(“Amplitude”)</p>
<pre><code>plt.tight_layout()
plt.suptitle("ECG Signals", fontsize=20, y=1.05, weight="bold")
plt.savefig(f"signals_per_class.svg",
                format="svg",bbox_inches='tight', pad_inches=0.2)
plt.savefig(f"signals_per_class.png", 
                format="png",bbox_inches='tight', pad_inches=0.2)
#%time
</code></pre>
<p>signals = [’ '.join(df_mitbih.iloc[i, :-1].apply(str).values) for i in range(df_mitbih.shape[0])]<br>
y = df_mitbih.iloc[:, -1].values.tolist()<br>
print(len(signals), len(y))</p>
<p>print(f’data has {len(set([sig for line in signals for sig in line.split()]))} out of 16 372 411 unique values.’)<br>
class ECGDataset(Dataset):</p>
<pre><code>def __init__(self, df):
    self.df = df
    self.data_columns = self.df.columns[:-2].tolist()

def __getitem__(self, idx):
    signal = self.df.loc[idx, self.data_columns].astype('float32')
    signal = torch.FloatTensor([signal.values])                 
    target = torch.LongTensor(np.array(self.df.loc[idx, 'class']))
    return signal, target

def __len__(self):
    return len(self.df)
</code></pre>
<p>def get_dataloader(phase: str, batch_size: int = 96) → DataLoader:<br>
‘’’<br>
Dataset and DataLoader.<br>
Parameters:<br>
pahse: training or validation phase.<br>
batch_size: data per iteration.<br>
Returns:<br>
data generator<br>
‘’’<br>
df = pd.read_csv(config.train_csv_path)<br>
train_df, val_df = train_test_split(<br>
df, test_size=0.15, random_state=config.seed, stratify=df[‘label’]<br>
)<br>
train_df, val_df = train_df.reset_index(drop=True), val_df.reset_index(drop=True)<br>
df = train_df if phase == ‘train’ else val_df<br>
dataset = ECGDataset(df)<br>
dataloader = DataLoader(dataset=dataset, batch_size=batch_size, num_workers=4)<br>
return dataloader<br>
class Swish(nn.Module):<br>
def forward(self, x):<br>
return x * torch.sigmoid(x)</p>
<p>x = torch.linspace(-10.0, 10.0, 100)<br>
swish = Swish()<br>
swish_out = swish(x)<br>
relu_out = torch.relu(x)</p>
<p>plt.title(‘Swish function’)<br>
plt.plot(x.numpy(), swish_out.numpy(), label=‘Swish’)<br>
plt.plot(x.numpy(), relu_out.numpy(), label=‘ReLU’)<br>
plt.legend();<br>
plt.show()<br>
class ConvNormPool(nn.Module):<br>
“”“Conv Skip-connection module”""<br>
def <strong>init</strong>(<br>
self,<br>
input_size,<br>
hidden_size,<br>
kernel_size,<br>
norm_type=‘bachnorm’<br>
):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.kernel_size = kernel_size
    self.conv_1 = nn.Conv1d(
        in_channels=input_size,
        out_channels=hidden_size,
        kernel_size=kernel_size
    )
    self.conv_2 = nn.Conv1d(
        in_channels=hidden_size,
        out_channels=hidden_size,
        kernel_size=kernel_size
    )
    self.conv_3 = nn.Conv1d(
        in_channels=hidden_size,
        out_channels=hidden_size,
        kernel_size=kernel_size
    )
    self.swish_1 = Swish()
    self.swish_2 = Swish()
    self.swish_3 = Swish()
    if norm_type == 'group':
        self.normalization_1 = nn.GroupNorm(
            num_groups=8,
            num_channels=hidden_size
        )
        self.normalization_2 = nn.GroupNorm(
            num_groups=8,
            num_channels=hidden_size
        )
        self.normalization_3 = nn.GroupNorm(
            num_groups=8,
            num_channels=hidden_size
        )
    else:
        self.normalization_1 = nn.BatchNorm1d(num_features=hidden_size)
        self.normalization_2 = nn.BatchNorm1d(num_features=hidden_size)
        self.normalization_3 = nn.BatchNorm1d(num_features=hidden_size)
        
    self.pool = nn.MaxPool1d(kernel_size=2)
def forward(self, input):
    conv1 = self.conv_1(input)
    x = self.normalization_1(conv1)
    x= self.swish_1(x)
    x = F.pad(x, pad=(self.kernel_size - 1, 0))
    
    x = self.conv_2(x)
    x = self.normalization_2(x)
    x = self.swish_2(x)
    x = F.pad(x, pad=(self.kernel_size - 1, 0))
    
    conv3 = self.conv_3(x)
    x = self.normalization_3(conv1+conv3)
    x = self.swish_3(x)
    x = F.pad(x, pad=(self.kernel_size - 1, 0))   
    
    x = self.pool(x)
    return x
</code></pre>
<p>class CNN(nn.Module):<br>
def <strong>init</strong>(<br>
self,<br>
input_size = 1,<br>
hid_size = 256,<br>
kernel_size = 5,<br>
num_classes = 5,<br>
):</p>
<pre><code>    super().__init__()
    
    self.conv1 = ConvNormPool(
        input_size=input_size,
        hidden_size=hid_size,
        kernel_size=kernel_size,
    )
    self.conv2 = ConvNormPool(
        input_size=hid_size,
        hidden_size=hid_size//2,
        kernel_size=kernel_size,
    )
    self.conv3 = ConvNormPool(
        input_size=hid_size//2,
        hidden_size=hid_size//4,
        kernel_size=kernel_size,
    )
    self.avgpool = nn.AdaptiveAvgPool1d((1))
    self.fc = nn.Linear(in_features=hid_size//4, out_features=num_classes)
    
def forward(self, input):
    x = self.conv1(input)
    x = self.conv2(x)
    x = self.conv3(x)
    x = self.avgpool(x)        
    # print(x.shape) # num_features * num_channels
    x = x.view(-1, x.size(1) * x.size(2))
    x = F.softmax(self.fc(x), dim=1)
    return x
</code></pre>
<p>class RNN(nn.Module):<br>
“”“RNN module(cell type lstm or gru)”""<br>
def <strong>init</strong>(<br>
self,<br>
input_size,<br>
hid_size,<br>
num_rnn_layers=1,<br>
dropout_p = 0.2,<br>
bidirectional = False,<br>
rnn_type = ‘lstm’,<br>
):<br>
super().<strong>init</strong>()</p>
<pre><code>    if rnn_type == 'lstm':
        self.rnn_layer = nn.LSTM(
            input_size=input_size,
            hidden_size=hid_size,
            num_layers=num_rnn_layers,
            dropout=dropout_p if num_rnn_layers&gt;1 else 0,
            bidirectional=bidirectional,
            batch_first=True,
        )
    else:
        self.rnn_layer = nn.GRU(
            input_size=input_size,
            hidden_size=hid_size,
            num_layers=num_rnn_layers,
            dropout=dropout_p if num_rnn_layers&gt;1 else 0,
            bidirectional=bidirectional,
            batch_first=True,
        )
def forward(self, input):
    outputs, hidden_states = self.rnn_layer(input)
    return outputs, hidden_states
</code></pre>
<p>class RNNModel(nn.Module):<br>
def <strong>init</strong>(<br>
self,<br>
input_size,<br>
hid_size,<br>
rnn_type,<br>
bidirectional,<br>
n_classes=5,<br>
kernel_size=5,<br>
):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.rnn_layer = RNN(
        input_size=46,#hid_size * 2 if bidirectional else hid_size,
        hid_size=hid_size,
        rnn_type=rnn_type,
        bidirectional=bidirectional
    )
    self.conv1 = ConvNormPool(
        input_size=input_size,
        hidden_size=hid_size,
        kernel_size=kernel_size,
    )
    self.conv2 = ConvNormPool(
        input_size=hid_size,
        hidden_size=hid_size,
        kernel_size=kernel_size,
    )
    self.avgpool = nn.AdaptiveAvgPool1d((1))
    self.fc = nn.Linear(in_features=hid_size, out_features=n_classes)
def forward(self, input):
    x = self.conv1(input)
    x = self.conv2(x)
    x, _ = self.rnn_layer(x)
    x = self.avgpool(x)
    x = x.view(-1, x.size(1) * x.size(2))
    x = F.softmax(self.fc(x), dim=1)#.squeeze(1)
    return x
</code></pre>
<p>class RNNAttentionModel(nn.Module):<br>
def <strong>init</strong>(<br>
self,<br>
input_size,<br>
hid_size,<br>
rnn_type,<br>
bidirectional,<br>
n_classes=5,<br>
kernel_size=5,<br>
):<br>
super().<strong>init</strong>()<br>
self.rnn_layer = RNN(<br>
input_size=46,<br>
hid_size=hid_size,<br>
rnn_type=rnn_type,<br>
bidirectional=bidirectional<br>
)<br>
self.conv1 = ConvNormPool(<br>
input_size=input_size,<br>
hidden_size=hid_size,<br>
kernel_size=kernel_size,<br>
)<br>
self.conv2 = ConvNormPool(<br>
input_size=hid_size,<br>
hidden_size=hid_size,<br>
kernel_size=kernel_size,<br>
)<br>
self.avgpool = nn.AdaptiveMaxPool1d((1))<br>
self.attn = nn.Linear(hid_size, hid_size, bias=False)<br>
self.fc = nn.Linear(in_features=hid_size, out_features=n_classes)</p>
<pre><code>def forward(self, input):
    x = self.conv1(input)
    x = self.conv2(x)
    x_out, hid_states = self.rnn_layer(x)
    x = torch.cat([hid_states[0], hid_states[1]], dim=0).transpose(0, 1)
    x_attn = torch.tanh(self.attn(x))
    x = x_attn.bmm(x_out)
    x = x.transpose(2, 1)
    x = self.avgpool(x)
    x = x.view(-1, x.size(1) * x.size(2))
    x = F.softmax(self.fc(x), dim=-1)
    return x
</code></pre>
<p>class Meter:<br>
def <strong>init</strong>(self, n_classes=5):<br>
self.metrics = {}<br>
self.confusion = torch.zeros((n_classes, n_classes))</p>
<pre><code>def update(self, x, y, loss):
    x = np.argmax(x.detach().cpu().numpy(), axis=1)
    y = y.detach().cpu().numpy()
    self.metrics['loss'] += loss
    self.metrics['accuracy'] += accuracy_score(x,y)
    self.metrics['f1'] += f1_score(x,y,average='macro')
    self.metrics['precision'] += precision_score(x, y, average='macro', zero_division=1)
    self.metrics['recall'] += recall_score(x,y, average='macro', zero_division=1)
    
     
    self._compute_cm(x, y)
    
def _compute_cm(self, x, y):
    for prob, target in zip(x, y):
        if prob == target:
            self.confusion[target][target] += 1
        else:
            self.confusion[target][prob] += 1

def init_metrics(self):
    self.metrics['loss'] = 0
    self.metrics['accuracy'] = 0
    self.metrics['f1'] = 0
    self.metrics['precision'] = 0
    self.metrics['recall'] = 0
    
def get_metrics(self):
    return self.metrics

def get_confusion_matrix(self):
    return self.confusion
</code></pre>
<p>class Trainer:<br>
def <strong>init</strong>(self, net, lr, batch_size, num_epochs):<br>
<a href="http://self.net" rel="noopener nofollow ugc">self.net</a> = net.to(config.device)<br>
self.num_epochs = num_epochs<br>
self.criterion = nn.CrossEntropyLoss()<br>
self.optimizer = AdamW(self.net.parameters(), lr=lr)<br>
self.scheduler = CosineAnnealingLR(self.optimizer, T_max=num_epochs, eta_min=5e-6)<br>
self.best_loss = float(‘inf’)<br>
self.phases = [‘train’, ‘val’]<br>
self.dataloaders = {<br>
phase: get_dataloader(phase, batch_size) for phase in self.phases<br>
}<br>
self.train_df_logs = pd.DataFrame()<br>
self.val_df_logs = pd.DataFrame()</p>
<pre><code>def _train_epoch(self, phase):
    print(f"{phase} mode | time: {time.strftime('%H:%M:%S')}")
    
    self.net.train() if phase == 'train' else self.net.eval()
    meter = Meter()
    meter.init_metrics()
    
    for i, (data, target) in enumerate(self.dataloaders[phase]):
        data = data.to(config.device)
        target = target.to(config.device)
        
        output = self.net(data)
        loss = self.criterion(output, target)
                    
        if phase == 'train':
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()
        
        meter.update(output, target, loss.item())
         
    metrics = meter.get_metrics()
    metrics = {k:v / i for k, v in metrics.items()}
    df_logs = pd.DataFrame([metrics])
    confusion_matrix = meter.get_confusion_matrix()
    
    if phase == 'train':
        self.train_df_logs = pd.concat([self.train_df_logs, df_logs], axis=0)
    else:
        self.val_df_logs = pd.concat([self.val_df_logs, df_logs], axis=0)
    
    # show logs
    print('{}: {}, {}: {}, {}: {}, {}: {}, {}: {}'
          .format(*(x for kv in metrics.items() for x in kv))
         )
    fig, ax = plt.subplots(figsize=(5, 5))
    cm_ = ax.imshow(confusion_matrix, cmap='hot')
    ax.set_title('Confusion matrix', fontsize=15)
    ax.set_xlabel('Actual', fontsize=13)
    ax.set_ylabel('Predicted', fontsize=13)
    plt.colorbar(cm_)
    plt.show()
    return loss

def run(self):
    for epoch in range(self.num_epochs):
        self._train_epoch(phase='train')
        with torch.no_grad():
            val_loss = self._train_epoch(phase='val')
            self.scheduler.step()
        
        if val_loss &lt; self.best_loss:
            self.best_loss = val_loss
            print('\nNew checkpoint\n')
            self.best_loss = val_loss
            torch.save(self.net.state_dict(), f"best_model_epoc{epoch}.pth")
        #clear_output()
</code></pre>
<p><span class="hashtag">#model</span> = RNNAttentionModel(1, 64, ‘lstm’, False)<br>
<span class="hashtag">#model</span> = RNNModel(1, 64, ‘lstm’, True)</p>
<p><span class="hashtag">#model</span> = RNNAttentionModel(1, 64, ‘lstm’, False)<br>
<span class="hashtag">#model</span> = RNNModel(1, 64, ‘lstm’, True)<br>
model = CNN(num_classes=5, hid_size=128)<br>
trainer = Trainer(net=model, lr=1e-3, batch_size=96, num_epochs=10)<span class="hashtag">#100</span>)<br>
trainer.run()<br>
train_logs = trainer.train_df_logs<br>
train_logs.columns = [“train_”+ colname for colname in train_logs.columns]<br>
val_logs = trainer.val_df_logs<br>
val_logs.columns = [“val_”+ colname for colname in val_logs.columns]</p>
<p>logs = pd.concat([train_logs,val_logs], axis=1)<br>
logs.reset_index(drop=True, inplace=True)<br>
logs = logs.loc[:, [<br>
‘train_loss’, ‘val_loss’,<br>
‘train_accuracy’, ‘val_accuracy’,<br>
‘train_f1’, ‘val_f1’,<br>
‘train_precision’, ‘val_precision’,<br>
‘train_recall’, ‘val_recall’]<br>
]<br>
logs.head()<br>
logs.to_csv(‘cnn.csv’, index=False)<br>
cnn_model = CNN(num_classes=5, hid_size=128).to(config.device)<br>
cnn_model.load_state_dict(<br>
torch.load(config.cnn_state_path,<br>
map_location=config.device)<br>
);<br>
cnn_model.eval();<br>
logs = pd.read_csv(config.cnn_logs)<br>
colors = [’<span class="hashtag">#C042FF</span>’, ‘<span class="hashtag">#03C576FF</span>’, ‘<span class="hashtag">#FF355A</span>’, ‘<span class="hashtag">#03C5BF</span>’, ‘<span class="hashtag">#96C503</span>’, ‘<span class="hashtag">#C5035B</span>’]<br>
palettes = [sns.color_palette(colors, 2),<br>
sns.color_palette(colors, 4),<br>
sns.color_palette(colors[:2]+colors[-2:] + colors[2:-2], 6)]</p>
<p>fig, ax = plt.subplots(1, 2, figsize=(12, 4))</p>
<p>sns.lineplot(data=logs.iloc[:, :2], palette=palettes[0], markers=True, ax=ax[0], linewidth=2.5,)<br>
ax[0].set_title(“Loss Function during Model Training”, fontsize=14)<br>
ax[0].set_xlabel(“Epoch”, fontsize=14)</p>
<p>sns.lineplot(data=logs.iloc[:, 2:6], palette=palettes[1], markers=True, ax=ax[1], linewidth=2.5, legend=“full”)<br>
ax[1].set_title(“Metrics during Model Training”, fontsize=15)<br>
ax[1].set_xlabel(“Epoch”, fontsize=14)</p>
<p>plt.suptitle(‘CNN Model’, fontsize=18)</p>
<p>plt.tight_layout()<br>
fig.savefig(“cnn.png”, format=“png”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
fig.savefig(“cnn.svg”, format=“svg”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
lstm_model = RNNModel(1, 64, ‘lstm’, True).to(config.device)<br>
lstm_model.load_state_dict(<br>
torch.load(config.lstm_state_path,<br>
map_location=config.device)<br>
);<br>
lstm_model.eval();<br>
logs = pd.read_csv(config.lstm_logs)<br>
colors = [’<span class="hashtag">#C042FF</span>’, ‘<span class="hashtag">#03C576FF</span>’, ‘<span class="hashtag">#FF355A</span>’, ‘<span class="hashtag">#03C5BF</span>’, ‘<span class="hashtag">#96C503</span>’, ‘<span class="hashtag">#C5035B</span>’]<br>
palettes = [sns.color_palette(colors, 2),<br>
sns.color_palette(colors, 4),<br>
sns.color_palette(colors[:2]+colors[-2:] + colors[2:-2], 6)]</p>
<p>fig, ax = plt.subplots(1, 2, figsize=(12, 4))</p>
<p>sns.lineplot(data=logs.iloc[:, :2], palette=palettes[0], markers=True, ax=ax[0], linewidth=2.5,)<br>
ax[0].set_title(“Loss Function during Model Training”, fontsize=14)<br>
ax[0].set_xlabel(“Epoch”, fontsize=14)</p>
<p>sns.lineplot(data=logs.iloc[:, 2:6], palette=palettes[1], markers=True, ax=ax[1], linewidth=2.5, legend=“full”)<br>
ax[1].set_title(“Metrics during Model Training”, fontsize=15)<br>
ax[1].set_xlabel(“Epoch”, fontsize=14)</p>
<p>plt.suptitle(‘CNN+LSTM Model’, fontsize=18)</p>
<p>plt.tight_layout()<br>
fig.savefig(“lstm.png”, format=“png”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
fig.savefig(“lstm.svg”, format=“svg”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
attn_model = RNNAttentionModel(1, 64, ‘lstm’, False).to(config.device)<br>
attn_model.load_state_dict(<br>
torch.load(config.attn_state_path,<br>
map_location=config.device)<br>
);<br>
attn_model.eval();<br>
logs = pd.read_csv(config.attn_logs)<br>
colors = [’<span class="hashtag">#C042FF</span>’, ‘<span class="hashtag">#03C576FF</span>’, ‘<span class="hashtag">#FF355A</span>’, ‘<span class="hashtag">#03C5BF</span>’, ‘<span class="hashtag">#96C503</span>’, ‘<span class="hashtag">#C5035B</span>’]<br>
palettes = [sns.color_palette(colors, 2),<br>
sns.color_palette(colors, 4),<br>
sns.color_palette(colors[:2]+colors[-2:] + colors[2:-2], 6)]</p>
<p>fig, ax = plt.subplots(1, 2, figsize=(12, 4))</p>
<p>sns.lineplot(data=logs.iloc[:, :2], palette=palettes[0], markers=True, ax=ax[0], linewidth=2.5,)<br>
ax[0].set_title(“Loss Function during Model Training”, fontsize=14)<br>
ax[0].set_xlabel(“Epoch”, fontsize=14)</p>
<p>sns.lineplot(data=logs.iloc[:, 2:6], palette=palettes[1], markers=True, ax=ax[1], linewidth=2.5, legend=“full”)<br>
ax[1].set_title(“Metrics during Model Training”, fontsize=15)<br>
ax[1].set_xlabel(“Epoch”, fontsize=14)</p>
<p>plt.suptitle(‘CNN+LSTM+Attention Model’, fontsize=18)</p>
<p>plt.tight_layout()<br>
fig.savefig(“attn.png”, format=“png”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
fig.savefig(“attn.svg”, format=“svg”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
test_df = pd.read_csv(config.test_csv_path)<br>
print(test_df.shape)<br>
test_dataset = ECGDataset(test_df)<br>
test_dataloader = DataLoader(dataset=test_dataset, batch_size=96, num_workers=0, shuffle=False)<br>
def make_test_stage(dataloader, model, probs=False):<br>
cls_predictions = []<br>
cls_ground_truths = []</p>
<pre><code>for i, (data, cls_target) in enumerate(dataloader):
    with torch.no_grad():

        data = data.to(config.device)
        cls_target = cls_target.cpu()
        cls_prediction = model(data)
        
        if not probs:
            cls_prediction = torch.argmax(cls_prediction, dim=1)

        cls_predictions.append(cls_prediction.detach().cpu())
        cls_ground_truths.append(cls_target)

predictions_cls = torch.cat(cls_predictions).numpy()
ground_truths_cls = torch.cat(cls_ground_truths).numpy()
return predictions_cls, ground_truths_cls
</code></pre>
<p>models = [cnn_model, lstm_model, attn_model]<br>
y_pred, y_true = make_test_stage(test_dataloader, models[0])<br>
y_pred.shape, y_true.shape<br>
report = pd.DataFrame(<br>
classification_report(<br>
y_pred,<br>
y_true,<br>
output_dict=True<br>
)<br>
).transpose()<br>
colors = [’<span class="hashtag">#00FA9A</span>’, ‘<span class="hashtag">#D2B48C</span>’, ‘<span class="hashtag">#FF69B4</span>’]<span class="hashtag">#random</span>.choices(list(mcolors.CSS4_COLORS.values()), k = 3)<br>
report_plot = report.apply(lambda x: x*100)<br>
ax = report_plot[[“precision”, “recall”, “f1-score”]].plot(kind=‘bar’,<br>
figsize=(13, 4), legend=True, fontsize=15, color=colors)</p>
<p>ax.set_xlabel(“Estimators”, fontsize=15)<br>
ax.set_xticklabels(<br>
list(id_to_label.values())+[“accuracy avg”, “marco avg”, “weighted avg”],<br>
rotation=15, fontsize=11)<br>
ax.set_ylabel(“Percentage”, fontsize=15)<br>
plt.title(“CNN Model Classification Report”, fontsize=20)</p>
<p>for percentage, p in zip(<br>
report[[‘precision’, ‘recall’, ‘f1-score’]].values,<br>
ax.patches):<br>
percentage = " ".join([str(round(i<em>100, 2))+"%" for i in percentage])<br>
x = p.get_x() + p.get_width() - 0.4<br>
y = p.get_y() + p.get_height() / 4<br>
ax.annotate(percentage, (x, y), fontsize=8, rotation=15, fontweight=‘bold’)<br>
fig.savefig(“cnn_report.png”, format=“png”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
fig.savefig(“cnn_report.svg”, format=“svg”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
plt.show()<br>
y_pred, y_true = make_test_stage(test_dataloader, models[1])<br>
y_pred.shape, y_true.shape<br>
report = pd.DataFrame(<br>
classification_report(<br>
y_pred,<br>
y_true,<br>
output_dict=True<br>
)<br>
).transpose()<br>
colors = [’<span class="hashtag">#00FA9A</span>’, ‘<span class="hashtag">#D2B48C</span>’, ‘<span class="hashtag">#FF69B4</span>’]<span class="hashtag">#random</span>.choices(list(mcolors.CSS4_COLORS.values()), k = 3)<br>
report_plot = report.apply(lambda x: x</em>100)<br>
ax = report_plot[[“precision”, “recall”, “f1-score”]].plot(kind=‘bar’,<br>
figsize=(13, 4), legend=True, fontsize=15, color=colors)</p>
<p>ax.set_xlabel(“Estimators”, fontsize=15)<br>
ax.set_xticklabels(<br>
list(id_to_label.values())+[“accuracy avg”, “marco avg”, “weighted avg”],<br>
rotation=15, fontsize=11)<br>
ax.set_ylabel(“Percentage”, fontsize=15)<br>
plt.title(“CNN+LSTM Model Classification Report”, fontsize=20)</p>
<p>for percentage, p in zip(<br>
report[[‘precision’, ‘recall’, ‘f1-score’]].values,<br>
ax.patches):</p>
<pre><code>percentage = " ".join([str(round(i*100, 2))+"%" for i in percentage])
x = p.get_x() + p.get_width() - 0.4
y = p.get_y() + p.get_height() / 4
ax.annotate(percentage, (x, y), fontsize=8, rotation=15, fontweight='bold')
fig.savefig("lstm_report.png", format="png",  pad_inches=0.2, transparent=False, bbox_inches='tight')
</code></pre>
<p>fig.savefig(“lstm_report.svg”, format=“svg”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
plt.show()<br>
y_pred, y_true = make_test_stage(test_dataloader, models[2])<br>
y_pred.shape, y_true.shape<br>
report = pd.DataFrame(<br>
classification_report(<br>
y_pred,<br>
y_true,<br>
output_dict=True<br>
)<br>
).transpose()<br>
colors = [’<span class="hashtag">#00FA9A</span>’, ‘<span class="hashtag">#D2B48C</span>’, ‘<span class="hashtag">#FF69B4</span>’]<span class="hashtag">#random</span>.choices(list(mcolors.CSS4_COLORS.values()), k = 3)<br>
report_plot = report.apply(lambda x: x*100)<br>
ax = report_plot[[“precision”, “recall”, “f1-score”]].plot(kind=‘bar’,<br>
figsize=(13, 4), legend=True, fontsize=15, color=colors)</p>
<p>ax.set_xlabel(“Estimators”, fontsize=15)<br>
ax.set_xticklabels(<br>
list(id_to_label.values())+[“accuracy avg”, “marco avg”, “weighted avg”],<br>
rotation=15, fontsize=11)<br>
ax.set_ylabel(“Percentage”, fontsize=15)<br>
plt.title(“CNN+LSTM+Attention Model Classification Report”, fontsize=20)</p>
<p>for percentage, p in zip(<br>
report[[‘precision’, ‘recall’, ‘f1-score’]].values,<br>
ax.patches):<br>
percentage = " “.join([str(round(i*100, 2))+”%" for i in percentage])<br>
x = p.get_x() + p.get_width() - 0.4<br>
y = p.get_y() + p.get_height() / 4<br>
ax.annotate(percentage, (x, y), fontsize=8, rotation=15, fontweight=‘bold’)<br>
fig.savefig(“attn_report.png”, format=“png”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
fig.savefig(“attn_report.svg”, format=“svg”,  pad_inches=0.2, transparent=False, bbox_inches=‘tight’)<br>
plt.show()<br>
y_pred = np.zeros((y_pred.shape[0], 5), dtype=np.float32)<br>
for i, model in enumerate(models, 1):<br>
y_pred_, y_true = make_test_stage(test_dataloader, model, True)<br>
y_pred += y_pred_<br>
y_pred /= i<br>
y_pred = np.argmax(y_pred, axis=1)<br>
clf_report = classification_report(y_pred,<br>
y_true,<br>
labels=[0,1,2,3,4],<br>
target_names=list(id_to_label.values()),#[‘N’, ‘S’, ‘V’, ‘F’, ‘Q’],<br>
output_dict=True)</p>
<p>plt.figure(figsize=(10, 8))<br>
ax = sns.heatmap(pd.DataFrame(clf_report).iloc[:-1, :].T, annot=True)<br>
ax.set_xticklabels(ax.get_xticklabels(),fontsize=15)<br>
ax.set_yticklabels(ax.get_yticklabels(),fontsize=12, rotation=0)<br>
plt.title(“Ensemble Classification Report”, fontsize=20)<br>
plt.savefig(f"ensemble result.svg",format=“svg”,bbox_inches=‘tight’, pad_inches=0.2)<br>
plt.savefig(f"ensemble result.png", format=“png”,bbox_inches=‘tight’, pad_inches=0.2)<br>
clf_report</p>