Segmentation fault (core dumped) problem

Gorgen: <p>run the command “python train.py”, the issues “OpenBLAS blas_thread_init: pthread_create failed for thread 6 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 7 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 8 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 9 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 10 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 11 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 12 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 13 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 14 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 15 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 16 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 17 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 18 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 19 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 20 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 21 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 23 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 24 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 25 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 26 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 27 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 28 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 29 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 30 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 31 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 32 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 33 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 34 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 35 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 36 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 37 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 38 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 39 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 40 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 41 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 42 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 43 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 44 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 45 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 46 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 47 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 48 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 49 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 50 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 51 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 52 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 53 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 54 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 55 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 56 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 57 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 58 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 59 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 60 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 61 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 62 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
OpenBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Resource temporarily unavailable<br>
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1156891 max<br>
Traceback (most recent call last):<br>
File “train.py”, line 16, in <br>
from transformers import get_linear_schedule_with_warmup<br>
File “”, line 1032, in _handle_fromlist<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/transformers/utils/import_utils.py”, line 847, in <strong>getattr</strong><br>
module = self._get_module(self._class_to_module[name])<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/transformers/utils/import_utils.py”, line 857, in _get_module<br>
return importlib.import_module(”." + module_name, self.<strong>name</strong>)<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/importlib/<strong>init</strong>.py”, line 127, in import_module<br>
return _bootstrap._gcd_import(name[level:], package, level)<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/transformers/optimization.py”, line 26, in <br>
from .trainer_utils import SchedulerType<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer_utils.py”, line 46, in <br>
import tensorflow as tf<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/tensorflow/<strong>init</strong>.py”, line 473, in <br>
keras._load()<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/tensorflow/python/util/lazy_loader.py”, line 41, in _load<br>
module = importlib.import_module(self.<strong>name</strong>)<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/importlib/<strong>init</strong>.py”, line 127, in import_module<br>
return _bootstrap._gcd_import(name[level:], package, level)<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/<strong>init</strong>.py”, line 25, in <br>
from keras import models<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/models.py”, line 20, in <br>
from keras import metrics as metrics_module<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/metrics.py”, line 24, in <br>
from keras import activations<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/activations.py”, line 20, in <br>
from keras.layers import advanced_activations<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/layers/<strong>init</strong>.py”, line 30, in <br>
from keras.layers.preprocessing.image_preprocessing import CenterCrop<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/layers/preprocessing/image_preprocessing.py”, line 24, in <br>
from keras.preprocessing.image import smart_resize<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/preprocessing/<strong>init</strong>.py”, line 22, in <br>
from keras.preprocessing import image<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/preprocessing/image.py”, line 22, in <br>
from keras_preprocessing import image<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras_preprocessing/image/<strong>init</strong>.py”, line 5, in <br>
from .affine_transformations import *<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras_preprocessing/image/affine_transformations.py”, line 15, in <br>
from scipy import ndimage<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/scipy/ndimage/<strong>init</strong>.py”, line 153, in <br>
from .interpolation import *<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/scipy/ndimage/interpolation.py”, line 37, in <br>
from scipy import special<br>
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/scipy/special/<strong>init</strong>.py”, line 643, in <br>
from . import _ufuncs<br>
File “”, line 416, in parent<br>
KeyboardInterrupt<br>
Segmentation fault (core dumped)", how to solve it? thanks.</p>
<p>the train.py as follows “# -<em>- coding: utf-8 -</em>-<br>
import os<br>
os.environ[“CUDA_VISIBLE_DEVICES”]= ‘2’<br>
from tqdm import tqdm<br>
import os<br>
import random<br>
import torch<br>
import torch.nn as nn</p>
<p>from transformers import RobertaTokenizer<br>
from ERC_dataset import MELD_loader, Emory_loader, IEMOCAP_loader, DD_loader<br>
from model import ERC_model</p>
<h1>
<a name="from-erccombined-import-erc_model-1" class="anchor" href="#from-erccombined-import-erc_model-1"></a>from ERCcombined import ERC_model</h1>
<p>from torch.utils.data import Dataset, DataLoader<br>
from transformers import get_linear_schedule_with_warmup<br>
import pdb<br>
import argparse, logging<br>
from sklearn.metrics import precision_recall_fscore_support</p>
<p>from utils import make_batch_roberta, make_batch_bert, make_batch_gpt</p>
<p>def CELoss(pred_outs, labels):<br>
“”"<br>
pred_outs: [batch, clsNum]<br>
labels: [batch]<br>
“”"<br>
loss = nn.CrossEntropyLoss()<br>
loss_val = loss(pred_outs, labels)<br>
return loss_val</p>
<h2>
<a name="finetune-robeta-large-2" class="anchor" href="#finetune-robeta-large-2"></a>finetune RoBETa-large</h2>
<p>def main():<br>
“”“Dataset Loading”“”<br>
batch_size = args.batch<br>
dataset = args.dataset<br>
dataclass = args.cls<br>
sample = args.sample<br>
model_type = args.pretrained<br>
freeze = args.freeze<br>
initial = args.initial</p>
<pre><code>dataType = 'multi'
if dataset == 'MELD':
    if args.dyadic:
        dataType = 'dyadic'
    else:
        dataType = 'multi'
    data_path = './dataset/MELD/'+dataType+'/'
    DATA_loader = MELD_loader
elif dataset == 'EMORY':
    data_path = './dataset/EMORY/'
    DATA_loader = Emory_loader
elif dataset == 'iemocap':
    data_path = './dataset/iemocap/'
    DATA_loader = IEMOCAP_loader
elif dataset == 'dailydialog':
    data_path = './dataset/dailydialog/'
    DATA_loader = DD_loader    
    
if 'roberta' in model_type:
    make_batch = make_batch_roberta
elif model_type == 'bert-large-uncased':
    make_batch = make_batch_bert
else:
    make_batch = make_batch_gpt  
    
if freeze:
    freeze_type = 'freeze'
else:
    freeze_type = 'no_freeze'
    
train_path = data_path + dataset+'_train.txt'
dev_path = data_path + dataset+'_dev.txt'
test_path = data_path + dataset+'_test.txt'
        
train_dataset = DATA_loader(train_path, dataclass)
if sample &lt; 1.0:
    train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=False, num_workers=4, collate_fn=make_batch)
else:
    train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=4, collate_fn=make_batch)
train_sample_num = int(len(train_dataloader)*sample)

dev_dataset = DATA_loader(dev_path, dataclass)
dev_dataloader = DataLoader(dev_dataset, batch_size=1, shuffle=False, num_workers=4, collate_fn=make_batch)

test_dataset = DATA_loader(test_path, dataclass)
test_dataloader = DataLoader(test_dataset, batch_size=1, shuffle=False, num_workers=4, collate_fn=make_batch)

"""logging and path"""
save_path = os.path.join(dataset+'_models', model_type, initial, freeze_type, dataclass, str(sample))

print("###Save Path### ", save_path)
log_path = os.path.join(save_path, 'train.log')
if not os.path.exists(save_path):
    os.makedirs(save_path)
fileHandler = logging.FileHandler(log_path)

logger.addHandler(streamHandler)
logger.addHandler(fileHandler)    
logger.setLevel(level=logging.DEBUG)      

"""Model Loading"""
if 'gpt2' in model_type:
    last = True
else:
    last = False
    
print('DataClass: ', dataclass, '!!!') # emotion
clsNum = len(train_dataset.labelList)
model = ERC_model(model_type, clsNum, last, freeze, initial)
model = model.cuda()    
model.train() 

"""Training Setting"""        
training_epochs = args.epoch
save_term = int(training_epochs/5)
max_grad_norm = args.norm
lr = args.lr
num_training_steps = len(train_dataset)*training_epochs
num_warmup_steps = len(train_dataset)
# optimizer = torch.optim.AdamW(model.parameters(), lr=lr) # , eps=1e-06, weight_decay=0.01
optimizer = torch.optim.AdamW(model.train_params, lr=lr) # , eps=1e-06, weight_decay=0.01
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps)

"""Input &amp; Label Setting"""
best_dev_fscore, best_test_fscore = 0, 0
best_dev_fscore_macro, best_dev_fscore_micro, best_test_fscore_macro, best_test_fscore_micro = 0, 0, 0, 0    
best_epoch = 0
for epoch in tqdm(range(training_epochs)):
    model.train() 
    for i_batch, data in enumerate(train_dataloader):
        if i_batch &gt; train_sample_num:
            print(i_batch, train_sample_num)
            break
        
        """Prediction"""
        batch_input_tokens, batch_labels, batch_speaker_tokens = data
        batch_input_tokens, batch_labels = batch_input_tokens.cuda(), batch_labels.cuda()
        
        pred_logits = model(batch_input_tokens, batch_speaker_tokens)

        """Loss calculation &amp; training"""
        loss_val = CELoss(pred_logits, batch_labels)
        
        loss_val.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()
        
    """Dev &amp; Test evaluation"""
    model.eval()
    if dataset == 'dailydialog': # micro &amp; macro
        dev_acc, dev_pred_list, dev_label_list = _CalACC(model, dev_dataloader)
        dev_pre_macro, dev_rec_macro, dev_fbeta_macro, _ = precision_recall_fscore_support(dev_label_list, dev_pred_list, average='macro')
        dev_pre_micro, dev_rec_micro, dev_fbeta_micro, _ = precision_recall_fscore_support(dev_label_list, dev_pred_list, labels=[0,1,2,3,5,6], average='micro') # neutral x
        
        dev_fscore = dev_fbeta_macro+dev_fbeta_micro

        """Best Score &amp; Model Save"""
        if dev_fscore &gt; best_dev_fscore_macro + best_dev_fscore_micro:
            best_dev_fscore_macro = dev_fbeta_macro                
            best_dev_fscore_micro = dev_fbeta_micro
            
            test_acc, test_pred_list, test_label_list = _CalACC(model, test_dataloader)
            test_pre_macro, test_rec_macro, test_fbeta_macro, _ = precision_recall_fscore_support(test_label_list, test_pred_list, average='macro')
            test_pre_micro, test_rec_micro, test_fbeta_micro, _ = precision_recall_fscore_support(test_label_list, test_pred_list, labels=[0,1,2,3,5,6], average='micro') # neutral x                
            
            best_epoch = epoch
            _SaveModel(model, save_path)
    else: # weight
        dev_acc, dev_pred_list, dev_label_list = _CalACC(model, dev_dataloader)
        dev_pre, dev_rec, dev_fbeta, _ = precision_recall_fscore_support(dev_label_list, dev_pred_list, average='weighted')

        """Best Score &amp; Model Save"""
        if dev_fbeta &gt; best_dev_fscore:
            best_dev_fscore = dev_fbeta
            
            test_acc, test_pred_list, test_label_list = _CalACC(model, test_dataloader)
            test_pre, test_rec, test_fbeta, _ = precision_recall_fscore_support(test_label_list, test_pred_list, average='weighted')                
            
            best_epoch = epoch
            _SaveModel(model, save_path)
    
    if epoch % 5 == 0:
        logger.info('Epoch: {}'.format(epoch))
        if dataset == 'dailydialog': # micro &amp; macro
            logger.info('Devleopment ## accuracy: {}, macro-fscore: {}, micro-fscore: {}'.format(dev_acc, dev_fbeta_macro, dev_fbeta_micro))
            logger.info('') 
        else:
            logger.info('Devleopment ## accuracy: {}, precision: {}, recall: {}, fscore: {}'.format(dev_acc, dev_pre, dev_rec, dev_fbeta))
            logger.info('')
    
if dataset == 'dailydialog': # micro &amp; macro
    logger.info('Final Fscore ## test-accuracy: {}, test-macro: {}, test-micro: {}, test_epoch: {}'.format(test_acc, test_fbeta_macro, test_fbeta_micro, best_epoch))
else:
    logger.info('Final Fscore ## test-accuracy: {}, test-fscore: {}, test_epoch: {}'.format(test_acc, test_fbeta, best_epoch))         
</code></pre>
<p>def _CalACC(model, dataloader):<br>
model.eval()<br>
correct = 0<br>
label_list = []<br>
pred_list = []</p>
<pre><code># label arragne
with torch.no_grad():
    for i_batch, data in enumerate(dataloader):            
        """Prediction"""
        batch_input_tokens, batch_labels, batch_speaker_tokens = data
        batch_input_tokens, batch_labels = batch_input_tokens.cuda(), batch_labels.cuda()
        
        pred_logits = model(batch_input_tokens, batch_speaker_tokens) # (1, clsNum)
        
        """Calculation"""    
        pred_label = pred_logits.argmax(1).item()
        true_label = batch_labels.item()
        
        pred_list.append(pred_label)
        label_list.append(true_label)
        if pred_label == true_label:
            correct += 1
    acc = correct/len(dataloader)
return acc, pred_list, label_list
</code></pre>
<p>def _SaveModel(model, path):<br>
if not os.path.exists(path):<br>
os.makedirs(path)<br>
torch.save(model.state_dict(), os.path.join(path, ‘model.bin’))</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
torch.cuda.empty_cache()</p>
<pre><code>"""Parameters"""
parser  = argparse.ArgumentParser(description = "Emotion Classifier" )
parser.add_argument( "--batch", type=int, help = "batch_size", default = 1)

parser.add_argument( "--epoch", type=int, help = 'training epohcs', default = 10) # 12 for iemocap
parser.add_argument( "--norm", type=int, help = "max_grad_norm", default = 10)
parser.add_argument( "--lr", type=float, help = "learning rate", default = 1e-6) # 1e-5
parser.add_argument( "--sample", type=float, help = "sampling trainign dataset", default = 1.0) # 

parser.add_argument( "--dataset", help = 'MELD or EMORY or iemocap or dailydialog', default = 'MELD')

parser.add_argument( "--pretrained", help = 'roberta-large or bert-large-uncased or gpt2 or gpt2-large or gpt2-medium', default = 'roberta-large')    
parser.add_argument( "--initial", help = 'pretrained or scratch', default = 'pretrained')
parser.add_argument('-dya', '--dyadic', action='store_true', help='dyadic conversation')
parser.add_argument('-fr', '--freeze', action='store_true', help='freezing PM')
parser.add_argument( "--cls", help = 'emotion or sentiment', default = 'emotion')
    
args = parser.parse_args()

logger = logging.getLogger(__name__)
streamHandler = logging.StreamHandler()

main()
”
</code></pre>

ege_b: <p>I did have a similar error before. The reason was that the batch data elements (dicts in my case) did not have the same keys. E.g. I had</p>
<pre><code class="lang-auto">item1 = { key1: val1, key2: val2} 
</code></pre>
<p>and</p>
<pre><code class="lang-auto">item2 = { key1: val1} 
</code></pre>
<p>with possible further keys and the C Backend throws a segmentation fault due to the differing length/sizes of batch data. I did not recreate your error, although thought this might help. As far as I know, this had something to do with PyArrow &lt;=3.0 and is fixed on later versions.</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/62772819/huggingface-trainer-segmentation-fault">
  <header class="source">

      <a href="https://stackoverflow.com/questions/62772819/huggingface-trainer-segmentation-fault" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/13883836/efe23eds" target="_blank" rel="noopener nofollow ugc">
    <img alt="efe23eds" src="https://www.gravatar.com/avatar/e4f121d5ed07584501da08d41d9e84f0?s=256&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/62772819/huggingface-trainer-segmentation-fault" target="_blank" rel="noopener nofollow ugc">HuggingFace Trainer Segmentation Fault</a>
</h4>

<div class="tags">
  <strong>python, machine-learning, huggingface-transformers, language-model</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/13883836/efe23eds" target="_blank" rel="noopener nofollow ugc">
    efe23eds
  </a>
  on <a href="https://stackoverflow.com/questions/62772819/huggingface-trainer-segmentation-fault" target="_blank" rel="noopener nofollow ugc">10:06AM - 07 Jul 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/huggingface/transformers/issues/5590">
  <header class="source">

      <a href="https://github.com/huggingface/transformers/issues/5590" target="_blank" rel="noopener nofollow ugc">github.com/huggingface/transformers</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/huggingface/transformers/issues/5590" target="_blank" rel="noopener nofollow ugc">HF Trainer Segmentation Fault</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-07-08" data-time="03:58:31" data-timezone="UTC">03:58AM - 08 Jul 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-10-10" data-time="08:14:00" data-timezone="UTC">08:14AM - 10 Oct 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/ksjae" target="_blank" rel="noopener nofollow ugc">
          <img alt="ksjae" src="https://avatars.githubusercontent.com/u/17930170?v=4" class="onebox-avatar-inline" width="20" height="20">
          ksjae
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          wontfix
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container"># 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2-medium &amp; l<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">arge

Language I am using the model on (English, Chinese ...): Korean (with custom trained tokenizer)

The problem arises when using:
* [ O ] the official example scripts: (give details below)
https://huggingface.co/blog/how-to-train
* [ O ] my own modified scripts: (give details below)
```
from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling, LineByLineTextDataset
from transformers import GPT2Config, GPT2LMHeadModel, GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained("./data/TOKEN")

config = GPT2Config.from_pretrained('gpt2-medium')
model = GPT2LMHeadModel(config=config)
tokenizer = GPT2TokenizerFast.from_pretrained("./data/TOKEN", model_max_length=1024)

print('loading dataset...')
dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./data/kowiki.txt",
    block_size=512,
)

training_args = TrainingArguments(
    output_dir='./m',          # output directory
    num_train_epochs=1,              # total # of training epochs
    per_device_train_batch_size=1,  # batch size per device during training - the higher the better, but may OOM
    per_device_eval_batch_size=1,   # batch size for evaluation
    logging_dir='./logs',            # directory for storing logs
    save_steps=10000,
    do_train=True
)

trainer = Trainer(
    model=model,                         # the instantiated Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=dataset,         # training dataset
)
faulthandler.enable()
trainer.train()
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ O ] my own task or dataset: (give details below)
Text generation with prompt (trained on wiki &amp; novel)

## To reproduce

Steps to reproduce the behavior:

1. Modify path to data file
2. Use any file(tested with Korean - UTF8)
3. Use any tokenizer(tested with self &amp; GPT2 tokenizers)

&lt;!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.--&gt;

### Error message
```
loading dataset...
Epoch:   0%|                                              | 0/1 [00:00&lt;?, ?it/s]
Fatal Python error: Segmentation fault                | 0/99996 [00:00&lt;?, ?it/s]

Thread 0x00007f872dfff700 (most recent call first):
  File "/opt/conda/lib/python3.6/threading.py", line 299 in wait
  File "/opt/conda/lib/python3.6/threading.py", line 551 in wait
  File "/opt/conda/lib/python3.6/site-packages/tqdm/_monitor.py", line 69 in run
  File "/opt/conda/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/opt/conda/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f8736bb5700 (most recent call first):
  File "/opt/conda/lib/python3.6/threading.py", line 299 in wait
  File "/opt/conda/lib/python3.6/queue.py", line 173 in get
  File "/opt/conda/lib/python3.6/site-packages/tensorboard/summary/writer/event_file_writer.py", line 205 in run
  File "/opt/conda/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/opt/conda/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f88273e7740 (most recent call first):
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/comm.py", line 39 in broadcast_coalesced
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 21 in forward
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py", line 71 in _broadcast_coalesced_reshape
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py", line 88 in replicate
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 159 in replicate
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 154 in forward
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 577 in __call__
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 622 in _training_step
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 499 in train
  File "trainer.py", line 34 in &lt;module&gt;
Segmentation fault (core dumped)

```

## Expected behavior
Process through training(as normal)

## Environment info
&lt;!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! --&gt;
     
- `transformers` version: 3.0.2
- Platform: Linux-4.4.0-178-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0a0+9907a3e (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Planning to (don't see any flags!)</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>The stacktrace points to <code>Keras</code> importing <code>scipy</code> and a <code>KeyboardInterrupt</code>:</p>
<pre><code class="lang-python">File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/preprocessing/init.py”, line 22, in
from keras.preprocessing import image
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras/preprocessing/image.py”, line 22, in
from keras_preprocessing import image
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras_preprocessing/image/init.py”, line 5, in
from .affine_transformations import *
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/keras_preprocessing/image/affine_transformations.py”, line 15, in
from scipy import ndimage
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/scipy/ndimage/init.py”, line 153, in
from .interpolation import *
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/scipy/ndimage/interpolation.py”, line 37, in
from scipy import special
File “/homes/jl007/.conda/envs/torch/lib/python3.7/site-packages/scipy/special/init.py”, line 643, in
from . import _ufuncs
File “”, line 416, in parent
KeyboardInterrupt
Segmentation fault (core dumped)"
</code></pre>
<p>Are you triggering the <code>KeyboardInterrupt</code> and are you seeing the same issue when <code>Keras</code> is not imported?</p>

Gorgen: <p>solve it. I change environment. it works well. Many thanks</p>