GPU Inference Multithread problem

username: Emmanuel_Moran
cooked: <p>Hello,</p>
<p>I’ve been working with a Yolov3 Pytorch Implementation. Quite impresive the Inference time in GPU. But, i need to process simultaneously (multithreads) videos.</p>
<p>I design a simply main file which select some videos and send them to be processed by a function</p>
<pre><code class="lang-auto">VIDEOS_LIST = sortVideoList()
for index,value in enumerate(VIDEOS_LIST):
    if len(value) &gt; 0:
        for video in value:
        processThread = threading.Thread(target=generate_dets, args=(value,index))
        processThread.start()
</code></pre>
<p>Lets say that VIDEOS_LIST is a list of list of 3 internal lists:</p>
<pre><code class="lang-auto">VIDEOS_LIST=
[
[v1,v2,v3],
[v4,v5,v6],
[v7,v8,v9]
]
</code></pre>
<p>So each thread will “generate_dets” with the videos selected in its proper thread.</p>
<p>In different times, in diferent frames, in all the threads, the output of the neuronal network Yolov3 happend to have some garbage inside.</p>
<p>I though it was something with the frames, but the same videos processed with the same network but only in the main thread (in the GPU also) works fine.</p>
<p>this is not configuration, because i have other network in other DP framework and it does totally fine.</p>
<p>Any help?</p>
<p>this is the make_detections function:</p>
<pre><code class="lang-auto">def letterbox_image(img, inp_dim):
    '''resize image with unchanged aspect ratio using padding'''
    img_w, img_h = img.shape[1], img.shape[0]
    w, h = inp_dim
    new_w = int(img_w * min(w/img_w, h/img_h))
    new_h = int(img_h * min(w/img_w, h/img_h))
    resized_image = cv2.resize(img, (new_w, new_h),
                               interpolation=cv2.INTER_CUBIC)
    canvas = np.full((inp_dim[1], inp_dim[0], 3), 128)
    canvas[(h-new_h)//2:(h-new_h)//2 + new_h,
           (w-new_w)//2:(w-new_w)//2 + new_w, :] = resized_image
    return canvas


def prep_image(img, inp_dim):
    """
    Prepare image for inputting to the neural network.
    Returns a Variable
    """

    orig_im = img.copy()
    dim = orig_im.shape[1], orig_im.shape[0]
    img = (letterbox_image(orig_im, (inp_dim, inp_dim)))
    img_ = img[:, :, ::-1].transpose((2, 0, 1)).copy()
    img_ = torch.from_numpy(img_).float().div(255.0).unsqueeze(0)
    return img_, orig_im, dim


def Net_MakeDetections(model,frame):
    CUDA = torch.cuda.is_available()
    inp_dim = int(model.net_info["height"])
    assert inp_dim % 32 == 0
    assert inp_dim &gt; 32
    nms_thesh               =       float(0.3)
    classes                 =       open('data/coco.names', "r").read().split("\n")[:-1]
    confidence              =       float(0.3)
    num_classes             =       len(classes)


    img, orig_im, dim = prep_image(frame, inp_dim)
    im_dim = torch.FloatTensor(dim).repeat(1, 2)
    if CUDA:
        im_dim = im_dim.cuda()
        img = img.cuda()
    with torch.no_grad():
        output = model(Variable(img), CUDA)
    output = write_results(output, confidence, num_classes,nms=True, nms_conf=nms_thesh)
    im_dim = im_dim.repeat(output.size(0), 1)
    scaling_factor = torch.min(inp_dim/im_dim, 1)[0].view(-1, 1)
    output[:, [1, 3]] -= (inp_dim -scaling_factor*im_dim[:, 0].view(-1, 1))/2
    output[:, [2, 4]] -= (inp_dim - scaling_factor*im_dim[:, 1].view(-1, 1))/2
    output[:, 1: 5] /= scaling_factor

    for i in range(output.shape[0]):
        output[i, [1, 3]] = torch.clamp(output[i, [1, 3]], 0.0,im_dim[i, 0])
        output[i, [2, 4]] = torch.clamp(output[i, [2, 4]], 0.0,im_dim[i, 1])
    try:
        a = [out.cpu().numpy()[[1, 2, 3, 4, 5, 7]] for out in output if classes[int(out[-1])] in ['person']]
        #if len(a)==0:print(list_number,'---&gt; yolo does not have detections')
    except:
        print(' ----&gt; Exception at yolo_detections()',[int(out[-1]) for out in output])
        a=[]
    return a
</code></pre>
<p>NOTE: Write result just make some nms. Make_Detections() recieve the model (yolov3 pytorch) and the frame, quite simple.</p>
<p>Any help?</p>
<p>Kind regards,</p>