Effect of python eval() function on GPU training

shartzog: <p>How does the use of the python eval() function impact model performance on a GPU?  I ran a profile of my <a href="https://discuss.pytorch.org/t/wasting-my-time/92512">code</a> using torch.utils.bottleneck and found that multiple instances of the _convolutions, conv2d and other torch functions appeared in my GPU autograd trace with some utilizing ~33% GPU time and others utilizing next to none.  My current Net (nn.Module) class performs eval() for the functions that appear in the nn.Module <strong>init</strong> function during initialization, but the forward() function utilizes eval() for every call.  I’m only getting about 10 to 15% GPU utilization (GTX 2060) during training, so I’m curious about the effect that eval() might be having.  Refactoring the code to avoid that eval() function will be very difficult, so I’d rather not expend the effort if eval() has little or no effect.  Here’s the code for my Net (aka nn.Module) class…</p>
<pre><code class="lang-auto">class Net(nn.Module):
    """
    Build pytorch module using eval() on incoming model tensor and lists of eval strings
    for layers and params.
    """
    def __init__(self, modelinputtensor, layerlist, layerparams, **kwargs):
        """
        args:
        modelinputtensor:   example model input tensor (including an arbitrary batch dimension)
        layerlist:  list of pytorch nn fucntions as their 'F' namespace equivalents
                    Example: 'nn.MaxPool2d' should be supplied as 'F.max_pool2d'
        layerparams:    list of _independent_ params in their nn form and passed as a tuple.
        kwargs:
        activations:    list of activation functions for forward layers.  the length
                        of the list must match the length of layerlist exactly even
                        though the activation function supplied for any pooling
                        layers will be ignored and the final value supplied will
                        always be replaced by Sigmoid.
        Example:
            The first conv2d layer will have 3 params in a tuple
                of form (in_channels, out_channels, kernel_size).
            Subsequent conv2d layers will have _2_ params in a tuple
                of form (out_channels, kernel_size) since the in_channels
                are determined by the previous layer.
            Pooling layers will always have params of the form (x, y)
                corresponding to the pooling window size.
            Linear layers will always have a single param corresponding to
                the number of out features for the layer since input
                features are determined by the preceding layer)
        """
        super(Net, self).__init__()
        self.activations = kwargs.get('activations', ['F.relu' for layer in layerlist])
        self.lyrs, self.fwdlyrs = self.get_layers(modelinputtensor, layerlist, layerparams, self.activations, DEBUG)

    def forward(self, x):
        """
        """
        for f in self.fwdlyrs:
            x = eval(f)
        return torch.sigmoid(x)

    def get_layers(self, testtensor, funcs, params, activations, debug):
        """
        Build network layers from supplied test tensor, funcs, and param eval strings.
        """
        initlayers = nn.ModuleList()
        fwdlayers = list()
        if debug == 1:
            print(testtensor.size())
        lastsize = None
        lyr = 0
        with torch.no_grad():
            for fn, pa in zip(funcs, params):
                if lastsize is not None:
                    if fn.__name__ == 'conv2d':
                        pa = (lastsize[1], pa[0], pa[1])
                    elif fn.__name__ == 'linear':
                        if not testtensor.ndim == 2:
                            testtensor = testtensor.view(-1, self.num_flat_features(testtensor))
                            fwdlayers.append("x.view(-1,self.num_flat_features(x))")
                            lastsize = testtensor.size()
                        pa = (lastsize[1], pa)
                if fn.__name__ == 'conv2d':
                    paeval = ",".join(tuple(map(str, (pa[1], pa[0], pa[2], pa[2]))))
                    paeval = "torch.tensor(np.random.rand(" + paeval + "), dtype=torch.float32)"
                elif fn.__name__ == 'max_pool2d':
                    paeval = ",".join(tuple(map(str, pa)))
                elif fn.__name__ == 'linear':
                    paeval = ",".join(tuple(map(str, (pa[1], pa[0]))))
                    paeval = "torch.tensor(np.random.rand(" + paeval + "),dtype=torch.float32)"
                if not fn.__name__ == 'linear' or pa[0] &gt; pa[1]:
                    testtensor = fn(testtensor, eval(paeval))
                    lastsize = testtensor.size()
                    initlayers.append(eval(self.__get_init_equivalent(fn.__name__, pa)))
                    fwdlayers.append(self.__get_fwd_equivalent(fn.__name__, lyr))
                    lyr += 1
                    if debug == 1:
                        print(testtensor.size())
                elif debug == 1:
                    print('NetDictionary: Eliminating linear layer - out features &gt; previous layer')
        fwdlayers[-1] = 'self.lyrs[' + str(lyr - 1) + '](x)'
        return initlayers, fwdlayers

    def num_flat_features(self, x):
        """
        Calculate number of flat features in a given net layer.
        Useful for transitioning between conv and linear layers.
        """
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

    def __get_init_equivalent(self, funcname, initparams):
        """
        Construct eval string from supplied funtions and parameters for the
        style required in the torch.nn.Module __init__.
        """
        return 'nn.' + ''.join([val.capitalize()
                                for val in funcname.split('_')
                               ]) + '(' + ",".join(tuple(map(str, initparams))) + ')'

    def __get_fwd_equivalent(self, funcname, lyrnum):
        """
        Construct eval string from supplied funtions and parameters for the
        style required in the torch.nn.Module __init__.
        """
        if not funcname == 'max_pool2d':
            return self.activations[lyrnum] + '(self.lyrs[' + str(lyrnum) + '](x))'
        else:
            return 'self.lyrs[' + str(lyrnum) + '](x)'
</code></pre>

shartzog: <p>I’m currently working my way through this post:</p><aside class="quote quote-modified" data-post="19" data-topic="548">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/b9e5f3/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">How to prefetch data when processing with GPU?</a> 
  </div>
  <blockquote>
    I’ve spent some time working on this problem over a variety of projects. I’ve cut and pasted some past thoughts, bullets, etc from previous discussions. My background involves architecting systems that move large volumes of data from network cards to storage and then back again on request, with processing between the steps. A very similar set of concerns. 
The two main constraints that usually dominate your PyTorch training performance and ability to saturate the shiny GPUs are your total CPU IP…
  </blockquote>
</aside>
<p>
Let me know if any of you have thoughts specifically related to eval(), but for now, it looks like I have plenty of best practices to consider before tackling my eval() issue… <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"><br>
–SEH</p>