Case for calling forward() manually

ptrch_c_m: <p>I have implemented the <code>forward()</code> which does the forward pass of the model.</p>
<p>I have to define also another function, <code>some_function()</code>, in the model, that does some other processing first, and then does a forward pass of the model. I have seen that forward is almost never called manually, because <code>model.__call__()</code>has some extra code, although that is not when you call <code>forward()</code> from within the model, but externally. Should I call <code>self.forward()</code> manually from within <code>some_function()</code> in this case? What is the recommended way of doing this?</p>

ptrblck: <aside class="quote no-group" data-username="ptrch_c_m" data-post="1" data-topic="222547">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e274bd/48.png" class="avatar"> ptrch_c_m:</div>
<blockquote>
<p>Should I call <code>self.forward()</code> manually from within <code>some_function()</code> in this case? What is the recommended way of doing this?</p>
</blockquote>
</aside>
<p>Calling <code>forward</code> explicitly will skip the hook registration and might cause unexpected behavior. I would thus recommend sticking to the <code>__call__</code> op making sure all hooks are properly fired (if any are used).</p>

valerian.rey: <p>EDIT: saw the other answer after I posted, still keeping this for extra details.</p>
<p>For a <code>nn.Module</code>, <code>__call__</code> will basically call all registered hooks + the <code>forward</code> method, while <code>forward</code> just does the basic job of your layer or model without calling any hook.</p>
<p>If you want your hooks to run (which you probably should), <code>some_function</code> should call <code>__call__</code> rather than <code>forward</code>.</p>
<p>It’s still a bit weird in my opinion that you add an extra function to your model for pre-processing + calling the model. For instance, imagine you have the following module:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):

    def some_function(self, input):
        input = input * 2.
        self(input)  # equivalent to self.__call__(input)

    def forward(self, input):
        # do something
        pass
</code></pre>
<p>And you <a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_forward_pre_hook" rel="noopener nofollow ugc">register a pre-hook</a> to it. If you call <code>some_function</code>, your pre-hook will be called in-between the preprocessing (<code>input = input * 2.</code>) and the <code>forward</code> call. It would be more usual to have a pre-hook that is applied before everything (even before pre-processing).</p>
<p>So in my opinion you should ideally not have an extra function in your module to do some optional preprocessing + call <code>__call__</code>. You could either make the <code>forward</code> method call this pre-processing code itself:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self, do_preprocessing: bool):
        super().__init__(self)
        self.do_preprocessing = do_preprocessing

    def forward(self, input):
        if self.do_preprocessing:
            input = input * 2.

        # do something
        pass
</code></pre>
<p>Or you could have another module responsible for doing the preprocessing + calling the module that does the actual job:</p>
<pre><code class="lang-auto">class PreprocessingWrapper(nn.Module):
    def __init__(self, module: nn.Module):
        super().__init__(self)
        self.module = module

    def forward(self, input):
        input = input * 2.
        return self.module(input)
</code></pre>
<p>By the way I think all of this only makes a difference if you end up using module hooks. If you don’t use any and don’t wanna be bothered, I think all solutions will be equivalent.</p>

ptrch_c_m: <p><a class="mention" href="/u/valerian.rey">@valerian.rey</a> thank you a lot for your detailed answer. You are right to be confused about the extra function, so let me explain to make it more clear.</p>
<p>My raw data is text, and I have to make a bit time-consuming preprocessing so that I bring it to the form that the model uses. During training, which also involves evaluation, I have pre-processed the and stored the data, and call the model (<code>forward()</code> function) with this form of the data. The other function that I have implemented (<code>some_function()</code>), takes raw data, pre-processes it, and then uses the model’s <code>forward()</code>.</p>
<p>To sum up, I want the functionality of model’s <code>forward()</code>function, that takes the processed form of data, from another function within the same model, that receives as input raw text.</p>