How to make parameters of pytorch module functions searchable?

marsggbo: <p>When we define a Pytorch module, we have to specify the values for all parameters.</p>
<p>But now, I want to make the parameters searchable. For example, maybe we can define a new class <code>ValueChoice</code>, then we can define a convolution as follows:</p>
<pre><code class="lang-python">out_channels = ValueChoice([16,24])
kernel_size = ValueChoice([3,5,7])
convBN = nn.Sequential(
    nn.Conv2d(3, out_channels , kernel_size),
    nn.BatchNorm2d(out_channels)
)
</code></pre>
<p>my question is can we implement a class like <code>ValueChoice</code>? If so, how should we do?</p>

Valerio_Biscione: <p>I am not sure what you mean by sercheable, could you explain it again or give more examples?</p>

marsggbo: <p>Thanks for your reply.</p>
<p>I haven’t thought too much about the implementation details of <code>ValueChoice</code>.  My idea is that <code>ValueChoice</code> receives a list of candidate values, and it can return a value selected from the candidates. Therefore, we can use it to search an optimal architecture.</p>
<p><em>Searchable</em> means that <code>ValueChoice([3,5,7])</code>  can return a value of 3, 5, or 7 according to the predefined search strategy. For example, <code>nn.conv2d(3, 16, ValueChoice([3,5,7]))</code> can be decoded as <code>nn.conv2d(3, 16, 3)</code>, <code>nn.conv2d(3, 16, 5)</code>, or <code>nn.conv2d(3, 16, 7)</code>.</p>
<p>I have two questions about <code>ValueChoice</code> implementation, and I wish you can give me some suggestions:</p>
<ol>
<li>how to decode <code>ValueChoice</code> into a value</li>
<li>during the search process, <code>ValueChoice</code>  returns a different value at different epoch. My second question is how to <strong>change or inherit</strong> weights of modules. For example, at the 1-st epoch,  <code>ValueChoice</code> returns 3, then we obtain <code>nn.conv2d(3, 16, 3)</code>; but if <code>ValueChoice</code> returns 5 at the 2-nd epoch, then we obtain <code>nn.conv2d(3, 16, 5)</code>. Can these two convolutional operations share weights?</li>
</ol>

Valerio_Biscione: <ol>
<li>There are many ways of doing this</li>
</ol>
<pre><code class="lang-auto">import numpy as np
import torch.nn as nn 
class ValueChoice():
    def __init__(self, values):
        self.values = values
        np.random.shuffle(self.values)

    def __iter__(self):
        for i in self.values:
            yield i

a = ValueChoice([1, 2, 3])
i = iter(a)
print(nn.Conv2d(next(i), 2, 3)) # Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
print(nn.Conv2d(next(i), 2, 3)) # Conv2d(3, 2, kernel_size=(3, 3), stride=(1, 1))
print(nn.Conv2d(next(i), 2, 3)) # Conv2d(1, 2, kernel_size=(3, 3), stride=(1, 1))
print(nn.Conv2d(next(i), 2, 3)) # this will give StopIteration error: not enough values.
</code></pre>
<p>However, this will do what you ask, but not what you want. If you have something like<br>
kernel = ValueChoice([3, 5, 7])<br>
out_ch = ValueChoice([16, 32, 64])</p>
<p>You may want to search all possible permutations of these values, and my solution won’t do that.</p>
<p>Instead of coding something yourself I would suggest you use a library for hyperparameter tuning, something like that <a href="https://docs.ray.io/en/master/tune/index.html#" rel="noopener nofollow ugc">Tune: Scalable Hyperparameter Tuning — Ray v2.0.0.dev0</a><br>
Reinventing the wheel can be very fun and instructive, but also error prone and inefficient in the long run <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<ol start="2">
<li>You can find some ways of doing that, but not easily and probably it wouldn’t make much sense. The kernels have different size, so it isn’t clear how one would translate in the other.<br>
If however you have a different number of filters, let’s say <code>nn.Conv2d(3, 16, 3)</code> and <code>nn.Conv2d(3, 5, 3)</code> and want to copy the first into the second, you could copy 5 random sampled filter from the <code>nn.Conv2d(3, 16, 3)</code> into <code>nn.Conv2d(3, 5, 3)</code>. However, again, it seems quite a weird policy and I don’t personally think it’s a fruitful decision. But you are free to try.<br>
Also, doing this copy is not going to be easy. Standard <code>torch</code> uses <code>state_dict</code> to check that all values matches, and in your case it will throw an error because, well, they don’t! So you need to write your own copy function. You will need to access the parameters through <code>module.named_parameters()</code> and then manually copying them.</li>
</ol>