Can't pickle local object 'DataLoader.__init__.<locals>.<lambda>'

arslansadiq: <p>Hi all,<br>
I hope everybody reading this is having a great day.</p>
<p>So I have a problem with <strong>torchvision.transforms.Lambda()</strong> function when used with python function: <strong>enumerate</strong>. I am using it to make my uni-channeled image into multi-channeled tensor. It works fine and produce data loader instance for torchvision datasets, but when I instantiate the batch’s index with the <strong>command enumerate(&lt;batch_name&gt;)</strong> it given the following error:</p>
<pre><code class="lang-auto">
  File "C:\Users\Arsalan\Desktop\project\mlmi\trainer\trainer.py", line 63, in _train_epoch
    for batch_idx, (data, target) in enumerate(self.data_loader):

  File "C:\Users\Arsalan\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 501, in __iter__
    return _DataLoaderIter(self)

  File "C:\Users\Arsalan\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 289, in __init__
    w.start()

  File "C:\Users\Arsalan\Anaconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)

  File "C:\Users\Arsalan\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)

  File "C:\Users\Arsalan\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)

  File "C:\Users\Arsalan\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)

  File "C:\Users\Arsalan\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)

AttributeError: Can't pickle local object 'NotMNISTDataLoader.__init__.&lt;locals&gt;.&lt;lambda&gt;'
</code></pre>
<p>On a side note:</p>
<p>This lambda transform works fine on my linux machine but on windows it is giving this problem.</p>
<p>Would anybody be kind enough to tell me what is problem is</p>

albanD: <p><a class="mention" href="/u/peterjc123">@peterjc123</a> lambda objects cannot be pickled on Windows?</p>

arslansadiq: <p>yes on windows it can’t be be at the moment.<br>
Do you know any solutions?</p>

albanD: <aside class="quote no-group quote-modified" data-username="arslansadiq" data-post="3" data-topic="31857">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/47e85d/48.png" class="avatar"> arslansadiq:</div>
<blockquote>
<p>yes on windows it can be at the moment.</p>
</blockquote>
</aside>
<p>So they can be pickled?<br>
The error then does not come from the fact that you try to pickle a lambda function?</p>

arslansadiq: <p>Sorry I miss-typed, my earlier comment is edited</p>

albanD: <p>I guess you will need to change <code>NotMNISTDataLoader</code> and use a proper function there instead of a lambda.</p>

arslansadiq: <p>so, for my problem:<br>
transforming uni-channeled image into multi-channeled tensor, what can I do?<br>
because this works fine on linux, but for windows users what’s the alternative.</p>
<p>for example if my batch size is <strong>tensor(128, 32, 32)</strong>, after transformation I want <strong>tensor(128, 3, 32, 32)</strong></p>

albanD: <p>Do you have the code for the <code>__init__</code> of  <code>NotMNISTDataLoader</code> ? There is a lamda function in there like <code>lambda x: x+1</code> that needs to be changed to</p>
<pre><code class="lang-auto">def tmp_func(x):
    return x + 1
</code></pre>
<p>And replace the lambda by <code>tmp_func</code>.</p>

arslansadiq: <p>Here you go:</p>
<pre><code class="lang-auto">class NotMNISTDataLoader(BaseDataLoader):
    """
    NotMnist data loading demo using BaseDataLoader
    """
    def __init__(self, data_dir, batch_size, shuffle, validation_split, num_workers, training=True):
        print("NotMnist is used")
        self.data_dir = data_dir + "NotMNIST/"
        
        trsfm = transforms.Compose([
            transforms.Resize((32, 32)),
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,)),
            transforms.Lambda(lambda x: x.repeat(3, 1, 1))
            ])
       
        self.dataset = NotMNIST(self.data_dir, train = training, download = True, transform = trsfm)
        super(NotMNISTDataLoader, self).__init__(self.dataset, batch_size, shuffle, validation_split, num_workers)
</code></pre>

albanD: <pre><code class="lang-auto">        trsfm = transforms.Compose([
            transforms.Resize((32, 32)),
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,)),
            transforms.Lambda(lambda x: x.repeat(3, 1, 1))
            ])
</code></pre>
<p>Should become</p>
<pre><code class="lang-auto">        def tmp_func(x):
            return x.repeat(3, 1, 1)

        trsfm = transforms.Compose([
            transforms.Resize((32, 32)),
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,)),
            transforms.Lambda(tmp_func)
            ])
</code></pre>

arslansadiq: <p>Thank you so much man <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
It’s working now.<br>
Have a great day.</p>
<p>By the way can you tell me what’s the problem was before, why it can’t be pickled on windows but does on linux?</p>

albanD: <p>I have no idea <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:">  That why I pinged peterjc which know windows much better.</p>

arslansadiq: <p>Ok thank you for your help.<br>
Let’s hope he replies</p>

peterjc123: <p>No, it is not supported on Windows. The reason is that multiprocessing lib doesn’t have it implemented on Windows. There are some alternatives like dill that can pickle more objects.</p>

peterjc123: <p>Could you please do me a favor and write this one into the official Windows doc?</p>

arslansadiq: <p>Excuse my ignorance on the matter, but would you like to tell me how exactly can one use dill for <strong></strong> .<br>
I tried installing that too but that didn’t solve the problem either. May be I am not using it right and I have to make some changes in my code in order to use it.<br>
Can you tell me about it’s usage to solve my problem?</p>

arslansadiq: <p>You mean here:</p>
<p><a href="https://answers.microsoft.com/en-us/windows/forum?sort=LastReplyDate&amp;dir=Desc&amp;tab=All&amp;status=all&amp;mod=&amp;modAge=&amp;advFil=&amp;postedAfter=&amp;postedBefore=&amp;threadType=All&amp;isFilterExpanded=false&amp;page=1" rel="nofollow noopener">Windows Discussion Forum</a><br>
Or if I am wrong, would you share the link?</p>

peterjc123: <p>No, things should be done at the backend. I don’t think we will introduce dill to resolve this problem. So what <a class="mention" href="/u/alband">@albanD</a> said is the correct solution, at least for now.  As for the document, it can be accessed <a href="https://pytorch.org/docs/master/notes/windows.html" rel="nofollow noopener">here</a> and modified <a href="https://github.com/pytorch/pytorch/blob/master/docs/source/notes/windows.rst" rel="nofollow noopener">here</a>.</p>

sarthakpati: <p>Hi,</p>
<p>Has there been any update on this in the official repo? I just ran into this issue on the latest Torch from Conda.</p>
<p>Cheers!</p>

asura: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, I am facing the same issue while using Pool method. As you suggested, I removed lamda functions. Still the error is persisting. (I am using Windows, I used Colab to try the same code, but it is also showing the same error there … ) Can you suggest, what can be done?</p>