MemoryError on Raspberry Pi 2

username: ayyuce
cooked: <p>Hello,</p>
<p>I am trying to train <a href="https://github.com/jarfo/gcommands" rel="noopener nofollow ugc">this</a> on a Raspberry 2 which has 512MBs of RAM. I reduced the original Google Commands Dataset into 12 classes and 34000 training samples and I am using a Resnet model with 8 layers.</p>
<pre><code class="lang-auto">class ResNet(BaseModel):
    def __init__(self, config):
        super().__init__()
        self.n_layers = config["n_layers"]
        n_maps = config["n_feature_maps"]

        self.layers = nn.ModuleDict()

        self.layers["conv_0"] = nn.Conv2d(1, n_maps, (3, 3), padding=1, bias=False)

        for i in range(1, self.n_layers + 1):
            if config["use_dilation"]:
                padding_size = int(2**((i-1) // 3))
                dilation_size = int(2**((i-1) // 3))
                self.layers[f"conv_{i}"] = nn.Conv2d(n_maps, n_maps, (3, 3), padding=padding_size, dilation=dilation_size, bias=False)
            else:
                self.layers[f"conv_{i}"] = nn.Conv2d(n_maps, n_maps, (3, 3), padding=1, bias=False)
            self.layers[f"bn_{i}"] = nn.BatchNorm2d(n_maps, affine=False)

        if "pool" in config:
            self.layers["pool"] = nn.AvgPool2d(config["pool"])

        self.layers["output"] = nn.Linear(n_maps, config["n_labels"])

        self.activations = nn.ModuleDict({
            "relu": nn.ReLU()
        })

    def forward(self, x):
        x = x.unsqueeze(1)
        x = self.layers["conv_0"](x)
        x = self.activations["relu"](x)

        if "pool" in self.layers:
            x = self.layers["pool"](x)

        prev_x = x
        for i in range(1, self.n_layers + 1):
            x = self.layers[f"conv_{i}"](x)
            x = self.activations["relu"](x)

            if i % 2 == 0:
                x = x + prev_x
                prev_x = x

            x = self.layers[f"bn_{i}"](x)

        x = x.view(x.size(0), x.size(1), -1) # shape: (batch, features, o3)
        x = x.mean(2)
        x = self.layers["output"](x)
        return x
</code></pre>
<p>It can be trained successfully on my laptop, however, I get MemoryError on the rpi2. I split the wav list thus they all won’t be loaded to the memory at once. (There are almost 1,5 GBs of wav files, rpi2 has only 512 MBs of RAM) I set --num_workers argument to 0. (<a href="https://github.com/pytorch/pytorch/issues/13246" rel="noopener nofollow ugc">CPU memory gradually leaks when num_workers &gt; 0 in the DataLoader</a>) However I can’t solve the problem.</p>
<p><em>Environment:</em><br>
<em>PyTorch Version: 1.3.0</em><br>
<em>Python version: 3.7.3</em><br>
<em>OS: Raspbian GNU/Linux 10 (buster)</em><br>
<em>CPU: armv7l</em></p>
<p>What could be the problem?</p>
<p>Thank you so much!</p>

username: ptrblck
cooked: <p>Could you reduce the batch size to a single sample and rerun the code, in case your current batch size is larger?<br>
If that doesn’t help, could you remove the data loading as a debugging step and feed a single sample as <code>x = torch.randn(...)</code> to the model to check the memory usage?</p>

username: ayyuce
cooked: <p>Thank you so much for the reply!</p>
<p>I’ve tried with batch_size=1 and got the same error. With a random tensor, the memory usage is 0.004444MB.</p>

username: ptrblck
cooked: <p>This would most likely point to the data loading pipeline using a lot of memory.<br>
If I understand <a href="https://github.com/jarfo/gcommands/blob/8d7e3b467c2502cfc7c32405112a7f160d271c61/gcommands_loader.py">this loader</a> correctly, the <code>wav</code> files are all preloaded and just indexed in the <code>__getiteim__</code> method?<br>
If so, try to use lazy loading and try to move the actual data loading into the <code>__getitem__</code> method while the <code>__init__</code> method gets the paths etc.</p>