Federated - IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1) - Python

username: Ilias_S
cooked: <p>Hello, I have a problem when training a federated network. I get the error when passing a noise vector to one of my sub-networks. This error occurs only when I am in the federated training step. When training in host with the same code the training process goes on. The code that gives me the error is below:</p>
<pre><code class="lang-auto"> ## Train with all-fake batch
    # Generate batch of latent vectors
    noise = torch.zeros(10,128)
    # Generate fake image batch with G
    fake = model[0](noise)
    label = torch.zeros(len(noise))
    # Classify all fake batch with D
    output = model[1](fake.detach()).view(-1)
    # Calculate D's loss on the all-fake batch
</code></pre>
<p>the error is :</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
PureFrameworkTensorFoundError             Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/syft/frameworks/torch/tensors/interpreters/native.py in handle_func_command(cls, command)
    327             # Note that we return also args_type which helps handling case 3 in the docstring
--&gt; 328             new_args, new_kwargs, new_type, args_type = hook_args.unwrap_args_from_function(
    329                 cmd, args_, kwargs_, return_args_type=True

~/.local/lib/python3.8/site-packages/syft/generic/frameworks/hook/hook_args.py in unwrap_args_from_function(attr, args_, kwargs_, return_args_type)
    157         # Try running it
--&gt; 158         new_args = hook_args(args_)
    159 

~/.local/lib/python3.8/site-packages/syft/generic/frameworks/hook/hook_args.py in &lt;lambda&gt;(x)
    356 
--&gt; 357     return lambda x: f(lambdas, x)
    358 

~/.local/lib/python3.8/site-packages/syft/generic/frameworks/hook/hook_args.py in three_fold(lambdas, args_, **kwargs)
    535         lambdas[0](args_[0], **kwargs),
--&gt; 536         lambdas[1](args_[1], **kwargs),
    537         lambdas[2](args_[2], **kwargs),

~/.local/lib/python3.8/site-packages/syft/generic/frameworks/hook/hook_args.py in &lt;lambda&gt;(i)
    331         # Last if not, rule is probably == 1 so use type to return the right transformation.
--&gt; 332         else lambda i: forward_func[type(i)](i)
    333         for a, r in zip(args_, rules)  # And do this for all the args / rules provided

~/.local/lib/python3.8/site-packages/syft/frameworks/torch/hook/hook_args.py in &lt;lambda&gt;(i)
     29     if hasattr(i, "child")
---&gt; 30     else (_ for _ in ()).throw(PureFrameworkTensorFoundError),
     31     torch.nn.Parameter: lambda i: i.child

~/.local/lib/python3.8/site-packages/syft/frameworks/torch/hook/hook_args.py in &lt;genexpr&gt;(.0)
     29     if hasattr(i, "child")
---&gt; 30     else (_ for _ in ()).throw(PureFrameworkTensorFoundError),
     31     torch.nn.Parameter: lambda i: i.child

PureFrameworkTensorFoundError: 

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
&lt;ipython-input-24-eb14fd9fab4b&gt; in &lt;module&gt;
      2     start_time = time.time()
      3     print(f"Epoch Number {epoch + 1}")
----&gt; 4     federated_model = train()
      5     model = federated_model
      6 #     test(federated_model)

&lt;ipython-input-23-7027cb8c9ab0&gt; in train()
     57         for remote_index in range(len(compute_nodes)):
     58             data, target = remote_dataset[remote_index][data_index]
---&gt; 59             models[remote_index] = update(data, target, models[remote_index], optimizers[remote_index])
     60         for model in models:
     61             model.get()

&lt;ipython-input-23-7027cb8c9ab0&gt; in update(data, target, model, optimizer)
     24     label = torch.zeros(len(noise))
     25     # Classify all fake batch with D
---&gt; 26     output = model[1](fake.detach()).view(-1)
     27     # Calculate D's loss on the all-fake batch
     28 

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/Desktop/DE/Pytorch_ADAGAN.py in forward(self, input)
    167 
    168         def forward(self, input):
--&gt; 169             return self.main(input)
    170 
    171     def parameters(self):

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/.local/lib/python3.8/site-packages/torch/nn/modules/container.py in forward(self, input)
     98     def forward(self, input):
     99         for module in self:
--&gt; 100             input = module(input)
    101         return input
    102 

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
     85 
     86     def forward(self, input):
---&gt; 87         return F.linear(input, self.weight, self.bias)
     88 
     89     def extra_repr(self):

~/.local/lib/python3.8/site-packages/syft/generic/frameworks/hook/hook.py in overloaded_func(*args, **kwargs)
    591                 handle_func_command = syft.framework.Tensor.handle_func_command
    592 
--&gt; 593             response = handle_func_command(command)
    594 
    595             return response

~/.local/lib/python3.8/site-packages/syft/frameworks/torch/tensors/interpreters/native.py in handle_func_command(cls, command)
    361             # in the execute_command function
    362             try:
--&gt; 363                 response = cls._get_response(cmd, args_, kwargs_)
    364             except AttributeError:
    365                 # Change the library path to avoid errors on layers like AvgPooling

~/.local/lib/python3.8/site-packages/syft/frameworks/torch/tensors/interpreters/native.py in _get_response(cmd, args_, kwargs_)
    395 
    396         if isinstance(args_, tuple):
--&gt; 397             response = command_method(*args_, **kwargs_)
    398         else:
    399             response = command_method(args_, **kwargs_)

~/.local/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
-&gt; 1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
   1372         output = input.matmul(weight.t())

~/.local/lib/python3.8/site-packages/syft/generic/frameworks/hook/hook.py in overloaded_func(*args, **kwargs)
    591                 handle_func_command = syft.framework.Tensor.handle_func_command
    592 
--&gt; 593             response = handle_func_command(command)
    594 
    595             return response

~/.local/lib/python3.8/site-packages/syft/frameworks/torch/tensors/interpreters/native.py in handle_func_command(cls, command)
    361             # in the execute_command function
    362             try:
--&gt; 363                 response = cls._get_response(cmd, args_, kwargs_)
    364             except AttributeError:
    365                 # Change the library path to avoid errors on layers like AvgPooling

~/.local/lib/python3.8/site-packages/syft/frameworks/torch/tensors/interpreters/native.py in _get_response(cmd, args_, kwargs_)
    395 
    396         if isinstance(args_, tuple):
--&gt; 397             response = command_method(*args_, **kwargs_)
    398         else:
    399             response = command_method(args_, **kwargs_)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
</code></pre>
<p>Thank you in advance!</p>

username: Chirath_Pansilu
cooked: <p>Hey Can you give the code snippet for your loss calculation(or the whole training process) . It seems like there is a problem.</p>
<p>It is hard to tell just by looking at the given code.<br>
Thank You.</p>

username: Ilias_S
cooked: <p>yes of course , the full training code is:</p>
<pre><code class="lang-auto">def update(data, target, model, optimizer):
    model[0].send(data.location)
    model[1].send(data.location)
    
    model[1].zero_grad()
    
    real_cpu = data.float()
    b_size = real_cpu.size(0)

    label = torch.full((b_size,), 1)
    # Forward pass real batch through D
    print(model[0])
    output = model[1](data.float()).view(-1).float()
    # Calculate loss on all-real batch
    errD_real = F.mse_loss(output, target.float())
    # Calculate gradients for D in backward pass
    errD_real.backward()
    
    ## Train with all-fake batch
    # Generate batch of latent vectors
    noise = torch.randn(128,10)
    noise.send(data.location)
#     print(noise)
    # Generate fake image batch with G
    fake = model[0](noise)
    label = torch.zeros(len(noise))
    # Classify all fake batch with D
    output = model[1](fake.detach()).view(-1)
    # Calculate D's loss on the all-fake batch

    errD_fake = F.mse_loss(output, label)
    # Calculate the gradients for this batch
    errD_fake.backward()
    # Add the gradients from the all-real and all-fake batches
    errD = errD_real + errD_fake
    # Update D
    optimizer[1].step()
    
    return model
</code></pre>

username: alex_gilabert
cooked: <p>try</p>
<pre><code class="lang-auto">fake.detach().view(1,-1)
</code></pre>
<p>Torch tensors can be modified into many ways, you just need to play with that.</p>
<p>Let me know if that works <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Ilias_S
cooked: <p>I tried it and it still shows the error message. The thing is that the error occurs before <em>fake</em> is defined. Specifically, it occurs when trying to predict the noise vector ,  <code>fake = model[0](noise)</code>. When running normal training this does not occur.</p>

username: Chirath_Pansilu
cooked: <aside class="quote no-group" data-username="Ilias_S" data-post="5" data-topic="86439">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ilias_s/48/25902_2.png" class="avatar"> Ilias_S:</div>
<blockquote>
<p>Specifically, it occurs when trying to predict the noise vector , <code>fake = model[0](noise)</code> . When running normal training this does not occur.</p>
</blockquote>
</aside>
<p>If this is the case I think there might be something wrong with generator model.<br>
Can you provide that code.<br>
Thank you</p>

username: Ilias_S
cooked: <p>Unfortunately, I cannot share the Generator model’s code at this time. I do understand the problem that poses, but I am confident that the generator’s structure is not at fault. That is because the network trains normally without error when I don’t federate it, under the same circumstances.</p>