Change:
PyTorch version: 0.3.0 Python version: 3.5 GPU models and configuration: 8x NVIDIA Titan I found that requiresgrad setting in module is ignored when module is wrapped with nn.DataParallel. For example, This code should be executed because very small part of network activation (only fc layer) is stored for backward computation, but it results to run-time error: Internally, I found a crack in function which is in . In replicate function, it copies all parameter in module (# of replica times) with . In broadcasting code, it just defines new with default constructor parameter, which is always set to True. I think there can be some choices to fix this issue. is intended behavior for DataParallel, so we should use volatile (or ) to implement transfer learning in DataParallel model should be fixed to handle requires_grad when copying parameters. should be synchronizing in I think any of solution doesn't need lots of effort to fix it.

Discussion:
super(TestCase, self).assertEqual(len(x), len(y), message) for x_, y_ in zip(x, y): self.assertEqual(x_, y_, prec, message) <ins> elif isinstance(x, bool) and isinstance(y, bool): super(TestCase, self).assertEqual(x, y, message) </ins> elif isinstance(x, Number) and isinstance(y, Number): if abs(x) == float('inf') or abs(y) == float('inf'): if allow_inf:

torch.randn(4, 4).cuda()) @unittest.skipIf(not TEST_MULTIGPU, "multi-GPU not supported") <ins> def test_broadcast_not_requiring_grad(self): variables = [ Variable(torch.randn(1, 2).cuda(), requires_grad=True), Variable(torch.randn(1, 2).cuda(), requires_grad=False), Variable(torch.randn(1, 2).cuda(), requires_grad=False), Variable(torch.randn(1, 2).cuda(), requires_grad=True), Variable(torch.randn(1, 2).cuda(), requires_grad=True), ] broadcasted_variables = Broadcast.apply((0, 1), *variables) for output_idx, broadcasted_var in enumerate(broadcasted_variables): input_var = variables[output_idx % len(variables)] self.assertEqual(input_var.requires_grad, broadcasted_var.requires_grad) @unittest.skipIf(not TEST_MULTIGPU, "multi-GPU not supported") </ins> def test_replicate(self): module = nn.Linear(10, 5).float().cuda() input = Variable(torch.randn(2, 10).float().cuda())

ctx.num_inputs = len(inputs) ctx.input_device = inputs[0].get_device() outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus) <ins> non_differentiables = [] for idx, input_requires_grad in enumerate(ctx.needs_input_grad[1:]): if not input_requires_grad: for output in outputs: non_differentiables.append(output[idx]) ctx.mark_non_differentiable(*non_differentiables) </ins> return tuple([t for tensors in outputs for t in tensors]) @staticmethod

other types will be a shallow copy and can be corrupted if written to in the model's forward pass. <ins> .. warning:: Forward and backwrad hooks defined on :attr:`module` and its submodules won't be invoked anymore, unless the hooks are initialized in the :meth:`forward` method. </ins> Args: module: module to be parallelized device_ids: CUDA devices (default: all devices)

(e.g. BatchNorm stats) are broadcast form the module in process of rank 0, to all other replicas in the system in every iteration. <ins> .. warning:: Forward and backwrad hooks defined on :attr:`module` and its submodules won't be invoked anymore, unless the hooks are initialized in the :meth:`forward` method. </ins> Args: module: module to be parallelized device_ids: CUDA devices (default: all devices)