Issue 1:
🐛 Bug

The CyclicLR scheduler tries to access the optimizer's momentum parameter even if cycle_momentum=False.

To Reproduce

Steps to reproduce the behavior:

Setup a model named net

Run the following:

import torch.optim as optim
optimizer = optim.Adam(net.parameters(), lr=0.001)
scheduler = optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.0001, max_lr=0.01, cycle_momentum=False)

Resulting error:

KeyError Traceback (most recent call last)
<ipython-input-113-a9df59301cc9> in <module>
1 optimizer = optim.Adam(net.parameters(), lr=0.001)
----> 2 scheduler = optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.0001, max_lr=0.01, cycle_momentum=False)

~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in __init__(self, optimizer, base_lr, max_lr, step_size_up, step_size_down, mode, gamma, scale_fn, scale_mode, cycle_momentum, base_momentum, max_momentum, last_epoch)
591 for momentum, group in zip(base_momentums, optimizer.param_groups):
592 group['momentum'] = momentum
--> 593 self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))
594 self.max_momentums = self._format_param('max_momentum', optimizer, max_momentum)
595

~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in <lambda>(group)
591 for momentum, group in zip(base_momentums, optimizer.param_groups):
592 group['momentum'] = momentum
--> 593 self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))
594 self.max_momentums = self._format_param('max_momentum', optimizer, max_momentum)
595

KeyError: 'momentum'

Expected behavior

I expect the scheduler not try to access the momentum key if I've told it not to cycle the momentum parameter. The error seems to be due to a minor indentation mistake here:

https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L594-L595

The instance variables base_momentums and max_momentums are only ever accessed if cycle_momentum is true. Thus the two lines where they're being should be indented.

Environment

N/A

Additional comments

I'll push a PR to fix this in a minute, though may need a little help with PyTorch's specific contribution process.

Issue 2:
Issue description

If I use an optimizer like adam with no momentum and follow the message of line 578 below here, passing cycle_momentum=False, line 584 throws a KeyError because key 'momentum' is not set.

pytorch/torch/optim/lr_scheduler.py

Lines 576 to 585
in
173f224

if cycle_momentum:

if 'momentum' not in optimizer.defaults:

raise ValueError('optimizer must support momentum with `cycle_momentum` option enabled')

base_momentums = self._format_param('base_momentum', optimizer, base_momentum)

if last_epoch == -1:

for momentum, group in zip(base_momentums, optimizer.param_groups):

group['momentum'] = momentum

self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))

self.max_momentums = self._format_param('max_momentum', optimizer, max_momentum)

I believe the last two lines should be indented under the if-condition as well.

I also think the expected behavior is that cycle_momentum is set to False by default when the passed in optimizer doesn't support momentum, instead of throwing an error.

Code example

optimizer = optim.Adam(params_to_update, lr=5e-4)
scheduler = CyclicLR(optimizer, base_lr=5e-6, max_lr=5e-2, cycle_momentum=False, step_size_up=2500)

System Info

In my setup, I am using the stable version of pytorch, but I copied over CyclicLR and relative imports from master and I am using it in my project.

Collecting environment information...
PyTorch version: 1.0.1.post2
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Debian GNU/Linux 9.8 (stretch)
GCC version: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla P4
Nvidia driver version: 410.72
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] intel-numpy==1.15.1
[pip3] numpy==1.15.1
[pip3] torch==1.0.1.post2
[pip3] torchvision==0.2.2.post3
[conda] blas 1.0 mkl
[conda] mkl 2019.1 144
[conda] mkl-service 1.1.2 py37he904b0f_5
[conda] mkl_fft 1.0.6 py37hd81dba3_0
[conda] mkl_random 1.0.2 py37hd81dba3_0
[conda] pytorch 1.0.1 py3.7_cuda10.0.130_cudnn7.4.2_2 pytorch
[conda] torchvision 0.2.2 py_3 pytorch