Repository: pytorch/vision

Issue:
ResNet FPN backbone
## 🐛 Bug

In https://github.com/pytorch/vision/blob/master/torchvision/models/detection/backbone_utils.py#L99, it says

> freeze layers only if pretrained backbone is used.

Therefore, if pretrained backbone is NOT used, any layers should be frozen.

But when I run:
```
from torchvision.models.detection.backbone_utils import resnet_fpn_backbone
resnet_fpn_backbone('resnet50', pretrained=False, norm_layer=None, trainable_layers=5)
```
I can see that parameters `bn1.weight, bn1.bias, fc.weight, fc.bias` are frozen on the [line 101](https://github.com/pytorch/vision/blob/master/torchvision/models/detection/backbone_utils.py#L101).

## To Reproduce

Steps to reproduce the behavior:

Run 
```
from torchvision.models.detection.backbone_utils import resnet_fpn_backbone
resnet_fpn_backbone('resnet50', pretrained=False, norm_layer=None, trainable_layers=5)
```

and see that on the [line 101](https://github.com/pytorch/vision/blob/master/torchvision/models/detection/backbone_utils.py#L101) parameters `bn1.weight, bn1.bias, fc.weight, fc.bias` are frozen despite that `trainable_layers=5` which should correspond to no frozen layers as written on the [line 83](https://github.com/pytorch/vision/blob/master/torchvision/models/detection/backbone_utils.py#L83).


## Expected behavior

No layers are frozen.

## Environment

PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 10.15.7 (x86_64)
GCC version: Could not collect
Clang version: 12.0.0 (clang-1200.0.32.29)
CMake version: version 3.18.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] pytorch-lightning==1.1.1
[pip3] pytorch-lightning-bolts==0.2.5rc1
[pip3] torch==1.7.0
[pip3] torchvision==0.8.1
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.5                      281    conda-forge
[conda] mkl-service               2.3.0            py38h0b31af3_0    conda-forge
[conda] mkl_fft                   1.2.0            py38hc64f4ea_0
[conda] mkl_random                1.1.1            py38h959d312_0
[conda] numpy                     1.19.2           py38h456fd55_0
[conda] numpy-base                1.19.2           py38hcfb5961_0
[conda] pytorch                   1.7.0                   py3.8_0    pytorch
[conda] pytorch-lightning         1.1.1              pyhd8ed1ab_0    conda-forge
[conda] pytorch-lightning-bolts   0.2.5rc1                 pypi_0    pypi
[conda] torchvision               0.8.1                  py38_cpu    pytorch




Discussion:
I can reproduce the bug and I confirm that `bn1.weight` and `bn1.bias` get frozen. The fc layers do not matter as they are not used by the FPN.

Resolving patch:
diff --git a/torchvision/models/detection/backbone_utils.py b/torchvision/models/detection/backbone_utils.py
--- a/torchvision/models/detection/backbone_utils.py
+++ b/torchvision/models/detection/backbone_utils.py
@@ -96,7 +96,8 @@ def resnet_fpn_backbone(
     # select layers that wont be frozen
     assert 0 <= trainable_layers <= 5
     layers_to_train = ['layer4', 'layer3', 'layer2', 'layer1', 'conv1'][:trainable_layers]
-    # freeze layers only if pretrained backbone is used
+    if trainable_layers == 5:
+        layers_to_train.append('bn1')
     for name, parameter in backbone.named_parameters():
         if all([not name.startswith(layer) for layer in layers_to_train]):
             parameter.requires_grad_(False)
@@ -152,7 +153,6 @@ def mobilenet_backbone(
     assert 0 <= trainable_layers <= num_stages
     freeze_before = num_stages if trainable_layers == 0 else stage_indices[num_stages - trainable_layers]
 
-    # freeze layers only if pretrained backbone is used
     for b in backbone[:freeze_before]:
         for parameter in b.parameters():
             parameter.requires_grad_(False)


Test patch: