Resnet weights should not be downloaded when building DETR by from_pretrained
Author: LukeForeverYoung
Pull request: no
Labels: name: Good First Issue

## Describe the bug
Hello, the building process of DETR in transformers uses timm to construct a resnet backbone.
```
backbone = create_model(name, pretrained=True, features_only=True, out_indices=(1, 2, 3, 4), **kwargs)
```
However, `pretrained=True` should only be enabled when building an initial DETR. When building DETR with `from_pretrained`, the pretrained weights of Resnet are redundant. If there is a lack of internet connection or a poor network environment, this can cause the code to get stuck even if the model is building from local files.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0
- Platform: Linux-5.11.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyTorch version (GPU?): 1.8.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

Models:
DETR




Comments:
Hi,

That totally makes sense. 

To solve this, we can add an attribute to `DetrConfig` called `use_pretrained_backbone` which is set to `True` by default, but can be set to `False`. In that way, when initializing a DetrModel as follows:

```
from transformers import DetrConfig, DetrModel

config = DetrConfig(use_pretrained_backbone=False)
model = DetrModel(config)
```
You will have both a convolutional backbone + encoder-decoder Transformer with randomly initialized weights.

In case you want to build a `DetrForObjectDetection` from a local pytorch_model.bin file:
```
from transformers import DetrForObjectDetection

model = DetrForObjectDetection.from_pretrained("path_to_local_folder", use_pretrained_backbone=False)
```
However, we're also planning to have a ResNet implementation in the library soon which is independent from timm, cc @FrancescoSaverioZuppichini. This might also solve the issue.

Hi, Thanks for pointing this out. I agree with @NielsRogge about adding a flag to control the backbones' weight download. Definitely, having our standalone implementation of ResNet embedded in Detr solves the issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I'd like to work on this. Please clarify if the standalone resnet implementation is preferred over the config flag or not.

Hi @divinit7 , thanks for your interest :) Yes, we want to add a flag to avoid downloading the weights. In the future, we would like to use our new ResNet implementation by default while keeping the option to use timm. 

Hello. Was this issue fixed ? I would like to work on this. 

> Hello. Was this issue fixed ? I would like to work on this.

Hey, please go ahead. I got busy with studies and work.