Repository: pytorch/vision

Issue:
RegNet in torchvision ?
## 🚀 Feature
Add RegNet trunks in torchvision

## Motivation
RegNets were proposed in this paper https://arxiv.org/pdf/2003.13678.pdf, they're showing very interesting performance and speed. They have been open sourced [already](https://github.com/facebookresearch/pycls), but are not usable in a straightforward way for people used to having reference models in torchvision. Another implementation is available in [ClassyVision](https://github.com/facebookresearch/ClassyVision/blob/master/classy_vision/models/regnet.py) (I'm a co-author of this one), but it does not cover all use cases.

## Pitch
Start from the ClassyVision RegNet support and implement RegNets in torchvision.

## Alternatives
Let users use RegNets from external implementations

## Additional context
This has been discussed with @pdollar, one of the RegNet authors. CC @fmassa 


cc @vfdev-5


Discussion:
@blefaudeux thanks for the suggestion ! For which tasks you think about for this model: at least classification, right ?

> Another implementation is available in ClassyVision (I'm a co-author of this one), but it does not cover all use cases.

Could you detail which use-cases ClassyVision implementation does not cover ?


Would you like to draft a PR for that ? Otherwise, me or someone else can do that.
> @blefaudeux thanks for the suggestion ! For which tasks you think about for this model: at least classification, right ?
> 
> > Another implementation is available in ClassyVision (I'm a co-author of this one), but it does not cover all use cases.
> 
> Could you detail which use-cases ClassyVision implementation does not cover ?

Oh, I just meant that not everyone is using ClassyVision obviously, I for instance came across users telling me that they were sticking to EfficientNets or ResNets because they were only wiling to consider Torchvision.

> Would you like to draft a PR for that ? Otherwise, me or someone else can do that.

I'm not sure what there is to know for a model to be supported by torchvision, apart from the raw code (which I can handle indeed, or anybody else, no preference). Are there some licence pre-requisites, pre-trained models, authorship constraints (validation from the original authors ?), things like that ? I don't have that much time right now, so if the requirements are clear (or minimal :)) I can handle that starting from the implementation in ClassyVision, else if there's some know-how required I would gladly stay around to assist but not do it myself

> Are there some licence pre-requisites, pre-trained models, authorship constraints (validation from the original authors ?), things like that ? 

Excellent question. I'd say we have to provide model's implementation and ImageNet pretrained weights. In the docstring we provide some information about the model, link to the paper etc. For example, MNasNet:
https://github.com/pytorch/vision/blob/190a5f8a32f9ae775d4379c55db7e2deb6eada6b/torchvision/models/mnasnet.py#L204-L208

ImageNet pretrained weights are often coming from retraining, but there are cases when they were converted let's say from TF weights etc. If torchvision's implementation will be a copy from ClassyVision, maybe we can reuse their weights if there are ones...

@fmassa can you also comment out this question, please ?

> I don't have that much time right now, so if the requirements are clear (or minimal :)) I can handle that starting from the implementation in ClassyVision, else if there's some know-how required I would gladly stay around to assist but not do it myself

No worries. I send a PR where I can copy and adapt the implementation from ClassyVision and you could check if we are correct etc.
> ImageNet pretrained weights are often coming from retraining, but there are cases when they were converted let's say from TF weights etc. If torchvision's implementation will be a copy from ClassyVision, maybe we can reuse their weights if there are ones...

I could provide reference weights from a ClassyVision ImageNet training, fairly easy to reproduce (if we're ok to limit this to some members of the RegNet family, probably not all of them :)). Another option is to translate the weights in the PyCls repo/model zoo, but that's some work because the model definition is not exactly the same (even if the actual underlying architecture is, of course). CC @mannatsingh from Classy

Yes, all model weights are available in pycls, not sure how easy it is to convert them to classy vision format. Link: https://github.com/facebookresearch/pycls/blob/master/MODEL_ZOO.md
@blefaudeux @pdollar thanks for the details ! So, we prefer to have here certain families of RegNet implemented as in Classy than the ones in pycls format right ? I let you define those families you think the most interesting for the users. 
If there are families we would like to include, but you do not have the weights, we can retrain them too. 

It would be, probably, helpful for the future to add some info, here in torchvision, on the reason why we prefered one implementation (classy) to another (pycls) if they are a bit different...

> @blefaudeux @pdollar thanks for the details ! So, we prefer to have here certain families of RegNet implemented as in Classy than the ones in pycls format right ? I let you define those families you think the most interesting for the users.
> If there are families we would like to include, but you do not have the weights, we can retrain them too.
> 
> It would be, probably, helpful for the future to add some info, here in torchvision, on the reason why we prefered one implementation (classy) to another (pycls) if they are a bit different...

Sorry for the imprecision, I forgot that the context is clearly not trivial, trying to address that :
- the pycls implementation is the original one, straight from FAIR (research), it exposes a lot of features which were mostly there for experimentation (architecture search) but are not useful once the best models in the family were found
- the ClassyVision implementation was tentatively more production ready, a little easier to read, with all the code to reproduce the paper but not more
- what I meant by "not all the regnets" in the above was that RegNets define a family of models, each of them being fully defined by a couple of coefficients, loosely comparable with ResNext for instance (change the width, depth, etc.. even if the scaling is a little more complex here). I just meant that one has to decide on a couple of models whose weights would be provided, for instance the models comparable to RN50, RN101, things like that, but covering each and every data point may not be needed. Additionally RegNets can scale to very big models for vision, 128GF or more, and training these would be very costly and probably not needed for torchvision users, or at least that was my assumption.
@blefaudeux thanks for the explanation ! I see better the context. Yes, this sounds good ! 

> I just meant that one has to decide on a couple of models whose weights would be provided, for instance the models comparable to RN50, RN101, things like that

If you have an idea of which RegNetX, RegNetY it would be good to provide with pretrained weights. Maybe, those from the paper's tables 5 and 6 :
- REGNETX-3.2GF
- REGNETX-6.4GF
- REGNETX-12GF
- REGNETX-4.0GF
- REGNETX-8.0GF
- REGNETY-400MF
- REGNETY-600MF
- REGNETY-800MF
- REGNETY-1.6GF
- REGNETY-4.0GF
- REGNETY-8.0GF

What do you think ?
Looks good to me ! Probably worth it writing a small tool to transcribe the weights in between pycls and this more streamlined implementation, it's something I could do. Not sure whether @mannatsingh would have something handy around to help on that ?
@blefaudeux I don't have anything available to convert pycls weights to classy, unfortunately. 
While you should be able to get similar results by training the models from scratch in Classy (we verified this for a few models), it might be easier to just convert the weights available in pycls.
What is the motivation for only including a subset of the models? Is it because they have to be retrained?

The reason I ask is that the benefit of RegNets is they give good accuracy models across a wide range of flop regimes, as opposed to say ResNet which typically is only optimized for a narrow range of 4GF-12GF (ResNet50-ResNet152). On the other hand RegNets can be good at very small sizes (200MF) and very large sizes (32GF). The very small and very large models are potentially the most interesting (for say mobile and state-of-the-art research). So I would advocate including the full range of models if possible.

It may be better to figure out how to convert pycls weights to classy vision weights. i don't know classy vision well, but i can't imagine it being that hard?!? (famous last words :P )
@pdollar it's not so much of an issue with Classy actually, it's just that some names have changed in between the pycls and classy implementation of the RegNets, I should be able to fix that by loading/mapping names/saving again. I'm just a bit wary of something really subtle there, but from a distance it should not be too hard indeed and probably the best thing to do
Hi,

I'm not sure we would be ready to add RegNet to torchvision yet.

We generally have a requirement of number of citations of the paper containing the model model before we include it in torchvision, similarly to what we do for PyTorch. We can reconsider this decision in 6 months.

Users can obtain RegNet variants by using PyTorch Image Models https://github.com/rwightman/pytorch-image-models , 
Thanks for the context @fmassa . 

@blefaudeux one more thing to clarify, in your original note, you mention - 
> Another implementation is available in ClassyVision (I'm a co-author of this one), but it does not cover all use cases.

From what I understand, torchvision's implementations are even more strict (even fewer configuration parameters allowed, if any) - @fmassa can correct me if I'm wrong. Also, you should be able to generate any RegNet using the Classy implementation, so I wouldn't want anyone reading this issue to get the wrong impression :)
@mannatsingh ah ok, that's not what I meant with `does not cover all use cases.` I just meant that not every user of torchvision is using classy vision obviously ([MoCo](https://github.com/facebookresearch/moco) randomly comes to mind, they support torchvision out of the box but that's all), so I just meant that _having an implementation in Classy was not enough to make RegNets truly accessible to the pytorch ecosystem_. 

I did not know about `PyTorch Image Models` linked above by @fmassa, seems that I'm not the only one, but if it does the trick then why not. I personally think that the citation metric can easily be gamed, but that's probably not a good enough incentive.
We try to keep the model implementations fairly simple -- the space of configurations for a model is potentially infinite, and trying to expose too many options can make things very hard to understand for users.

> I personally think that the citation metric can easily be gamed, but that's probably not a good enough incentive.

I agree that citations per se is not a perfect metric. But given the amount of research and activity around computer vision nowadays, with 100s of papers every year claiming SOTA, we need *some* metric to be able to define what should be in torchvision or not -- otherwise we will end up having 100s of models which are all respectively SOTA during their respective submission time, but being SOTA doesn't involve only architectural changes to the model but also to the training recipe.

We will be adding more information about what are the criteria for a model / op to be include in torchvision in the CONTRIBUTING.md file, and we have an issue tracking it in https://github.com/pytorch/vision/issues/2651 , thanks for the discussion, let me us know if there are anything that you disagree / would like to add to the discussion.
Let's keep this issue open for now to track RegNets
ResNets are heavy in terms of compute. MobileNets have less compute, but heavy in terms of memory access (Depthwise layers). RegNets provide a good balance between these extremes - especially RegNetX. Highly recommend them. Whish to see them as part of torchvision.
@mathmanu we are going to be adding RegNets in torchvision in the coming months
Thank you. 

Note that RegNetY models are memory intensive due to the Squeeze-and-Exctication layers. Some of the advantage that RegNet provides will be removed by those layers. Doubling the memory transfer requirement to get a very small lift in accuracy is not a good tradeoff - especially for embedded devices.

Hence I am also looking forward to torchvision having models without Squeeze-and-Exctication, say RegNetX. If possible lite versions on MobileNetV3 as well  - as a reference tensorflow/models provide such models  - they call it either minimialistic or lite. References:
https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet
https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite


Thanks for the pointers @mathmanu !

cc @datumbox for thoughts if we should add the lite variants of MobileNetV3 or if they are already covered by the `mobilenet_v3_small ` variant
It's straight forward to extend the existing implementation to support the lite/minimalist versions of MobileNetV3. I think the most appealing reason to support it is that the minimalist version uses ReLU instead of hard swish and this plays nice with quantization. 

Based on the aforementioned references the minimalistic version exchanges 5.6 accuracy points for roughly a 23% speed improvement which is definitely non trivial. On the other hand, the specific versions are not described on the [paper](https://arxiv.org/abs/1905.02244) so the only reference for it is the official repo.

To answer the question on whether we should add it or not, I think if there is enough demand from the community to support them we can do it. I personally don't have a use-case where it is required and I think there are other architectures we should offer before considering the minimalist MobileNetV3 models but I could be wrong.
![1](https://user-images.githubusercontent.com/699682/130479409-9c7467a9-ac97-4168-894e-4ddafef8433a.jpg)
![2](https://user-images.githubusercontent.com/699682/130479415-df8c18cb-d7b5-452b-a9b0-cc118be70fa3.jpg)

Another major industry player using RegNets :-D
In other words, RegNets are running in a million cars or so!
Hope this is a useful datapoint for this discussion.

Source: https://www.youtube.com/watch?v=j0z4FweCy4M&t=2829s


@pdollar Thanks for the input. Agreed and it's on the top of our list to add it on TorchVision soon. It's part of the https://github.com/pytorch/vision/issues/3911 epic.
In the following weeks, I will be working on upstreaming RegNet from Classy Vision to TorchVision.
Great to hear everyone, thank you :)
@kazhang please feel free to get in touch me for any details / code reviews :) 

Resolving patch:
diff --git a/hubconf.py b/hubconf.py
--- a/hubconf.py
+++ b/hubconf.py
@@ -17,6 +17,10 @@
     mnasnet1_3
 from torchvision.models.efficientnet import efficientnet_b0, efficientnet_b1, efficientnet_b2, \
     efficientnet_b3, efficientnet_b4, efficientnet_b5, efficientnet_b6, efficientnet_b7
+from torchvision.models.regnet import regnet_y_400mf, regnet_y_800mf, \
+    regnet_y_1_6gf, regnet_y_3_2gf, regnet_y_8gf, regnet_y_16gf, regnet_y_32gf, \
+    regnet_x_400mf, regnet_x_800mf, regnet_x_1_6gf, regnet_x_3_2gf, regnet_x_8gf, \
+    regnet_x_16gf, regnet_x_32gf
 
 # segmentation
 from torchvision.models.segmentation import fcn_resnet50, fcn_resnet101, \
diff --git a/torchvision/models/__init__.py b/torchvision/models/__init__.py
--- a/torchvision/models/__init__.py
+++ b/torchvision/models/__init__.py
@@ -9,6 +9,7 @@
 from .mnasnet import *
 from .shufflenetv2 import *
 from .efficientnet import *
+from .regnet import *
 from . import segmentation
 from . import detection
 from . import video
diff --git a/torchvision/models/regnet.py b/torchvision/models/regnet.py
new file mode 100644
--- /dev/null
+++ b/torchvision/models/regnet.py
@@ -0,0 +1,578 @@
+# Modified from
+# https://github.com/facebookresearch/ClassyVision/blob/main/classy_vision/models/anynet.py
+# https://github.com/facebookresearch/ClassyVision/blob/main/classy_vision/models/regnet.py
+
+
+import math
+import torch
+
+from collections import OrderedDict
+from functools import partial
+from typing import Any, Callable, List, Optional, Tuple
+from torch import nn, Tensor
+
+from .._internally_replaced_utils import load_state_dict_from_url
+from torchvision.models.mobilenetv2 import ConvBNActivation, _make_divisible
+from torchvision.models.efficientnet import SqueezeExcitation
+
+
+__all__ = ["RegNet", "regnet_y_400mf", "regnet_y_800mf", "regnet_y_1_6gf",
+           "regnet_y_3_2gf", "regnet_y_8gf", "regnet_y_16gf", "regnet_y_32gf",
+           "regnet_x_400mf", "regnet_x_800mf", "regnet_x_1_6gf", "regnet_x_3_2gf",
+           "regnet_x_8gf", "regnet_x_16gf", "regnet_x_32gf"]
+
+
+model_urls = {
+    "regnet_y_400mf": "https://download.pytorch.org/models/regnet_y_400mf-540e987b.pth",
+    "regnet_y_800mf": "https://download.pytorch.org/models/regnet_y_800mf-1b27b58c.pth",
+    "regnet_y_8gf": "https://download.pytorch.org/models/regnet_y_8gf-49ff86b5.pth",
+    "regnet_x_400mf": "https://download.pytorch.org/models/regnet_x_400mf-adf1edd5.pth",
+    "regnet_x_800mf": "https://download.pytorch.org/models/regnet_x_800mf-5cb79b7e.pth",
+    "regnet_x_8gf": "https://download.pytorch.org/models/regnet_x_8gf-4c4e575e.pth",
+}
+
+
+class SimpleStemIN(ConvBNActivation):
+    """Simple stem for ImageNet: 3x3, BN, ReLU."""
+
+    def __init__(
+        self,
+        width_in: int,
+        width_out: int,
+        norm_layer: Callable[..., nn.Module],
+        activation_layer: Callable[..., nn.Module],
+    ) -> None:
+        super().__init__(width_in, width_out, kernel_size=3, stride=2,
+                         norm_layer=norm_layer, activation_layer=activation_layer)
+
+
+class BottleneckTransform(nn.Sequential):
+    """Bottleneck transformation: 1x1, 3x3 [+SE], 1x1."""
+
+    def __init__(
+        self,
+        width_in: int,
+        width_out: int,
+        stride: int,
+        norm_layer: Callable[..., nn.Module],
+        activation_layer: Callable[..., nn.Module],
+        group_width: int,
+        bottleneck_multiplier: float,
+        se_ratio: Optional[float],
+    ) -> None:
+        layers: OrderedDict[str, nn.Module] = OrderedDict()
+        w_b = int(round(width_out * bottleneck_multiplier))
+        g = w_b // group_width
+
+        layers["a"] = ConvBNActivation(width_in, w_b, kernel_size=1, stride=1,
+                                       norm_layer=norm_layer, activation_layer=activation_layer)
+        layers["b"] = ConvBNActivation(w_b, w_b, kernel_size=3, stride=stride, groups=g,
+                                       norm_layer=norm_layer, activation_layer=activation_layer)
+
+        if se_ratio:
+            # The SE reduction ratio is defined with respect to the
+            # beginning of the block
+            width_se_out = int(round(se_ratio * width_in))
+            layers["se"] = SqueezeExcitation(
+                input_channels=w_b,
+                squeeze_channels=width_se_out,
+                activation=activation_layer,
+            )
+
+        layers["c"] = ConvBNActivation(w_b, width_out, kernel_size=1, stride=1,
+                                       norm_layer=norm_layer, activation_layer=nn.Identity)
+        super().__init__(layers)
+
+
+class ResBottleneckBlock(nn.Module):
+    """Residual bottleneck block: x + F(x), F = bottleneck transform."""
+
+    def __init__(
+        self,
+        width_in: int,
+        width_out: int,
+        stride: int,
+        norm_layer: Callable[..., nn.Module],
+        activation_layer: Callable[..., nn.Module],
+        group_width: int = 1,
+        bottleneck_multiplier: float = 1.0,
+        se_ratio: Optional[float] = None,
+    ) -> None:
+        super().__init__()
+
+        # Use skip connection with projection if shape changes
+        self.proj = None
+        should_proj = (width_in != width_out) or (stride != 1)
+        if should_proj:
+            self.proj = ConvBNActivation(width_in, width_out, kernel_size=1,
+                                         stride=stride, norm_layer=norm_layer, activation_layer=nn.Identity)
+        self.f = BottleneckTransform(
+            width_in,
+            width_out,
+            stride,
+            norm_layer,
+            activation_layer,
+            group_width,
+            bottleneck_multiplier,
+            se_ratio,
+        )
+        self.activation = activation_layer(inplace=True)
+
+    def forward(self, x: Tensor) -> Tensor:
+        if self.proj is not None:
+            x = self.proj(x) + self.f(x)
+        else:
+            x = x + self.f(x)
+        return self.activation(x)
+
+
+class AnyStage(nn.Sequential):
+    """AnyNet stage (sequence of blocks w/ the same output shape)."""
+
+    def __init__(
+        self,
+        width_in: int,
+        width_out: int,
+        stride: int,
+        depth: int,
+        block_constructor: Callable[..., nn.Module],
+        norm_layer: Callable[..., nn.Module],
+        activation_layer: Callable[..., nn.Module],
+        group_width: int,
+        bottleneck_multiplier: float,
+        se_ratio: Optional[float] = None,
+        stage_index: int = 0,
+    ) -> None:
+        super().__init__()
+
+        for i in range(depth):
+            block = block_constructor(
+                width_in if i == 0 else width_out,
+                width_out,
+                stride if i == 0 else 1,
+                norm_layer,
+                activation_layer,
+                group_width,
+                bottleneck_multiplier,
+                se_ratio,
+            )
+
+            self.add_module(f"block{stage_index}-{i}", block)
+
+
+class BlockParams:
+    def __init__(
+        self,
+        depths: List[int],
+        widths: List[int],
+        group_widths: List[int],
+        bottleneck_multipliers: List[float],
+        strides: List[int],
+        se_ratio: Optional[float] = None,
+    ) -> None:
+        self.depths = depths
+        self.widths = widths
+        self.group_widths = group_widths
+        self.bottleneck_multipliers = bottleneck_multipliers
+        self.strides = strides
+        self.se_ratio = se_ratio
+
+    @classmethod
+    def from_init_params(
+        cls,
+        depth: int,
+        w_0: int,
+        w_a: float,
+        w_m: float,
+        group_width: int,
+        bottleneck_multiplier: float = 1.0,
+        se_ratio: Optional[float] = None,
+        **kwargs: Any,
+    ) -> "BlockParams":
+        """
+        Programatically compute all the per-block settings,
+        given the RegNet parameters.
+
+        The first step is to compute the quantized linear block parameters,
+        in log space. Key parameters are:
+        - `w_a` is the width progression slope
+        - `w_0` is the initial width
+        - `w_m` is the width stepping in the log space
+
+        In other terms
+        `log(block_width) = log(w_0) + w_m * block_capacity`,
+        with `bock_capacity` ramping up following the w_0 and w_a params.
+        This block width is finally quantized to multiples of 8.
+
+        The second step is to compute the parameters per stage,
+        taking into account the skip connection and the final 1x1 convolutions.
+        We use the fact that the output width is constant within a stage.
+        """
+
+        QUANT = 8
+        STRIDE = 2
+
+        if w_a < 0 or w_0 <= 0 or w_m <= 1 or w_0 % 8 != 0:
+            raise ValueError("Invalid RegNet settings")
+        # Compute the block widths. Each stage has one unique block width
+        widths_cont = torch.arange(depth) * w_a + w_0
+        block_capacity = torch.round(torch.log(widths_cont / w_0) / math.log(w_m))
+        block_widths = (
+            torch.round(torch.divide(w_0 * torch.pow(w_m, block_capacity), QUANT))
+            * QUANT
+        ).int().tolist()
+        num_stages = len(set(block_widths))
+
+        # Convert to per stage parameters
+        split_helper = zip(
+            block_widths + [0],
+            [0] + block_widths,
+            block_widths + [0],
+            [0] + block_widths,
+        )
+        splits = [w != wp or r != rp for w, wp, r, rp in split_helper]
+
+        stage_widths = [w for w, t in zip(block_widths, splits[:-1]) if t]
+        stage_depths = torch.diff(torch.tensor([d for d, t in enumerate(splits) if t])).int().tolist()
+
+        strides = [STRIDE] * num_stages
+        bottleneck_multipliers = [bottleneck_multiplier] * num_stages
+        group_widths = [group_width] * num_stages
+
+        # Adjust the compatibility of stage widths and group widths
+        stage_widths, group_widths = cls._adjust_widths_groups_compatibilty(
+            stage_widths, bottleneck_multipliers, group_widths
+        )
+
+        return cls(
+            depths=stage_depths,
+            widths=stage_widths,
+            group_widths=group_widths,
+            bottleneck_multipliers=bottleneck_multipliers,
+            strides=strides,
+            se_ratio=se_ratio,
+        )
+
+    def _get_expanded_params(self):
+        return zip(
+            self.widths, self.strides, self.depths, self.group_widths, self.bottleneck_multipliers
+        )
+
+    @staticmethod
+    def _adjust_widths_groups_compatibilty(
+            stage_widths: List[int], bottleneck_ratios: List[float],
+            group_widths: List[int]) -> Tuple[List[int], List[int]]:
+        """
+        Adjusts the compatibility of widths and groups,
+        depending on the bottleneck ratio.
+        """
+        # Compute all widths for the current settings
+        widths = [int(w * b) for w, b in zip(stage_widths, bottleneck_ratios)]
+        group_widths_min = [min(g, w_bot) for g, w_bot in zip(group_widths, widths)]
+
+        # Compute the adjusted widths so that stage and group widths fit
+        ws_bot = [_make_divisible(w_bot, g) for w_bot, g in zip(widths, group_widths_min)]
+        stage_widths = [int(w_bot / b) for w_bot, b in zip(ws_bot, bottleneck_ratios)]
+        return stage_widths, group_widths_min
+
+
+class RegNet(nn.Module):
+    def __init__(
+        self,
+        block_params: BlockParams,
+        num_classes: int = 1000,
+        stem_width: int = 32,
+        stem_type: Optional[Callable[..., nn.Module]] = None,
+        block_type: Optional[Callable[..., nn.Module]] = None,
+        norm_layer: Optional[Callable[..., nn.Module]] = None,
+        activation: Optional[Callable[..., nn.Module]] = None,
+    ) -> None:
+        super().__init__()
+
+        if stem_type is None:
+            stem_type = SimpleStemIN
+        if norm_layer is None:
+            norm_layer = nn.BatchNorm2d
+        if block_type is None:
+            block_type = ResBottleneckBlock
+        if activation is None:
+            activation = nn.ReLU
+
+        # Ad hoc stem
+        self.stem = stem_type(
+            3,  # width_in
+            stem_width,
+            norm_layer,
+            activation,
+        )
+
+        current_width = stem_width
+
+        blocks = []
+        for i, (
+            width_out,
+            stride,
+            depth,
+            group_width,
+            bottleneck_multiplier,
+        ) in enumerate(block_params._get_expanded_params()):
+            blocks.append(
+                (
+                    f"block{i+1}",
+                    AnyStage(
+                        current_width,
+                        width_out,
+                        stride,
+                        depth,
+                        block_type,
+                        norm_layer,
+                        activation,
+                        group_width,
+                        bottleneck_multiplier,
+                        block_params.se_ratio,
+                        stage_index=i + 1,
+                    ),
+                )
+            )
+
+            current_width = width_out
+
+        self.trunk_output = nn.Sequential(OrderedDict(blocks))
+
+        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
+        self.fc = nn.Linear(in_features=current_width, out_features=num_classes)
+
+        # Init weights and good to go
+        self._reset_parameters()
+
+    def forward(self, x: Tensor) -> Tensor:
+        x = self.stem(x)
+        x = self.trunk_output(x)
+
+        x = self.avgpool(x)
+        x = x.flatten(start_dim=1)
+        x = self.fc(x)
+
+        return x
+
+    def _reset_parameters(self) -> None:
+        # Performs ResNet-style weight initialization
+        for m in self.modules():
+            if isinstance(m, nn.Conv2d):
+                # Note that there is no bias due to BN
+                fan_out = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
+                nn.init.normal_(m.weight, mean=0.0, std=math.sqrt(2.0 / fan_out))
+            elif isinstance(m, nn.BatchNorm2d):
+                nn.init.ones_(m.weight)
+                nn.init.zeros_(m.bias)
+            elif isinstance(m, nn.Linear):
+                nn.init.normal_(m.weight, mean=0.0, std=0.01)
+                nn.init.zeros_(m.bias)
+
+
+def _regnet(arch: str, block_params: BlockParams, pretrained: bool, progress: bool, **kwargs: Any) -> RegNet:
+    model = RegNet(block_params, norm_layer=partial(nn.BatchNorm2d, eps=1e-05, momentum=0.1), **kwargs)
+    if pretrained:
+        if arch not in model_urls:
+            raise ValueError(f"No checkpoint is available for model type {arch}")
+        state_dict = load_state_dict_from_url(model_urls[arch], progress=progress)
+        model.load_state_dict(state_dict)
+    return model
+
+
+def regnet_y_400mf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetY_400MF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=16, w_0=48, w_a=27.89, w_m=2.09,
+                                          group_width=8, se_ratio=0.25, **kwargs)
+    return _regnet("regnet_y_400mf", params, pretrained, progress, **kwargs)
+
+
+def regnet_y_800mf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetY_800MF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=14, w_0=56, w_a=38.84, w_m=2.4,
+                                          group_width=16, se_ratio=0.25, **kwargs)
+    return _regnet("regnet_y_800mf", params, pretrained, progress, **kwargs)
+
+
+def regnet_y_1_6gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetY_1.6GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=27, w_0=48, w_a=20.71, w_m=2.65,
+                                          group_width=24, se_ratio=0.25, **kwargs)
+    return _regnet("regnet_y_1_6gf", params, pretrained, progress, **kwargs)
+
+
+def regnet_y_3_2gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetY_3.2GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=21, w_0=80, w_a=42.63, w_m=2.66,
+                                          group_width=24, se_ratio=0.25, **kwargs)
+    return _regnet("regnet_y_3_2gf", params, pretrained, progress, **kwargs)
+
+
+def regnet_y_8gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetY_8GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=17, w_0=192, w_a=76.82, w_m=2.19,
+                                          group_width=56, se_ratio=0.25, **kwargs)
+    return _regnet("regnet_y_8gf", params, pretrained, progress, **kwargs)
+
+
+def regnet_y_16gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetY_16GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=18, w_0=200, w_a=106.23, w_m=2.48,
+                                          group_width=112, se_ratio=0.25, **kwargs)
+    return _regnet("regnet_y_16gf", params, pretrained, progress, **kwargs)
+
+
+def regnet_y_32gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetY_32GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=20, w_0=232, w_a=115.89, w_m=2.53,
+                                          group_width=232, se_ratio=0.25, **kwargs)
+    return _regnet("regnet_y_32gf", params, pretrained, progress, **kwargs)
+
+
+def regnet_x_400mf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetX_400MF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=22, w_0=24, w_a=24.48, w_m=2.54,
+                                          group_width=16, **kwargs)
+    return _regnet("regnet_x_400mf", params, pretrained, progress, **kwargs)
+
+
+def regnet_x_800mf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetX_800MF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=16, w_0=56, w_a=35.73, w_m=2.28,
+                                          group_width=16, **kwargs)
+    return _regnet("regnet_x_800mf", params, pretrained, progress, **kwargs)
+
+
+def regnet_x_1_6gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetX_1.6GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=18, w_0=80, w_a=34.01, w_m=2.25,
+                                          group_width=24, **kwargs)
+    return _regnet("regnet_x_1_6gf", params, pretrained, progress, **kwargs)
+
+
+def regnet_x_3_2gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetX_3.2GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=25, w_0=88, w_a=26.31, w_m=2.25,
+                                          group_width=48, **kwargs)
+    return _regnet("regnet_x_3_2gf", params, pretrained, progress, **kwargs)
+
+
+def regnet_x_8gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetX_8GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=23, w_0=80, w_a=49.56, w_m=2.88,
+                                          group_width=120, **kwargs)
+    return _regnet("regnet_x_8gf", params, pretrained, progress, **kwargs)
+
+
+def regnet_x_16gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetX_16GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=22, w_0=216, w_a=55.59, w_m=2.1,
+                                          group_width=128, **kwargs)
+    return _regnet("regnet_x_16gf", params, pretrained, progress, **kwargs)
+
+
+def regnet_x_32gf(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> RegNet:
+    """
+    Constructs a RegNetX_32GF architecture from
+    `"Designing Network Design Spaces" <https://arxiv.org/abs/2003.13678>`_.
+
+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+        progress (bool): If True, displays a progress bar of the download to stderr
+    """
+    params = BlockParams.from_init_params(depth=23, w_0=320, w_a=69.86, w_m=2.0,
+                                          group_width=168, **kwargs)
+    return _regnet("regnet_x_32gf", params, pretrained, progress, **kwargs)
+
+# TODO(kazhang): Add RegNetZ_500MF and RegNetZ_4GF


Test patch:
diff --git a/test/expect/ModelTester.test_regnet_x_16gf_expect.pkl b/test/expect/ModelTester.test_regnet_x_16gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_x_16gf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_x_1_6gf_expect.pkl b/test/expect/ModelTester.test_regnet_x_1_6gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_x_1_6gf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_x_32gf_expect.pkl b/test/expect/ModelTester.test_regnet_x_32gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_x_32gf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_x_3_2gf_expect.pkl b/test/expect/ModelTester.test_regnet_x_3_2gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_x_3_2gf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_x_400mf_expect.pkl b/test/expect/ModelTester.test_regnet_x_400mf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_x_400mf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_x_800mf_expect.pkl b/test/expect/ModelTester.test_regnet_x_800mf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_x_800mf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_x_8gf_expect.pkl b/test/expect/ModelTester.test_regnet_x_8gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_x_8gf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_y_16gf_expect.pkl b/test/expect/ModelTester.test_regnet_y_16gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_y_16gf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_y_1_6gf_expect.pkl b/test/expect/ModelTester.test_regnet_y_1_6gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_y_1_6gf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_y_32gf_expect.pkl b/test/expect/ModelTester.test_regnet_y_32gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_y_32gf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_y_3_2gf_expect.pkl b/test/expect/ModelTester.test_regnet_y_3_2gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_y_3_2gf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_y_400mf_expect.pkl b/test/expect/ModelTester.test_regnet_y_400mf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_y_400mf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_y_800mf_expect.pkl b/test/expect/ModelTester.test_regnet_y_800mf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_y_800mf_expect.pkl differ
diff --git a/test/expect/ModelTester.test_regnet_y_8gf_expect.pkl b/test/expect/ModelTester.test_regnet_y_8gf_expect.pkl
new file mode 100644
Binary files /dev/null and b/test/expect/ModelTester.test_regnet_y_8gf_expect.pkl differ