Repository: microsoft/torchgeo
Pull request #2677: SoftCon pretrained models
Author: login: wangyi111
Labels: name: documentation
description: Improvements or additions to documentation
name: models
description: Models and pretrained weights
name: testing
description: Continuous integration testing
name: backwards-incompatible
description: Changes that are not backwards compatible
Created: 2025-03-25T20:35:31Z | Merged: 2025-03-29T13:03:20Z

Add [SoftCon](https://github.com/zhu-xlab/softcon) pretrained models for Sentinel-1 GRD (2 bands) and Sentinel-2 TOA (13 bands). SoftCon conducts multi-label soft contrastive learning with land cover semantics, as well as continual pretraining from DINO (resnet) or DINOv2 (vits).

![image](https://github.com/user-attachments/assets/0e914033-0411-4af7-a60f-4fd086b9c334)


There're 6 new weights in total, including 1 ResNet and 2 ViTs for both Sentinel-1 and Sentinel-2:
- ResNet50
- ViT-S/14 DINOv2
- ViT-B/14 DINOv2

DINOv2 ViTs have been implemented in timm since [v0.9.1](https://github.com/huggingface/pytorch-image-models/releases/tag/v0.9.1).


Diff:
diff --git a/docs/api/models.rst b/docs/api/models.rst
index 4262b306f61..30f7a84b555 100644
--- a/docs/api/models.rst
+++ b/docs/api/models.rst
@@ -89,10 +89,14 @@ Vision Transformer
 .. autofunction:: vit_base_patch16_224
 .. autofunction:: vit_large_patch16_224
 .. autofunction:: vit_huge_patch14_224
+.. autofunction:: vit_small_patch14_dinov2
+.. autofunction:: vit_base_patch14_dinov2
 .. autoclass:: ViTSmall16_Weights
 .. autoclass:: ViTBase16_Weights
 .. autoclass:: ViTLarge16_Weights
 .. autoclass:: ViTHuge14_Weights
+.. autoclass:: ViTSmall14_DINOv2_Weights
+.. autoclass:: ViTBase14_DINOv2_Weights
 
 Utility Functions
 ^^^^^^^^^^^^^^^^^
diff --git a/docs/api/weights/sentinel1.csv b/docs/api/weights/sentinel1.csv
index fd55e54fbd6..75afb518232 100644
--- a/docs/api/weights/sentinel1.csv
+++ b/docs/api/weights/sentinel1.csv
@@ -1,6 +1,7 @@
 Weight,Channels,Source,Citation,License
-ResNet50_Weights.SENTINEL1_ALL_DECUR, 2,`link <https://github.com/zhu-xlab/DeCUR>`__,`link <https://arxiv.org/abs/2309.05300>`__,"Apache-2.0"
-ResNet50_Weights.SENTINEL1_ALL_MOCO, 2,`link <https://github.com/zhu-xlab/SSL4EO-S12>`__,`link <https://arxiv.org/abs/2211.07044>`__,"CC-BY-4.0"
+ResNet50_Weights.SENTINEL1_GRD_DECUR, 2,`link <https://github.com/zhu-xlab/DeCUR>`__,`link <https://arxiv.org/abs/2309.05300>`__,"Apache-2.0"
+ResNet50_Weights.SENTINEL1_GRD_MOCO, 2,`link <https://github.com/zhu-xlab/SSL4EO-S12>`__,`link <https://arxiv.org/abs/2211.07044>`__,"CC-BY-4.0"
+ResNet50_Weights.SENTINEL1_GRD_SOFTCON, 2,`link <https://github.com/zhu-xlab/softcon>`__,`link <https://arxiv.org/abs/2405.20462>`__,"CC-BY-4.0"
 ViTSmall16_Weights.SENTINEL1_GRD_MAE,2,`link <https://github.com/zhu-xlab/SSL4EO-S12>`__,`link <https://arxiv.org/abs/2211.07044>`__,"CC-BY-4.0"
 ViTSmall16_Weights.SENTINEL1_GRD_FGMAE,2,`link <https://github.com/zhu-xlab/FGMAE>`__,`link <https://arxiv.org/abs/2310.18653>`__,"CC-BY-4.0"
 ViTBase16_Weights.SENTINEL1_GRD_MAE,2,`link <https://github.com/zhu-xlab/SSL4EO-S12>`__,`link <https://arxiv.org/abs/2211.07044>`__,"CC-BY-4.0"
@@ -9,5 +10,7 @@ ViTLarge16_Weights.SENTINEL1_GRD_MAE,2,`link <https://github.com/zhu-xlab/SSL4EO
 ViTLarge16_Weights.SENTINEL1_GRD_FGMAE,2,`link <https://github.com/zhu-xlab/FGMAE>`__,`link <https://arxiv.org/abs/2310.18653>`__,"CC-BY-4.0"
 ViTHuge14_Weights.SENTINEL1_GRD_MAE,2,`link <https://github.com/zhu-xlab/SSL4EO-S12>`__,`link <https://arxiv.org/abs/2211.07044>`__,"CC-BY-4.0"
 ViTHuge14_Weights.SENTINEL1_GRD_FGMAE,2,`link <https://github.com/zhu-xlab/FGMAE>`__,`link <https://arxiv.org/abs/2310.18653>`__,"CC-BY-4.0"
+ViTSmall14_DINOv2_Weights.SENTINEL1_GRD_SOFTCON, 2,`link <https://github.com/zhu-xlab/softcon>`__,`link <https://arxiv.org/abs/2405.20462>`__,"CC-BY-4.0"
+ViTBase14_DINOv2_Weights.SENTINEL1_GRD_SOFTCON, 2,`link <https://github.com/zhu-xlab/softcon>`__,`link <https://arxiv.org/abs/2405.20462>`__,"CC-BY-4.0"
 Swin_V2_B_Weights.SENTINEL1_MI_SATLAS,2,`link <https://github.com/allenai/satlas>`__,`link <https://arxiv.org/abs/2211.15660>`__,ODC-BY
 Swin_V2_B_Weights.SENTINEL1_SI_SATLAS,2,`link <https://github.com/allenai/satlas>`__,`link <https://arxiv.org/abs/2211.15660>`__,ODC-BY
diff --git a/docs/api/weights/sentinel2.csv b/docs/api/weights/sentinel2.csv
index ac2441fa5e4..348083c7655 100644
--- a/docs/api/weights/sentinel2.csv
+++ b/docs/api/weights/sentinel2.csv
@@ -5,6 +5,7 @@ ResNet18_Weights.SENTINEL2_RGB_SECO, 3,`link <https://github.com/ServiceNow/seas
 ResNet50_Weights.SENTINEL2_ALL_DECUR,13,`link <https://github.com/zhu-xlab/DeCUR>`__,`link <https://arxiv.org/abs/2309.05300>`__,"Apache-2.0",,,,
 ResNet50_Weights.SENTINEL2_ALL_DINO,13,`link <https://github.com/zhu-xlab/SSL4EO-S12>`__,`link <https://arxiv.org/abs/2211.07044>`__,"CC-BY-4.0",90.7,99.1,63.6,
 ResNet50_Weights.SENTINEL2_ALL_MOCO,13,`link <https://github.com/zhu-xlab/SSL4EO-S12>`__,`link <https://arxiv.org/abs/2211.07044>`__,"CC-BY-4.0",91.8,99.1,60.9,
+ResNet50_Weights.SENTINEL2_ALL_SOFTCON, 13,`link <https://github.com/zhu-xlab/softcon>`__,`link <https://arxiv.org/abs/2405.20462>`__,"CC-BY-4.0"
 ResNet50_Weights.SENTINEL2_MI_MS_SATLAS,9,`link <https://github.com/allenai/satlaspretrain_models>`__,`link <https://arxiv.org/abs/2211.15660>`__,ODC-BY,,,,
 ResNet50_Weights.SENTINEL2_MI_RGB_SATLAS,3,`link <https://github.com/allenai/satlaspretrain_models>`__,`link <https://arxiv.org/abs/2211.15660>`__,ODC-BY,,,,
 ResNet50_Weights.SENTINEL2_SI_MS_SATLAS,9,`link <https://github.com/allenai/satlaspretrain_models>`__,`link <https://arxiv.org/abs/2211.15660>`__,ODC-BY,,,,
@@ -25,6 +26,8 @@ ViTLarge16_Weights.SENTINEL2_ALL_MAE,13,`link <https://github.com/zhu-xlab/SSL4E
 ViTLarge16_Weights.SENTINEL2_ALL_FGMAE,13,`link <https://github.com/zhu-xlab/FGMAE>`__,`link <https://arxiv.org/abs/2310.18653>`__,"CC-BY-4.0",,,,
 ViTHuge14_Weights.SENTINEL2_ALL_MAE,13,`link <https://github.com/zhu-xlab/SSL4EO-S12>`__,`link <https://arxiv.org/abs/2211.07044>`__,"CC-BY-4.0",,,,
 ViTHuge14_Weights.SENTINEL2_ALL_FGMAE,13,`link <https://github.com/zhu-xlab/FGMAE>`__,`link <https://arxiv.org/abs/2310.18653>`__,"CC-BY-4.0",,,,
+ViTSmall14_DINOv2_Weights.SENTINEL2_ALL_SOFTCON, 13,`link <https://github.com/zhu-xlab/softcon>`__,`link <https://arxiv.org/abs/2405.20462>`__,"CC-BY-4.0"
+ViTBase14_DINOv2_Weights.SENTINEL2_ALL_SOFTCON, 13,`link <https://github.com/zhu-xlab/softcon>`__,`link <https://arxiv.org/abs/2405.20462>`__,"CC-BY-4.0"
 Swin_V2_T_Weights.SENTINEL2_MI_MS_SATLAS,9,`link <https://github.com/allenai/satlas>`__,`link <https://arxiv.org/abs/2211.15660>`__,ODC-BY,,,,
 Swin_V2_T_Weights.SENTINEL2_MI_RGB_SATLAS,3,`link <https://github.com/allenai/satlas>`__,`link <https://arxiv.org/abs/2211.15660>`__,ODC-BY,,,,
 Swin_V2_T_Weights.SENTINEL2_SI_MS_SATLAS,9,`link <https://github.com/allenai/satlas>`__,`link <https://arxiv.org/abs/2211.15660>`__,ODC-BY,,,,
diff --git a/hubconf.py b/hubconf.py
index 20a39975af5..449ac3620f1 100644
--- a/hubconf.py
+++ b/hubconf.py
@@ -19,9 +19,11 @@
     scalemae_large_patch16,
     swin_v2_b,
     swin_v2_t,
+    vit_base_patch14_dinov2,
     vit_base_patch16_224,
     vit_huge_patch14_224,
     vit_large_patch16_224,
+    vit_small_patch14_dinov2,
     vit_small_patch16_224,
 )
 
@@ -37,9 +39,11 @@
     'scalemae_large_patch16',
     'swin_v2_b',
     'swin_v2_t',
+    'vit_base_patch14_dinov2',
     'vit_base_patch16_224',
     'vit_huge_patch14_224',
     'vit_large_patch16_224',
+    'vit_small_patch14_dinov2',
     'vit_small_patch16_224',
 )
 
diff --git a/tests/models/test_vit.py b/tests/models/test_vit.py
index c6b53489f4a..bffa5968201 100644
--- a/tests/models/test_vit.py
+++ b/tests/models/test_vit.py
@@ -11,13 +11,17 @@
 from torchvision.models._api import WeightsEnum
 
 from torchgeo.models import (
+    ViTBase14_DINOv2_Weights,
     ViTBase16_Weights,
     ViTHuge14_Weights,
     ViTLarge16_Weights,
+    ViTSmall14_DINOv2_Weights,
     ViTSmall16_Weights,
+    vit_base_patch14_dinov2,
     vit_base_patch16_224,
     vit_huge_patch14_224,
     vit_large_patch16_224,
+    vit_small_patch14_dinov2,
     vit_small_patch16_224,
 )
 
@@ -228,3 +232,115 @@ def test_transforms(self, mocked_weights: WeightsEnum) -> None:
     @pytest.mark.slow
     def test_vit_download(self, weights: WeightsEnum) -> None:
         vit_huge_patch14_224(weights=weights)
+
+
+class TestViTSmall14_DINOv2:
+    @pytest.fixture(params=[*ViTSmall14_DINOv2_Weights])
+    def weights(self, request: SubRequest) -> WeightsEnum:
+        return request.param
+
+    @pytest.fixture
+    def mocked_weights(
+        self,
+        tmp_path: Path,
+        monkeypatch: MonkeyPatch,
+        weights: WeightsEnum,
+        features_only: bool,
+        load_state_dict_from_url: None,
+    ) -> WeightsEnum:
+        path = tmp_path / f'{weights}.pth'
+        model = timm.create_model(
+            weights.meta['model'],
+            in_chans=weights.meta['in_chans'],
+            img_size=weights.meta['img_size'],
+            features_only=features_only,
+        )
+        target_model = model.model if features_only else model
+        torch.save(target_model.state_dict(), path)
+        try:
+            monkeypatch.setattr(weights.value, 'url', str(path))
+        except AttributeError:
+            monkeypatch.setattr(weights, 'url', str(path))
+        return weights
+
+    def test_vit(self) -> None:
+        vit_small_patch14_dinov2()
+
+    def test_vit_weights(
+        self, mocked_weights: WeightsEnum, features_only: bool
+    ) -> None:
+        vit_small_patch14_dinov2(weights=mocked_weights, features_only=features_only)
+
+    def test_bands(self, mocked_weights: WeightsEnum) -> None:
+        if 'bands' in mocked_weights.meta:
+            assert len(mocked_weights.meta['bands']) == mocked_weights.meta['in_chans']
+
+    def test_transforms(self, mocked_weights: WeightsEnum) -> None:
+        c = mocked_weights.meta['in_chans']
+        img_size = mocked_weights.meta['img_size']
+        if isinstance(img_size, int):
+            h = w = img_size
+        else:
+            h, w = img_size
+        sample = {'image': torch.arange(c * h * w, dtype=torch.float).view(c, h, w)}
+        mocked_weights.transforms(sample)
+
+    @pytest.mark.slow
+    def test_vit_download(self, weights: WeightsEnum) -> None:
+        vit_small_patch14_dinov2(weights=weights)
+
+
+class TestViTBase14_DINOv2:
+    @pytest.fixture(params=[*ViTBase14_DINOv2_Weights])
+    def weights(self, request: SubRequest) -> WeightsEnum:
+        return request.param
+
+    @pytest.fixture
+    def mocked_weights(
+        self,
+        tmp_path: Path,
+        monkeypatch: MonkeyPatch,
+        weights: WeightsEnum,
+        features_only: bool,
+        load_state_dict_from_url: None,
+    ) -> WeightsEnum:
+        path = tmp_path / f'{weights}.pth'
+        model = timm.create_model(
+            weights.meta['model'],
+            in_chans=weights.meta['in_chans'],
+            img_size=weights.meta['img_size'],
+            features_only=features_only,
+        )
+        target_model = model.model if features_only else model
+        torch.save(target_model.state_dict(), path)
+        try:
+            monkeypatch.setattr(weights.value, 'url', str(path))
+        except AttributeError:
+            monkeypatch.setattr(weights, 'url', str(path))
+        return weights
+
+    def test_vit(self) -> None:
+        vit_base_patch14_dinov2()
+
+    def test_vit_weights(
+        self, mocked_weights: WeightsEnum, features_only: bool
+    ) -> None:
+        vit_base_patch14_dinov2(weights=mocked_weights, features_only=features_only)
+
+    def test_bands(self, mocked_weights: WeightsEnum) -> None:
+        if 'bands' in mocked_weights.meta:
+            assert len(mocked_weights.meta['bands']) == mocked_weights.meta['in_chans']
+
+    def test_transforms(self, mocked_weights: WeightsEnum) -> None:
+        c = mocked_weights.meta['in_chans']
+        img_size = mocked_weights.meta['img_size']
+        if isinstance(img_size, int):
+            h = w = img_size
+        else:
+            h, w = img_size
+        sample = {'image': torch.arange(c * h * w, dtype=torch.float).view(c, h, w)}
+        mocked_weights.transforms(sample)
+
+    @pytest.mark.slow
+    def test_vit_download(self, weights: WeightsEnum) -> None:
+        vit_base_patch14_dinov2(weights=weights)
diff --git a/torchgeo/models/__init__.py b/torchgeo/models/__init__.py
index e49ca3a08ac..75dd1715fa9 100644
--- a/torchgeo/models/__init__.py
+++ b/torchgeo/models/__init__.py
@@ -31,13 +31,17 @@
 from .scale_mae import ScaleMAE, ScaleMAELarge16_Weights, scalemae_large_patch16
 from .swin import Swin_V2_B_Weights, Swin_V2_T_Weights, swin_v2_b, swin_v2_t
 from .vit import (
+    ViTBase14_DINOv2_Weights,
     ViTBase16_Weights,
     ViTHuge14_Weights,
     ViTLarge16_Weights,
+    ViTSmall14_DINOv2_Weights,
     ViTSmall16_Weights,
+    vit_base_patch14_dinov2,
     vit_base_patch16_224,
     vit_huge_patch14_224,
     vit_large_patch16_224,
+    vit_small_patch14_dinov2,
     vit_small_patch16_224,
 )
 
@@ -65,9 +69,11 @@
     'ScaleMAELarge16_Weights',
     'Swin_V2_B_Weights',
     'Swin_V2_T_Weights',
+    'ViTBase14_DINOv2_Weights',
     'ViTBase16_Weights',
     'ViTHuge14_Weights',
     'ViTLarge16_Weights',
+    'ViTSmall14_DINOv2_Weights',
     'ViTSmall16_Weights',
     'copernicusfm_base',
     'croma_base',
@@ -86,8 +92,10 @@
     'scalemae_large_patch16',
     'swin_v2_b',
     'swin_v2_t',
+    'vit_base_patch14_dinov2',
     'vit_base_patch16_224',
     'vit_huge_patch14_224',
     'vit_large_patch16_224',
+    'vit_small_patch14_dinov2',
     'vit_small_patch16_224',
 )
diff --git a/torchgeo/models/resnet.py b/torchgeo/models/resnet.py
index 93a4cd92210..fa9f127d87a 100644
--- a/torchgeo/models/resnet.py
+++ b/torchgeo/models/resnet.py
@@ -534,7 +534,8 @@ class ResNet50_Weights(WeightsEnum):  # type: ignore[misc]
         },
     )
 
-    SENTINEL1_ALL_DECUR = Weights(
+    # ALL is deprecated, use GRD instead
+    SENTINEL1_ALL_DECUR = SENTINEL1_GRD_DECUR = Weights(
         url='https://huggingface.co/torchgeo/decur/resolve/9328eeb90c686a88b30f8526ed757b4bc0f12027/rn50_ssl4eo-s12_sar_decur_ep100-f0e69ba2.pth',
         transforms=_ssl4eo_s12_transforms_s1,
         meta={
@@ -548,7 +549,8 @@ class ResNet50_Weights(WeightsEnum):  # type: ignore[misc]
         },
     )
 
-    SENTINEL1_ALL_MOCO = Weights(
+    # ALL is deprecated, use GRD instead
+    SENTINEL1_ALL_MOCO = SENTINEL1_GRD_MOCO = Weights(
         url='https://hf.co/torchgeo/resnet50_sentinel1_all_moco/resolve/e79862c667853c10a709bdd77ea8ffbad0e0f1cf/resnet50_sentinel1_all_moco-906e4356.pth',
         transforms=_ssl4eo_s12_transforms_s1,
         meta={
@@ -562,6 +564,20 @@ class ResNet50_Weights(WeightsEnum):  # type: ignore[misc]
         },
     )
 
+    SENTINEL1_GRD_SOFTCON = Weights(
+        url='https://huggingface.co/wangyi111/softcon/resolve/62ff465b2e7467dbfc70758ec1e9d08ab87fc46b/B2_rn50_softcon.pth',
+        transforms=_ssl4eo_s12_transforms_s1,
+        meta={
+            'dataset': 'SSL4EO-S12',
+            'in_chans': 2,
+            'model': 'resnet50',
+            'publication': 'https://arxiv.org/abs/2405.20462',
+            'repo': 'https://github.com/zhu-xlab/softcon',
+            'ssl_method': 'softcon',
+            'bands': _sentinel1_grd_bands,
+        },
+    )
+
     SENTINEL2_ALL_DECUR = Weights(
         url='https://huggingface.co/torchgeo/decur/resolve/eba7ae5945d482a4319be046d34b552db5dd9950/rn50_ssl4eo-s12_ms_decur_ep100-fc6b09ff.pth',
         transforms=_ssl4eo_s12_transforms_s2_10k,
@@ -604,6 +620,20 @@ class ResNet50_Weights(WeightsEnum):  # type: ignore[misc]
         },
     )
 
+    SENTINEL2_ALL_SOFTCON = Weights(
+        url='https://huggingface.co/wangyi111/softcon/resolve/62ff465b2e7467dbfc70758ec1e9d08ab87fc46b/B13_rn50_softcon.pth',
+        transforms=_ssl4eo_s12_transforms_s2_stats,
+        meta={
+            'dataset': 'SSL4EO-S12',
+            'in_chans': 13,
+            'model': 'resnet50',
+            'publication': 'https://arxiv.org/abs/2405.20462',
+            'repo': 'https://github.com/zhu-xlab/softcon',
+            'ssl_method': 'softcon',
+            'bands': _sentinel2_toa_bands,
+        },
+    )
+
     SENTINEL2_MI_MS_SATLAS = Weights(
         url='https://hf.co/torchgeo/satlas/resolve/081d6607431bf36bdb59c223777cbb267131b8f2/sentinel2_resnet50_mi_ms-da5413d2.pth',
         transforms=_satlas_sentinel2_transforms,
diff --git a/torchgeo/models/vit.py b/torchgeo/models/vit.py
index e94dca1627f..a07b72b2c2d 100644
--- a/torchgeo/models/vit.py
+++ b/torchgeo/models/vit.py
@@ -21,6 +21,7 @@
     _sentinel1_grd_bands,
     _sentinel2_toa_bands,
     _ssl4eo_s12_transforms_s1,
+    _ssl4eo_s12_transforms_s2_stats,
 )
 
 # https://github.com/zhu-xlab/SSL4EO-S12/blob/d2868adfada65e40910bfcedfc49bc3b20df2248/src/benchmark/transfer_classification/linear_BE_moco.py#L167
@@ -478,6 +479,86 @@ class ViTHuge14_Weights(WeightsEnum):  # type: ignore[misc]
     )
 
 
+class ViTSmall14_DINOv2_Weights(WeightsEnum):  # type: ignore[misc]
+    """Vision Transformer Small Patch Size 14 (DINOv2) weights.
+
+    For `timm <https://github.com/huggingface/pytorch-image-models>`_
+    *vit_small_patch14_dinov2* implementation.
+
+    .. versionadded:: 0.7
+    """
+
+    SENTINEL2_ALL_SOFTCON = Weights(
+        url='https://huggingface.co/wangyi111/softcon/resolve/bae909781911f8ec034b4b959992fae17b973c0c/B13_vits14_softcon_enc.pth',
+        transforms=_ssl4eo_s12_transforms_s2_stats,
+        meta={
+            'dataset': 'SSL4EO-S12',
+            'in_chans': 13,
+            'img_size': 224,
+            'model': 'vit_small_patch14_dinov2',
+            'publication': 'https://arxiv.org/abs/2405.20462',
+            'repo': 'https://github.com/zhu-xlab/softcon',
+            'ssl_method': 'softcon',
+            'bands': _sentinel2_toa_bands,
+        },
+    )
+
+    SENTINEL1_GRD_SOFTCON = Weights(
+        url='https://huggingface.co/wangyi111/softcon/resolve/bae909781911f8ec034b4b959992fae17b973c0c/B2_vits14_softcon_enc.pth',
+        transforms=_ssl4eo_s12_transforms_s1,
+        meta={
+            'dataset': 'SSL4EO-S12',
+            'in_chans': 2,
+            'img_size': 224,
+            'model': 'vit_small_patch14_dinov2',
+            'publication': 'https://arxiv.org/abs/2405.20462',
+            'repo': 'https://github.com/zhu-xlab/softcon',
+            'ssl_method': 'softcon',
+            'bands': _sentinel1_grd_bands,
+        },
+    )
+
+
+class ViTBase14_DINOv2_Weights(WeightsEnum):  # type: ignore[misc]
+    """Vision Transformer Base Patch Size 14 (DINOv2) weights.
+
+    For `timm <https://github.com/huggingface/pytorch-image-models>`_
+    *vit_base_patch14_dinov2* implementation.
+
+    .. versionadded:: 0.7
+    """
+
+    SENTINEL2_ALL_SOFTCON = Weights(
+        url='https://huggingface.co/wangyi111/softcon/resolve/bae909781911f8ec034b4b959992fae17b973c0c/B13_vitb14_softcon_enc.pth',
+        transforms=_ssl4eo_s12_transforms_s2_stats,
+        meta={
+            'dataset': 'SSL4EO-S12',
+            'in_chans': 13,
+            'img_size': 224,
+            'model': 'vit_base_patch14_dinov2',
+            'publication': 'https://arxiv.org/abs/2405.20462',
+            'repo': 'https://github.com/zhu-xlab/softcon',
+            'ssl_method': 'softcon',
+            'bands': _sentinel2_toa_bands,
+        },
+    )
+
+    SENTINEL1_GRD_SOFTCON = Weights(
+        url='https://huggingface.co/wangyi111/softcon/resolve/bae909781911f8ec034b4b959992fae17b973c0c/B2_vitb14_softcon_enc.pth',
+        transforms=_ssl4eo_s12_transforms_s1,
+        meta={
+            'dataset': 'SSL4EO-S12',
+            'in_chans': 2,
+            'img_size': 224,
+            'model': 'vit_base_patch14_dinov2',
+            'publication': 'https://arxiv.org/abs/2405.20462',
+            'repo': 'https://github.com/zhu-xlab/softcon',
+            'ssl_method': 'softcon',
+            'bands': _sentinel1_grd_bands,
+        },
+    )
+
+
 def vit_small_patch16_224(
     weights: ViTSmall16_Weights | None = None, *args: Any, **kwargs: Any
 ) -> VisionTransformer | nn.ModuleDict:
@@ -649,3 +730,91 @@ def vit_huge_patch14_224(
         assert set(unexpected_keys) <= {'norm.weight', 'norm.bias'}
 
     return model
+
+
+def vit_small_patch14_dinov2(
+    weights: ViTSmall14_DINOv2_Weights | None = None, *args: Any, **kwargs: Any
+) -> VisionTransformer | nn.ModuleDict:
+    """Vision Transform (ViT) small patch size 14 model for DINOv2.
+
+    If you use this model in your research, please cite the following paper:
+
+    * https://arxiv.org/abs/2304.07193
+
+    .. versionadded:: 0.7
+
+    Args:
+        weights: Pre-trained model weights to use.
+        *args: Additional arguments to pass to :func:`timm.create_model`.
+        **kwargs: Additional keyword arguments to pass to :func:`timm.create_model`.
+
+    Returns:
+        A DINOv2 ViT small 14 model.
+    """
+    if weights:
+        kwargs['in_chans'] = weights.meta['in_chans']
+        kwargs['img_size'] = weights.meta['img_size']
+    # FeatureGetterNet (extends nn.ModuleDict) is returned when features_only=True
+    model: VisionTransformer | nn.ModuleDict = timm.create_model(
+        'vit_small_patch14_dinov2', *args, **kwargs
+    )
+
+    if kwargs.get('features_only', False):
+        model = cast(nn.ModuleDict, model)
+        target_model = cast(VisionTransformer, model.model)
+    else:
+        model = cast(VisionTransformer, model)
+        target_model = model
+
+    if weights:
+        missing_keys, unexpected_keys = target_model.load_state_dict(
+            weights.get_state_dict(progress=True), strict=False
+        )
+        assert set(missing_keys) <= {'head.weight', 'head.bias'}
+        assert set(unexpected_keys) <= {'norm.weight', 'norm.bias'}
+
+    return model
+
+
+def vit_base_patch14_dinov2(
+    weights: ViTBase14_DINOv2_Weights | None = None, *args: Any, **kwargs: Any
+) -> VisionTransformer | nn.ModuleDict:
+    """Vision Transform (ViT) base patch size 14 model for DINOv2.
+
+    If you use this model in your research, please cite the following paper:
+
+    * https://arxiv.org/abs/2304.07193
+
+    .. versionadded:: 0.7
+
+    Args:
+        weights: Pre-trained model weights to use.
+        *args: Additional arguments to pass to :func:`timm.create_model`.
+        **kwargs: Additional keyword arguments to pass to :func:`timm.create_model`.
+
+    Returns:
+        A DINOv2 ViT base 14 model.
+    """
+    if weights:
+        kwargs['in_chans'] = weights.meta['in_chans']
+        kwargs['img_size'] = weights.meta['img_size']
+    # FeatureGetterNet (extends nn.ModuleDict) is returned when features_only=True
+    model: VisionTransformer | nn.ModuleDict = timm.create_model(
+        'vit_base_patch14_dinov2', *args, **kwargs
+    )
+
+    if kwargs.get('features_only', False):
+        model = cast(nn.ModuleDict, model)
+        target_model = cast(VisionTransformer, model.model)
+    else:
+        model = cast(VisionTransformer, model)
+        target_model = model
+
+    if weights:
+        missing_keys, unexpected_keys = target_model.load_state_dict(
+            weights.get_state_dict(progress=True), strict=False
+        )
+        assert set(missing_keys) <= {'head.weight', 'head.bias'}
+        assert set(unexpected_keys) <= {'norm.weight', 'norm.bias'}
+
+    return model


Reviews:
user: login: wangyi111
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T20:58:30Z
user: login: adamjstewart
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T21:11:02Z
user: login: adamjstewart
body: Let's rebase after #2673 is merged and apply the same changes from #2659.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-27T17:27:45Z
user: login: adamjstewart
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-29T13:03:03Z

Review comments:
diff_hunk: @@ -18,7 +18,12 @@
     scalemae_large_patch16,
     swin_v2_b,
     swin_v2_t,
+    vit_base_patch16_224,
+    vit_huge_patch14_224,
+    vit_large_patch16_224,
     vit_small_patch16_224,
+    vit_small_patch14_dinov2,
path: hubconf.py
user: login: wangyi111
body: Fixing ruff error would split apart the position of dinov2 small and base. Should we do that?
created_at: 2025-03-25T20:58:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -18,7 +18,12 @@
     scalemae_large_patch16,
     swin_v2_b,
     swin_v2_t,
+    vit_base_patch16_224,
+    vit_huge_patch14_224,
+    vit_large_patch16_224,
     vit_small_patch16_224,
+    vit_small_patch14_dinov2,
path: hubconf.py
user: login: adamjstewart
body: Yes that's fine, listen to ruff. 
created_at: 2025-03-25T21:11:02Z
author_association: MEMBER

PR comments:
user: login: wangyi111
created_at: 2025-03-25T20:41:27Z
body: This PR also includes a renaming of existing Sentinel1_ALL weights to Sentinel1_GRD weights. Might be more suitable when we later add Sentinel-1 SLC models.
author_association: CONTRIBUTOR
user: login: wangyi111
created_at: 2025-03-28T21:45:31Z
body: `tests/models/test_vit.py` failed not sure for what reason..
author_association: CONTRIBUTOR
user: login: adamjstewart
created_at: 2025-03-29T10:49:43Z
body: Since it isn't a timeout and there is no error msg, I'm guessing OOM error. We may have once again reached the point where there are so many large models we need to instantiate that we have filled up the [minimal RAM on the CI runners](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories). Let me see if I can squeeze out more performance...
author_association: MEMBER