internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Lightning-Universe/lightning-flash

Issue:
TypeError: __init__() got an unexpected keyword argument 'global_pool'
## 🐛 Bug

Flash backbones is hardcoding the global_pool argument but this isn't relevant for Visual Transformer backbones and causes them to crash 

`TypeError: __init__() got an unexpected keyword argument 'global_pool'`

### To Reproduce

`model = ImageClassifier(backbone="vit_small_patch16_224",  num_classes=10,  serializer=Labels())`

### Expected behavior

global_pool should only be passed if expected model backbone should initialize and not crash

### Environment

 - PyTorch Version (e.g., 1.0):
 - OS (e.g., Linux):
 - How you installed PyTorch (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:
 - GPU models and configuration:
 - Any other relevant information:

### Additional context

<!-- Add any other context about the problem here. -->



Discussion:

Resolving patch:
diff --git a/flash/image/backbones.py b/flash/image/backbones.py
--- a/flash/image/backbones.py
+++ b/flash/image/backbones.py
@@ -161,11 +161,9 @@ def _fn_timm(
         model_name: str,
         pretrained: bool = True,
         num_classes: int = 0,
-        global_pool: str = '',
+        **kwargs,
     ) -> Tuple[nn.Module, int]:
-        backbone = timm.create_model(
-            model_name, pretrained=pretrained, num_classes=num_classes, global_pool=global_pool
-        )
+        backbone = timm.create_model(model_name, pretrained=pretrained, num_classes=num_classes, **kwargs)
         num_features = backbone.num_features
         return backbone, num_features
 


Test patch:
Repository: Lightning-Universe/lightning-flash

Issue:
Flash `Trainer.from_argparse_args` not working
## 🐛 Bug

Looks like from_argparse_args isn't working for flash trainer. Likely related to #343

cc @aribornstein 


Discussion:

Resolving patch:
diff --git a/flash/core/trainer.py b/flash/core/trainer.py
--- a/flash/core/trainer.py
+++ b/flash/core/trainer.py
@@ -11,8 +11,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import inspect
 import warnings
-from argparse import ArgumentParser
+from argparse import ArgumentParser, Namespace
 from functools import wraps
 from typing import Callable, List, Optional, Union
 
@@ -29,6 +30,22 @@
 from flash.core.finetuning import _DEFAULTS_FINETUNE_STRATEGIES, instantiate_default_finetuning_callbacks
 
 
+def from_argparse_args(cls, args: Union[Namespace, ArgumentParser], **kwargs):
+    """Modified version of ``pytorch_lightning.utilities.argparse.from_argparse_args`` which populates ``valid_kwargs``
+    from ``pytorch_lightning.Trainer``."""
+    if isinstance(args, ArgumentParser):
+        args = cls.parse_argparser(args)
+
+    params = vars(args)
+
+    # we only want to pass in valid PLTrainer args, the rest may be user specific
+    valid_kwargs = inspect.signature(PlTrainer.__init__).parameters
+    trainer_kwargs = dict((name, params[name]) for name in valid_kwargs if name in params)
+    trainer_kwargs.update(**kwargs)
+
+    return cls(**trainer_kwargs)
+
+
 def _defaults_from_env_vars(fn: Callable) -> Callable:
     """Copy of ``pytorch_lightning.trainer.connectors.env_vars_connector._defaults_from_env_vars``. Required to fix
     build error in readthedocs."""
@@ -180,3 +197,9 @@ def add_argparse_args(cls, *args, **kwargs) -> ArgumentParser:
         # the lightning trainer implementation does not support subclasses.
         # context: https://github.com/PyTorchLightning/lightning-flash/issues/342#issuecomment-848892447
         return add_argparse_args(PlTrainer, *args, **kwargs)
+
+    @classmethod
+    def from_argparse_args(cls, args: Union[Namespace, ArgumentParser], **kwargs) -> 'Trainer':
+        # the lightning trainer implementation does not support subclasses.
+        # context: https://github.com/PyTorchLightning/lightning-flash/issues/342#issuecomment-848892447
+        return from_argparse_args(PlTrainer, args, **kwargs)


Test patch:
diff --git a/tests/core/test_trainer.py b/tests/core/test_trainer.py
--- a/tests/core/test_trainer.py
+++ b/tests/core/test_trainer.py
@@ -118,3 +118,11 @@ def test_add_argparse_args():
     parser = Trainer.add_argparse_args(parser)
     args = parser.parse_args(['--gpus=1'])
     assert args.gpus == 1
+
+
+def test_from_argparse_args():
+    parser = ArgumentParser()
+    parser = Trainer.add_argparse_args(parser)
+    args = parser.parse_args(['--max_epochs=200'])
+    trainer = Trainer.from_argparse_args(args)
+    assert trainer.max_epochs == 200
Repository: Lightning-Universe/lightning-flash

Issue:
`METADATA` can't be a dict
## 🐛 Bug

If a dict is used for the `METADATA` key then it will give an error. This should be supported.



Discussion:

Resolving patch:
diff --git a/flash/core/data/batch.py b/flash/core/data/batch.py
--- a/flash/core/data/batch.py
+++ b/flash/core/data/batch.py
@@ -145,8 +145,8 @@ def __init__(
         self._collate_context = CurrentFuncContext("collate", preprocess)
         self._per_batch_transform_context = CurrentFuncContext(f"per_batch_transform{extension}", preprocess)
 
+    @staticmethod
     def _extract_metadata(
-        self,
         samples: List[Dict[str, Any]],
     ) -> Tuple[List[Dict[str, Any]], Optional[List[Dict[str, Any]]]]:
         metadata = [s.pop(DefaultDataKeys.METADATA, None) if isinstance(s, Mapping) else None for s in samples]
@@ -229,8 +229,18 @@ def __init__(
         self.save_fn = convert_to_modules(save_fn)
         self.save_per_sample = convert_to_modules(save_per_sample)
 
+    @staticmethod
+    def _extract_metadata(batch: Any) -> Tuple[Any, Optional[Any]]:
+        if isinstance(batch, Mapping):
+            return batch, batch.get(DefaultDataKeys.METADATA, None)
+        return batch, None
+
     def forward(self, batch: Sequence[Any]):
+        batch, metadata = self._extract_metadata(batch)
         uncollated = self.uncollate_fn(self.per_batch_transform(batch))
+        if metadata:
+            for sample, sample_metadata in zip(uncollated, metadata):
+                sample[DefaultDataKeys.METADATA] = sample_metadata
 
         final_preds = type(uncollated)([self.serializer(self.per_sample_transform(sample)) for sample in uncollated])
 


Test patch:
Repository: Lightning-Universe/lightning-flash

Issue:
Serve sanity check never called with PL master
## 🐛 Bug

`LightningModule.run_sanity_check` has been renamed to `_run_sanity_check`


Discussion:

Resolving patch:
diff --git a/flash/core/trainer.py b/flash/core/trainer.py
--- a/flash/core/trainer.py
+++ b/flash/core/trainer.py
@@ -87,8 +87,15 @@ def __init__(self, *args, serve_sanity_check: bool = False, **kwargs):
 
         self.serve_sanity_check = serve_sanity_check
 
+    def _run_sanity_check(self, ref_model):
+        if hasattr(super(), "_run_sanity_check"):
+            super()._run_sanity_check(ref_model)
+
+        self.run_sanity_check(ref_model)
+
     def run_sanity_check(self, ref_model):
-        super().run_sanity_check(ref_model)
+        if hasattr(super(), "run_sanity_check"):
+            super().run_sanity_check(ref_model)
 
         if self.serve_sanity_check and ref_model.is_servable and _SERVE_AVAILABLE:
             ref_model.run_serve_sanity_check()


Test patch:
Repository: Lightning-Universe/lightning-flash

Issue:
[Typo error] Double declaration of a dictionary
Hello, in this method
https://github.com/PyTorchLightning/lightning-flash/blob/3a9bc6b5da729f3514eddde235716c46104a5c90/flash/core/model.py#L326
the variable
```python
logs = {}
```
has been declared twice. Can I delete one of them?
Thanks.


Discussion:
Hi @PeppeSaccardi thanks for the issue!

> has been declared twice. Can I delete one of them?

Absolutely! PRs welcome :smiley:


Resolving patch:
diff --git a/flash/core/model.py b/flash/core/model.py
--- a/flash/core/model.py
+++ b/flash/core/model.py
@@ -334,7 +334,7 @@ def step(self, batch: Any, batch_idx: int, metrics: nn.ModuleDict) -> Any:
         output = {"y_hat": y_hat}
         y_hat = self.to_loss_format(output["y_hat"])
         losses = {name: l_fn(y_hat, y) for name, l_fn in self.loss_fn.items()}
-        logs = {}
+
         y_hat = self.to_metrics_format(output["y_hat"])
 
         logs = {}


Test patch:
Repository: Lightning-Universe/lightning-flash

Issue:
Error changing the number of input channels of a pretrained ResNet
State dict error when using in_chans with resnets. Need to resolve the weights for the first layer. See: https://fastai.github.io/timmdocs/models#How-is-timm-able-to-use-pretrained-weights-and-handle-images-that-are-not-3-channel-RGB-images?

### Discussed in https://github.com/PyTorchLightning/lightning-flash/discussions/740

<div type='discussions-op-text'>

<sup>Originally posted by **aykutcayir34** September  7, 2021</sup>
I want to use a single channel resnet-18 model in flashlightning image classifier. Is there a convenient way to change the number of channels of a pre-trained model in flashlightning?</div>


Discussion:

Resolving patch:
diff --git a/flash/image/classification/backbones/resnet.py b/flash/image/classification/backbones/resnet.py
--- a/flash/image/classification/backbones/resnet.py
+++ b/flash/image/classification/backbones/resnet.py
@@ -25,8 +25,12 @@
 from torch.hub import load_state_dict_from_url
 
 from flash.core.registry import FlashRegistry
+from flash.core.utilities.imports import _TIMM_AVAILABLE
 from flash.core.utilities.url_error import catch_url_error
 
+if _TIMM_AVAILABLE:
+    from timm.models.helpers import adapt_input_conv
+
 
 def conv3x3(in_planes: int, out_planes: int, stride: int = 1, groups: int = 1, dilation: int = 1) -> nn.Conv2d:
     """3x3 convolution with padding."""
@@ -351,6 +355,8 @@ def _resnet(
             )
 
     if model_weights is not None:
+        in_chans = backbone.conv1.weight.shape[1]
+        model_weights["conv1.weight"] = adapt_input_conv(in_chans, model_weights["conv1.weight"])
         backbone.load_state_dict(model_weights)
 
     return backbone, num_features


Test patch:
Repository: Lightning-Universe/lightning-flash

Issue:
The test step should not use the same metrics objects as validation
## 🐛 Bug

https://github.com/PyTorchLightning/lightning-flash/blob/eb1fcd3925ee297d08704d48c5fc57624665ad96/flash/core/model.py#L424


Discussion:
We do a deepcopy of the metrics object here: https://github.com/PyTorchLightning/lightning-flash/blob/eb1fcd3925ee297d08704d48c5fc57624665ad96/flash/core/model.py#L335 We could add something to do the same for test too. Simple test for correctness here that should be expanded to include val and test: https://github.com/PyTorchLightning/lightning-flash/blob/eb1fcd3925ee297d08704d48c5fc57624665ad96/tests/core/test_model.py#L345

Resolving patch:
diff --git a/flash/core/model.py b/flash/core/model.py
--- a/flash/core/model.py
+++ b/flash/core/model.py
@@ -354,6 +354,7 @@ def __init__(
 
         self.train_metrics = nn.ModuleDict({} if metrics is None else get_callable_dict(metrics))
         self.val_metrics = nn.ModuleDict({} if metrics is None else get_callable_dict(deepcopy(metrics)))
+        self.test_metrics = nn.ModuleDict({} if metrics is None else get_callable_dict(deepcopy(metrics)))
         self.learning_rate = learning_rate
         # TODO: should we save more? Bug on some regarding yaml if we save metrics
         self.save_hyperparameters("learning_rate", "optimizer")
@@ -454,7 +455,7 @@ def validation_step(self, batch: Any, batch_idx: int) -> None:
         )
 
     def test_step(self, batch: Any, batch_idx: int) -> None:
-        output = self.step(batch, batch_idx, self.val_metrics)
+        output = self.step(batch, batch_idx, self.test_metrics)
         self.log_dict(
             {f"test_{k}": v for k, v in output[OutputKeys.LOGS].items()},
             on_step=False,


Test patch:
diff --git a/tests/core/test_model.py b/tests/core/test_model.py
--- a/tests/core/test_model.py
+++ b/tests/core/test_model.py
@@ -437,6 +437,7 @@ def i_will_create_a_misconfiguration_exception(optimizer):
 def test_classification_task_metrics():
     train_dataset = FixedDataset([0, 1])
     val_dataset = FixedDataset([1, 1])
+    test_dataset = FixedDataset([0, 0])
 
     model = OnesModel()
 
@@ -444,6 +445,13 @@ class CheckAccuracy(Callback):
         def on_train_end(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
             assert math.isclose(trainer.callback_metrics["train_accuracy_epoch"], 0.5)
 
+        def on_validation_end(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
+            assert math.isclose(trainer.callback_metrics["val_accuracy"], 1.0)
+
+        def on_test_end(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
+            assert math.isclose(trainer.callback_metrics["test_accuracy"], 0.0)
+
     task = ClassificationTask(model)
     trainer = flash.Trainer(max_epochs=1, callbacks=CheckAccuracy(), gpus=torch.cuda.device_count())
     trainer.fit(task, train_dataloader=DataLoader(train_dataset), val_dataloaders=DataLoader(val_dataset))
+    trainer.test(task, dataloaders=DataLoader(test_dataset))
Repository: Lightning-Universe/lightning-flash

Issue:
Is there a way to seed experiments?
## ❓ Questions and Help

#### What is your question?
Is there a way to seed experiments? Attempts at using `seed_everything` from pytorch lightning do not appear to work (also with the the workers argument set to True).

#### What's your environment?

 - OS: Linux
 - Packaging: pip
 - Version: 0.5.2



Discussion:
Hey @bartonp2 Thanks for the question :smiley: As far as we are aware `seed_everything` should be working in flash (and I think some of our tests depend on it) so if it isn't then I would consider that a bug. Could you share a reproducible script that illustrates the problem?
You are right, the seeding works.

The problem turned out to be in the `SemanticSegmentationFolderInput` class. Specifically, the `load_data()` function lists files in an arbitrary order and the intersection() function scrambles the order again. The problem is solved by adding:
```           
files.sort()
mask_files.sort()
```
to the end of the `load_data()` function.

This might we be worth changing in order to make results reproducible.
Ah, @bartonp2 thanks for your work debugging this. Would you be interested in opening a PR to fix this? All of our file handling is currently in here:
https://github.com/PyTorchLightning/lightning-flash/blob/master/flash/core/data/utilities/paths.py

Let me know if you're interested and I can assign you to the issue. Also happy to help with it if you'd like :smiley:
I could although I don't have time the next couple of weeks. But I could do it after.
That would be great :smiley: I don't think there will be much progress on Flash over the next few weeks anyway

Resolving patch:
diff --git a/flash/image/segmentation/input.py b/flash/image/segmentation/input.py
--- a/flash/image/segmentation/input.py
+++ b/flash/image/segmentation/input.py
@@ -124,6 +124,7 @@ def load_data(
     ) -> List[Dict[str, Any]]:
         self.load_labels_map(num_classes, labels_map)
         files = os.listdir(folder)
+        files.sort()
         if mask_folder is not None:
             mask_files = os.listdir(mask_folder)
 
@@ -137,6 +138,8 @@ def load_data(
 
             files = [os.path.join(folder, file) for file in all_files]
             mask_files = [os.path.join(mask_folder, file) for file in all_files]
+            files.sort()
+            mask_files.sort()
             return super().load_data(files, mask_files)
         return super().load_data(files)
 


Test patch:
Repository: Lightning-Universe/lightning-flash

Issue:
TabularForecastingData must include predict_data_frame
## 📚 Documentation

The [NBEATS tutorial](https://lightning-flash.readthedocs.io/en/stable/notebooks/flash_tutorials/electricity_forecasting.html) uses `0.5.x`, which references `model.predict()` at the end to generate predictions. In `0.6.0`, `model.predict()` was deprecated in favor of `trainer.predict(model, datamodule=datamodule)`.

If you try to do this by passing the `datamodule` created via:

```python
datamodule = TabularForecastingData.from_data_frame(
    time_idx="time_idx",
    target="price actual",
    group_ids=["constant"],
    max_encoder_length=max_encoder_length,
    max_prediction_length=max_prediction_length,
    time_varying_unknown_reals=["price actual"],
    train_data_frame=df_energy_daily[df_energy_daily["time_idx"] <= training_cutoff],
    val_data_frame=df_energy_daily,
    batch_size=256,
)
```

to `trainer.predict(...)`, you'll get the following error:

```python
MisconfigurationException: No `predict_dataloader()` method defined to run `Trainer.predict`.
```

The solution is to be found [here](https://lightning-flash.readthedocs.io/en/stable/reference/tabular_forecasting.html), which clearly shows how to make the prediction datamodule:

```python
# 4. Generate predictions
datamodule = TabularForecastingData.from_data_frame(predict_data_frame=data, parameters=datamodule.parameters)
predictions = trainer.predict(model, datamodule=datamodule)
```

__Suggestion:__
* Update the tutorial to use `0.6.0`
* Add a small section to the [API docs](https://lightning-flash.readthedocs.io/en/stable/api/generated/flash.tabular.forecasting.data.TabularForecastingData.html#flash.tabular.forecasting.data.TabularForecastingData.from_data_frame) explaining that `predict_data_frame` must be specified in order to make use of `trainer.predict(...)` 


Discussion:

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -28,7 +28,7 @@
 _SHOULD_COPY_NOTEBOOKS = True
 
 try:
-    from helpers import HelperCLI
+    from assistant import AssistantCLI
 except Exception:
     _SHOULD_COPY_NOTEBOOKS = False
     warnings.warn("To build the code, please run: `git submodule update --init --recursive`", stacklevel=2)
@@ -62,7 +62,7 @@ def _load_py_module(fname, pkg="flash"):
 
 # -- Project documents -------------------------------------------------------
 if _SHOULD_COPY_NOTEBOOKS:
-    HelperCLI.copy_notebooks(_PATH_RAW_NB, _PATH_HERE, "notebooks", patterns=["flash_tutorials"])
+    AssistantCLI.copy_notebooks(_PATH_RAW_NB, _PATH_HERE, "notebooks", patterns=["flash_tutorials"])
 
 
 def _transform_changelog(path_in: str, path_out: str) -> None:


Test patch:
Repository: Lightning-Universe/lightning-flash

Issue:
Add docs extension to mark pages as beta
## 📚 Documentation

Something like this:

```
.. beta::
```

To then give a warning pop-up that says this page / section / task / etc. is in beta



Discussion:

Resolving patch:
diff --git a/docs/extensions/__init__.py b/docs/extensions/__init__.py
new file mode 100644
diff --git a/docs/extensions/stability.py b/docs/extensions/stability.py
new file mode 100644
--- /dev/null
+++ b/docs/extensions/stability.py
@@ -0,0 +1,46 @@
+# Copyright The PyTorch Lightning team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from docutils import nodes
+from docutils.parsers.rst import Directive
+from docutils.statemachine import StringList
+
+ADMONITION_TEMPLATE = """
+.. raw:: html
+
+    <div class="admonition warning {type}">
+    <p class="admonition-title">{title}</p>
+    <p>This {scope} is currently in Beta. The interfaces and functionality may change without warning in future
+    releases.</p>
+    </div>
+"""
+
+
+class Beta(Directive):
+    has_content = True
+    required_arguments = 1
+    optional_arguments = 0
+
+    def run(self):
+
+        scope = self.arguments[0]
+
+        admonition_rst = ADMONITION_TEMPLATE.format(type="beta", title="Beta", scope=scope)
+        admonition_list = StringList(admonition_rst.split("\n"))
+        admonition = nodes.paragraph()
+        self.state.nested_parse(admonition_list, self.content_offset, admonition)
+        return [admonition]
+
+
+def setup(app):
+    app.add_directive("beta", Beta)
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -23,6 +23,7 @@
 _PATH_ROOT = os.path.join(_PATH_HERE, "..", "..")
 _PATH_RAW_NB = os.path.join(_PATH_ROOT, "_notebooks")
 sys.path.insert(0, os.path.abspath(_PATH_ROOT))
+sys.path.insert(0, os.path.abspath(os.path.join(_PATH_HERE, "..", "extensions")))
 sys.path.append(os.path.join(_PATH_RAW_NB, ".actions"))
 
 _SHOULD_COPY_NOTEBOOKS = True
@@ -122,6 +123,7 @@ def _transform_changelog(path_in: str, path_out: str) -> None:
     "sphinx_paramlinks",
     "sphinx_togglebutton",
     "pt_lightning_sphinx_theme.extensions.lightning_tutorials",
+    "stability",
 ]
 
 # autodoc: Default to members and undoc-members


Test patch:
Repository: Lightning-Universe/lightning-flash

Issue:
Docs page describing Beta meaning
## 📚 Documentation

Add a page in our docs describing that beta means that one or all of the following are true:
- the feature has unstable dependencies
- the feature may change without notice in future versions
- the feature is not compatible with other flash / pl features
- the performance of the feature has not been verified

Anything else?



Discussion:

Resolving patch:
diff --git a/docs/extensions/stability.py b/docs/extensions/stability.py
--- a/docs/extensions/stability.py
+++ b/docs/extensions/stability.py
@@ -20,8 +20,14 @@
 
     <div class="admonition warning {type}">
     <p class="admonition-title">{title}</p>
-    <p>This {scope} is currently in Beta. The interfaces and functionality may change without warning in future
-    releases.</p>
+    <p>
+
+This {scope} is currently in Beta. The API and functionality may change without warning in future
+releases. :ref:`More details <stability>`.
+
+.. raw:: html
+
+    </p>
     </div>
 """
 


Test patch: