internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: nf-core/tools

Issue:
[BUG] Regex behaviour and comment in code do not agree
### Description of the bug

See

https://github.com/nf-core/tools/blob/171127bd850040e76febd0945e6980b7afcaad69/nf_core/modules/lint/main_nf.py#L242-L244

The regex as it is in the code extracts `v1.2.0_cv1.img` instead of `v1.2.0_cv1` from `https://containers.biocontainers.pro/s3/SingImgsRepo/biocontainers/v1.2.0_cv1/biocontainers_v1.2.0_cv1.img' :`

See the [original](https://regex101.com/r/Zz1I05/1) and [fixed](https://regex101.com/r/Zz1I05/2) regex.

### Command used and terminal output

_No response_

### System information

_No response_


Discussion:

Resolving patch:
diff --git a/nf_core/modules/lint/main_nf.py b/nf_core/modules/lint/main_nf.py
--- a/nf_core/modules/lint/main_nf.py
+++ b/nf_core/modules/lint/main_nf.py
@@ -241,7 +241,7 @@ def check_process_section(self, lines):
         if l.startswith("https://containers") or l.startswith("https://depot"):
             # e.g. "https://containers.biocontainers.pro/s3/SingImgsRepo/biocontainers/v1.2.0_cv1/biocontainers_v1.2.0_cv1.img' :" -> v1.2.0_cv1
             # e.g. "https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0' :" -> 0.11.9--0
-            singularity_tag = re.search(r"(?:/)?(?:biocontainers_)?(?::)?([A-Za-z\d\-_.]+)(?:\.img)?['\"]", l).group(1)
+            singularity_tag = re.search(r"(?:/)?(?:biocontainers_)?(?::)?([A-Za-z\d\-_.]+?)(?:\.img)?['\"]", l).group(1)
         if l.startswith("biocontainers/") or l.startswith("quay.io/"):
             # e.g. "quay.io/biocontainers/krona:2.7.1--pl526_5' }" -> 2.7.1--pl526_5
             # e.g. "biocontainers/biocontainers:v1.2.0_cv1' }" -> v1.2.0_cv1


Test patch:
Repository: nf-core/tools

Issue:
modules info broken in dev 
### Description of the bug

Running `nf-core modules info` on `nf-core/tools version 2.5.dev0` produces a `TypeError`

### Command used and terminal output

```console
nf-core modules info

TypeError: __init__() takes 6 positional arguments but 7 were given

/Users/arthurgymer/.pyenv/versions/3.9.7/envs/nf-core/lib/python3.9/site-packages/nf_core/__main__.py:711 in info                                                                                │
│                                                                                                                                                                                                  │
│    710 │   try:                                                                                                                                                                                  │
│ ❱  711 │   │   module_info = nf_core.modules.ModuleInfo(                                                                                                                                         │
│    712 │   │   │   dir,
```


### System information

nf-core/tools version 2.5.dev0
python 3.9.7
macOS



Discussion:

Resolving patch:
diff --git a/nf_core/modules/info.py b/nf_core/modules/info.py
--- a/nf_core/modules/info.py
+++ b/nf_core/modules/info.py
@@ -20,8 +20,8 @@
 
 
 class ModuleInfo(ModuleCommand):
-    def __init__(self, pipeline_dir, tool, remote_url, branch, no_pull):
-        super().__init__(pipeline_dir, remote_url, branch, no_pull)
+    def __init__(self, pipeline_dir, tool, remote_url, branch, no_pull, base_path):
+        super().__init__(pipeline_dir, remote_url, branch, no_pull, base_path)
         self.meta = None
         self.local_path = None
         self.remote_location = None


Test patch:
Repository: nf-core/tools

Issue:
Use of future functionality
### Description of the bug

The `shutil.copytree` argument [`dirs_exist_ok`](https://docs.python.org/3/library/shutil.html#shutil.copytree) is only available in Python version 3.8+ but nf-core tool still support Python 3.7

https://github.com/nf-core/tools/blob/b5400d84d307343353b6ab09aad204231f74fb0e/nf_core/modules/lint/module_changes.py#L28

Two options: drop Python 3.7 support or change that code.



Discussion:
I guess there's no tests for this functionality because the tests don't fail with Python 3.7.

Resolving patch:
diff --git a/nf_core/modules/lint/module_changes.py b/nf_core/modules/lint/module_changes.py
--- a/nf_core/modules/lint/module_changes.py
+++ b/nf_core/modules/lint/module_changes.py
@@ -25,7 +25,7 @@ def module_changes(module_lint_object, module):
         # If the module is patched, we need to apply
         # the patch in reverse before comparing with the remote
         tempdir = Path(tempfile.mkdtemp())
-        shutil.copytree(module.module_dir, tempdir, dirs_exist_ok=True)
+        shutil.copytree(module.module_dir, tempdir)
         try:
             new_lines = ModulesDiffer.try_apply_patch(
                 module.module_name, module_lint_object.modules_repo.fullname, module.patch_path, tempdir, reverse=True


Test patch:
Repository: nf-core/tools

Issue:
Creating a pipeline from a template allows for dashes
### Description of the bug

When calling `nf-core create` from a template, the name field may contain, for example, dashes which are not accepted when populating the fields interactively. This will generate a broken pipeline due to Groovy class names with hyphens. So the input from the template should be better validated. Incidentally, the example in the [documentation](https://nf-co.re/tools/#creating-a-new-pipeline) also contains a hyphen which seems a terrible idea.

### Command used and terminal output

```console
nf-core create --template-yaml template.yml
```


### System information

nf-core/tools version 2.5.1


Discussion:

Resolving patch:
diff --git a/nf_core/create.py b/nf_core/create.py
--- a/nf_core/create.py
+++ b/nf_core/create.py
@@ -163,6 +163,11 @@ def create_param_dict(self, name, description, author, version, template_yaml_pa
         param_dict["logo_dark"] = f"{param_dict['name_noslash']}_logo_dark.png"
         param_dict["version"] = version
 
+        # Check that the pipeline name matches the requirements
+        if not re.match(r"^[a-z]+$", param_dict["short_name"]):
+            log.error("[red]Invalid workflow name: must be lowercase without punctuation.")
+            sys.exit(1)
+
         return param_dict, skip_paths
 
     def customize_template(self, template_areas):


Test patch:
diff --git a/tests/test_launch.py b/tests/test_launch.py
--- a/tests/test_launch.py
+++ b/tests/test_launch.py
@@ -73,7 +73,7 @@ def test_make_pipeline_schema(self, tmp_path):
         """Create a workflow, but delete the schema file, then try to load it"""
         test_pipeline_dir = os.path.join(tmp_path, "wf")
         create_obj = nf_core.create.PipelineCreate(
-            "test_pipeline", "", "", outdir=test_pipeline_dir, no_git=True, plain=True
+            "testpipeline", "", "", outdir=test_pipeline_dir, no_git=True, plain=True
         )
         create_obj.init_pipeline()
         os.remove(os.path.join(test_pipeline_dir, "nextflow_schema.json"))
diff --git a/tests/test_schema.py b/tests/test_schema.py
--- a/tests/test_schema.py
+++ b/tests/test_schema.py
@@ -31,7 +31,7 @@ def setUp(self):
         self.tmp_dir = tempfile.mkdtemp()
         self.template_dir = os.path.join(self.tmp_dir, "wf")
         create_obj = nf_core.create.PipelineCreate(
-            "test_pipeline", "", "", outdir=self.template_dir, no_git=True, plain=True
+            "testpipeline", "", "", outdir=self.template_dir, no_git=True, plain=True
         )
         create_obj.init_pipeline()
 
diff --git a/tests/test_sync.py b/tests/test_sync.py
--- a/tests/test_sync.py
+++ b/tests/test_sync.py
@@ -25,7 +25,7 @@ class TestModules(unittest.TestCase):
     def setUp(self):
         """Create a new pipeline to test"""
         self.tmp_dir = tempfile.mkdtemp()
-        self.pipeline_dir = os.path.join(self.tmp_dir, "test_pipeline")
+        self.pipeline_dir = os.path.join(self.tmp_dir, "testpipeline")
         self.create_obj = nf_core.create.PipelineCreate(
             "testing", "test pipeline", "tester", outdir=self.pipeline_dir, plain=True
         )
Repository: nf-core/tools

Issue:
nf-core modules install is asking for two files
### Description of the bug

nf-core modules install is asking for two files: main.nf and nextflow.config. I think this is not necessary, since one can use a different naming convention for both files.

https://github.com/nf-core/modules/issues/2272

I think this can be just a warning since you can decide to install the modules outside a nf-core pipeline


### Command used and terminal output

_No response_

### System information

_No response_


Discussion:

Resolving patch:
diff --git a/nf_core/modules/modules_command.py b/nf_core/modules/modules_command.py
--- a/nf_core/modules/modules_command.py
+++ b/nf_core/modules/modules_command.py
@@ -64,7 +64,9 @@ def has_valid_directory(self):
         main_nf = os.path.join(self.dir, "main.nf")
         nf_config = os.path.join(self.dir, "nextflow.config")
         if not os.path.exists(main_nf) and not os.path.exists(nf_config):
-            raise UserWarning(f"Could not find a 'main.nf' or 'nextflow.config' file in '{self.dir}'")
+            if Path(self.dir).resolve().parts[-1].startswith("nf-core"):
+                raise UserWarning(f"Could not find a 'main.nf' or 'nextflow.config' file in '{self.dir}'")
+            log.warning(f"Could not find a 'main.nf' or 'nextflow.config' file in '{self.dir}'")
         return True
 
     def has_modules_file(self):


Test patch:
diff --git a/tests/modules/install.py b/tests/modules/install.py
--- a/tests/modules/install.py
+++ b/tests/modules/install.py
@@ -22,7 +22,8 @@ def test_modules_install_nopipeline(self):
 @with_temporary_folder
 def test_modules_install_emptypipeline(self, tmpdir):
     """Test installing a module - empty dir given"""
-    self.mods_install.dir = tmpdir
+    os.mkdir(os.path.join(tmpdir, "nf-core-pipe"))
+    self.mods_install.dir = os.path.join(tmpdir, "nf-core-pipe")
     with pytest.raises(UserWarning) as excinfo:
         self.mods_install.install("foo")
     assert "Could not find a 'main.nf' or 'nextflow.config' file" in str(excinfo.value)
Repository: nf-core/tools

Issue:
nf-core launch --params-in error
### Description of the bug

nf-core launch with --params-in /path/to/nf-params.json results in "Parameter output file already exists! nf-params.json". is this an error? my understanding is that it should use this file and not try to create a new one

### Command used and terminal output

nf-core launch slamseq --params-in /Users/mbzmch/Desktop/x.NEXTFLOW/WRK/nf-params.json 

WARNING  Parameter output file already exists! nf-params.json                                                                         launch.py:123
Do you want to overwrite this file? [y/n]: n
INFO     Exiting. Use --params-out to specify a custom filename. 

### System information

_No response_


Discussion:

Resolving patch:
diff --git a/nf_core/launch.py b/nf_core/launch.py
--- a/nf_core/launch.py
+++ b/nf_core/launch.py
@@ -120,12 +120,19 @@ def launch_pipeline(self):
 
         # Check if the output file exists already
         if os.path.exists(self.params_out):
-            log.warning(f"Parameter output file already exists! {os.path.relpath(self.params_out)}")
+            # if params_in has the same name as params_out, don't ask to overwrite
+            if self.params_in and os.path.abspath(self.params_in) == os.path.abspath(self.params_out):
+                log.warning(
+                    f"The parameter input file has the same name as the output file! {os.path.relpath(self.params_out)} will be overwritten."
+                )
+            else:
+                log.warning(f"Parameter output file already exists! {os.path.relpath(self.params_out)}")
             if Confirm.ask("[yellow]Do you want to overwrite this file?"):
-                os.remove(self.params_out)
-                log.info(f"Deleted {self.params_out}\n")
+                if not (self.params_in and os.path.abspath(self.params_in) == os.path.abspath(self.params_out)):
+                    os.remove(self.params_out)
+                    log.info(f"Deleted {self.params_out}\n")
             else:
-                log.info("Exiting. Use --params-out to specify a custom filename.")
+                log.info("Exiting. Use --params-out to specify a custom output filename.")
                 return False
 
         log.info(
@@ -716,6 +723,6 @@ def launch_workflow(self):
         """Launch nextflow if required"""
         log.info(f"[bold underline]Nextflow command:[/]\n[magenta]{self.nextflow_cmd}\n\n")
 
-        if Confirm.ask("Do you want to run this command now? "):
+        if Confirm.ask("Do you want to run this command now? ", default=True):
             log.info("Launching workflow! :rocket:")
             subprocess.call(self.nextflow_cmd, shell=True)


Test patch:
Repository: nf-core/tools

Issue:
Bump minimum required Nextflow version
### Description of feature

Latest stable release brings lots of new features that we probably want to use at module level (eg. `bin` directories).


Discussion:

Resolving patch:
diff --git a/nf_core/lint/readme.py b/nf_core/lint/readme.py
--- a/nf_core/lint/readme.py
+++ b/nf_core/lint/readme.py
@@ -40,7 +40,7 @@ def readme(self):
 
     if "nextflow_badge" not in ignore_configs:
         # Check that there is a readme badge showing the minimum required version of Nextflow
-        # [![Nextflow](https://img.shields.io/badge/nextflow%20DSL2-%E2%89%A521.10.3-23aa62.svg)](https://www.nextflow.io/)
+        # [![Nextflow](https://img.shields.io/badge/nextflow%20DSL2-%E2%89%A522.10.1-23aa62.svg)](https://www.nextflow.io/)
         # and that it has the correct version
         nf_badge_re = r"\[!\[Nextflow\]\(https://img\.shields\.io/badge/nextflow%20DSL2-!?(?:%E2%89%A5|%3E%3D)([\d\.]+)-23aa62\.svg\)\]\(https://www\.nextflow\.io/\)"
         match = re.search(nf_badge_re, content)
@@ -63,7 +63,7 @@ def readme(self):
             warned.append("README did not have a Nextflow minimum version badge.")
 
     # Check that the minimum version mentioned in the quick start section is consistent
-    # Looking for: "1. Install [`Nextflow`](https://www.nextflow.io/docs/latest/getstarted.html#installation) (`>=21.10.3`)"
+    # Looking for: "1. Install [`Nextflow`](https://www.nextflow.io/docs/latest/getstarted.html#installation) (`>=22.10.1`)"
     nf_version_re = r"1\.\s*Install\s*\[`Nextflow`\]\(https://www.nextflow.io/docs/latest/getstarted.html#installation\)\s*\(`>=(\d*\.\d*\.\d*)`\)"
     match = re.search(nf_version_re, content)
     if match:


Test patch:
Repository: nf-core/tools

Issue:
nf-core subworkflows info list subworkflows in a strange order
### Description of the bug

Order of remote subworkflows with the command info is out of order

![Screenshot from 2022-11-22 10-52-14](https://user-images.githubusercontent.com/1019628/203283886-f54c78a1-03db-4d97-913d-b04e4b7455a8.png)

### Command used and terminal output

_No response_

### System information

_No response_


Discussion:

Resolving patch:
diff --git a/nf_core/components/info.py b/nf_core/components/info.py
--- a/nf_core/components/info.py
+++ b/nf_core/components/info.py
@@ -117,6 +117,7 @@ def init_mod_name(self, component):
                         )
             else:
                 components = self.modules_repo.get_avail_components(self.component_type)
+            components.sort()
             component = questionary.autocomplete(
                 f"Please select a {self.component_type[:-1]}",
                 choices=components,
diff --git a/nf_core/components/install.py b/nf_core/components/install.py
--- a/nf_core/components/install.py
+++ b/nf_core/components/install.py
@@ -186,7 +186,7 @@ def collect_and_verify_name(self, component, modules_repo):
         if component is None:
             component = questionary.autocomplete(
                 f"{'Tool' if self.component_type == 'modules' else 'Subworkflow'} name:",
-                choices=modules_repo.get_avail_components(self.component_type),
+                choices=modules_repo.get_avail_components(self.component_type).sort(),
                 style=nf_core.utils.nfcore_question_style,
             ).unsafe_ask()
 
diff --git a/nf_core/components/update.py b/nf_core/components/update.py
--- a/nf_core/components/update.py
+++ b/nf_core/components/update.py
@@ -353,7 +353,7 @@ def get_single_component_info(self, component):
         if component is None:
             component = questionary.autocomplete(
                 f"{self.component_type[:-1].title()} name:",
-                choices=choices,
+                choices=choices.sort(),
                 style=nf_core.utils.nfcore_question_style,
             ).unsafe_ask()
 


Test patch:
Repository: nf-core/tools

Issue:
--git-remote for custom module repository requires https:// for install (but not other commands)
### Description of the bug

I noticed some command under `nf-core modules ...` work with a `--git-remote git@...` (e.g. list) but others do not (e.g. install) and fail in a unique way where it keeps rewriting the modules.json.  

It looks like https://github.com/nf-core/tools/blob/master/nf_core/modules/modules_json.py#L378 is what requires http here. 

Would be great if it was consistent so git@ worked everywhere, or nowhere (and was documented more clearly).

### Command used and terminal output

_No response_

### System information

nf-core 2.6


Discussion:
Just hit the same snag. We use some modules in a private repository so we need to use SSH access.
Further investigation showed that this code path is unlikely to work properly.

This looks for a key that does not exist (`modules`) in `repos` when using `install` (when `repos` is `None`):

https://github.com/nf-core/tools/blob/dec66abe1c36a8975a952e1f80f045cab65bbf72/nf_core/modules/modules_json.py#L159

This will fail  because the nf-core repositories won't have the required keys when `repos` is `None` (there won't be a key corresponding to `modules_dir` as the dictionary is initialized fresh). The same line raises `NameError` because `url` is used before being defined in the list comprehension:

https://github.com/nf-core/tools/blob/dec66abe1c36a8975a952e1f80f045cab65bbf72/nf_core/modules/modules_json.py#L161
Also hit us unfortunately already as our internal private repos are all SSH git access via keys only :-( 

Can we add this to 2.7 release? @mashehu ? Cannot even test it entirely at the moment due to this here with our internal setup :-( 

Resolving patch:
diff --git a/nf_core/modules/modules_json.py b/nf_core/modules/modules_json.py
--- a/nf_core/modules/modules_json.py
+++ b/nf_core/modules/modules_json.py
@@ -478,7 +478,12 @@ def has_git_url_and_modules(self):
             elif (
                 not isinstance(repo_url, str)
                 or repo_url == ""
-                or not repo_url.startswith("http")
+                or not (
+                    repo_url.startswith("http")
+                    or repo_url.startswith("ftp")
+                    or repo_url.startswith("ssh")
+                    or repo_url.startswith("git")
+                )
                 or not isinstance(repo_entry["modules"], dict)
                 or repo_entry["modules"] == {}
             ):


Test patch:
Repository: nf-core/tools

Issue:
Module lint: Check container syntax
### Description of feature

x-ref https://github.com/nf-core/tools/issues/1627

Check that the `container` syntax is correct and can be parsed properly. eg. single quotes vs double quotes.


Discussion:

Resolving patch:
diff --git a/nf_core/modules/lint/main_nf.py b/nf_core/modules/lint/main_nf.py
--- a/nf_core/modules/lint/main_nf.py
+++ b/nf_core/modules/lint/main_nf.py
@@ -254,14 +254,14 @@ def check_process_section(self, lines, fix_version, progress_bar):
                 self.passed.append(("process_standard_label", "Correct process label", self.main_nf))
     else:
         self.warned.append(("process_standard_label", "Process label unspecified", self.main_nf))
-    for l in lines:
+    for i, l in enumerate(lines):
         if _container_type(l) == "bioconda":
             bioconda_packages = [b for b in l.split() if "bioconda::" in b]
         l = l.strip(" '\"")
         if _container_type(l) == "singularity":
             # e.g. "https://containers.biocontainers.pro/s3/SingImgsRepo/biocontainers/v1.2.0_cv1/biocontainers_v1.2.0_cv1.img' :" -> v1.2.0_cv1
             # e.g. "https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0' :" -> 0.11.9--0
-            match = re.search(r"(?:/)?(?:biocontainers_)?(?::)?([A-Za-z\d\-_.]+?)(?:\.img)?['\"]", l)
+            match = re.search(r"(?:/)?(?:biocontainers_)?(?::)?([A-Za-z\d\-_.]+?)(?:\.img)?'", l)
             if match is not None:
                 singularity_tag = match.group(1)
                 self.passed.append(("singularity_tag", f"Found singularity tag: {singularity_tag}", self.main_nf))
@@ -271,13 +271,19 @@ def check_process_section(self, lines, fix_version, progress_bar):
         if _container_type(l) == "docker":
             # e.g. "quay.io/biocontainers/krona:2.7.1--pl526_5' }" -> 2.7.1--pl526_5
             # e.g. "biocontainers/biocontainers:v1.2.0_cv1' }" -> v1.2.0_cv1
-            match = re.search(r"(?:[/])?(?::)?([A-Za-z\d\-_.]+)['\"]", l)
+            match = re.search(r"(?:[/])?(?::)?([A-Za-z\d\-_.]+)'", l)
             if match is not None:
                 docker_tag = match.group(1)
                 self.passed.append(("docker_tag", f"Found docker tag: {docker_tag}", self.main_nf))
             else:
                 self.failed.append(("docker_tag", "Unable to parse docker tag", self.main_nf))
                 docker_tag = None
+        if l.startswith("container"):
+            container_section = l + lines[i + 1] + lines[i + 2]
+            if container_section.count('"') > 2:
+                self.failed.append(
+                    ("container_links", "Too many double quotes found when specifying containers", self.main_nf)
+                )
 
     # Check that all bioconda packages have build numbers
     # Also check for newer versions


Test patch:
Repository: neuralmagic/sparseml

Issue:
Fix transformers batchsize



Discussion:

Resolving patch:
diff --git a/src/sparseml/pytorch/image_classification/utils/trainer.py b/src/sparseml/pytorch/image_classification/utils/trainer.py
--- a/src/sparseml/pytorch/image_classification/utils/trainer.py
+++ b/src/sparseml/pytorch/image_classification/utils/trainer.py
@@ -33,6 +33,7 @@
     default_device,
     is_parallel_model,
 )
+from sparsezoo import Zoo
 
 
 _LOGGER = logging.getLogger(__file__)
@@ -327,6 +328,10 @@ def _run_train_epoch(
         )
 
     def _setup_checkpoint_manager(self):
+        if self.checkpoint_path and self.checkpoint_path.startswith("zoo"):
+            self.checkpoint_path = Zoo.load_model_from_stub(
+                self.checkpoint_path
+            ).download_framework_files(extensions=[".pth"])[0]
         checkpoint_state = torch.load(self.checkpoint_path)
         checkpoint_manager = None
         checkpoint_recipe = checkpoint_state.get("recipe")
diff --git a/src/sparseml/transformers/sparsification/trainer.py b/src/sparseml/transformers/sparsification/trainer.py
--- a/src/sparseml/transformers/sparsification/trainer.py
+++ b/src/sparseml/transformers/sparsification/trainer.py
@@ -254,9 +254,10 @@ def create_optimizer(self):
             if torch.distributed.is_initialized()
             else self.args._n_gpu
         )
+        n_device = n_gpu if n_gpu > 0 else 1
         total_batch_size = (
             self.args.per_device_train_batch_size
-            * n_gpu
+            * n_device
             * self.args.gradient_accumulation_steps
         )
         self.manager_steps_per_epoch = math.ceil(


Test patch: