Repository: nf-core/tools

Issue:
Not all `definitions` in JSON schema have a "properties", leading to an error.
### Description of the bug

Relevant code: https://github.com/nf-core/tools/blob/a1bf2ad5cd6dc1e74c5959c467de556c94672693/nf_core/pipeline-template/lib/NfcoreSchema.groovy#L390-L402

Nextflow prints an error when including an nf-core workflow as a subworkflow and the schema doesn't match.
```
N E X T F L O W  ~  version 21.10.6
Launching `main.nf` [nasty_noyce] - revision: 31a6c995e0
JSONObject["properties"] not found.
```

When using `nf-core schema build` to make the JSON schema, not all entries have the "properties" key, but the code expects it to be there. I'm not sure if this an issue for the builder or for the checker.


### Command used and terminal output

```console
Error:

$ nextflow run main.nf -profile docker
Picked up JAVA_TOOL_OPTIONS:  -Xmx3435m
N E X T F L O W  ~  version 21.10.6
Launching `main.nf` [nasty_noyce] - revision: 31a6c995e0
JSONObject["properties"] not found.

 -- Check script 'subworkflows/fetchngs/main.nf' at line: 20 or see '.nextflow.log' file for more details
```


Relevant part of the JSON Schema (built using `nf-core schema build`)
```json
{
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "https://raw.githubusercontent.com/nf-core/fetchngs/master/nextflow_schema.json",
    "title": "nf-core/fetchngs pipeline parameters",
    "description": "Pipeline to fetch metadata and raw FastQ files from public databases",
    "type": "object",
    "definitions": {
        "input_output_options": {
            "title": "Input/output options",
            "type": "object",
            "fa_icon": "fas fa-terminal",
            "description": "Define where the pipeline should find input data and save output data.",
            "required": [
                "input",
                "outdir"
            ],
            "properties": {
                "input": {
                    "type": "string",
                    "format": "file-path",
                    "mimetype": "text/csv",
                    "pattern": "^\\S+\\.txt$",
                    "schema": "assets/schema_input.json",
                    "description": "Path to text file containing information about the samples in the experiment.",
                    "help_text": "You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. See [usage docs](https://nf-co.re//usage#samplesheet-input).",
                    "fa_icon": "fas fa-file-csv"
                },
                "outdir": {
                    "type": "string",
                    "description": "Path to the output directory where the results will be saved.",
                    "default": "./results",
                    "fa_icon": "fas fa-folder-open"
                },
                "email": {
                    "type": "string",
                    "description": "Email address for completion summary.",
                    "fa_icon": "fas fa-envelope",
                    "help_text": "Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run.",
                    "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$"
                }
            }
        },
        "reference_genome_options": {
            "title": "Reference genome options",
            "type": "object",
            "fa_icon": "fas fa-dna",
            "description": "Reference genome related files and options required for the workflow."
        },
```
`reference_genome_options` has no `properties` tag.
Not all `definitions` in JSON schema have a "properties", leading to an error.
### Description of the bug

Relevant code: https://github.com/nf-core/tools/blob/a1bf2ad5cd6dc1e74c5959c467de556c94672693/nf_core/pipeline-template/lib/NfcoreSchema.groovy#L390-L402

Nextflow prints an error when including an nf-core workflow as a subworkflow and the schema doesn't match.
```
N E X T F L O W  ~  version 21.10.6
Launching `main.nf` [nasty_noyce] - revision: 31a6c995e0
JSONObject["properties"] not found.
```

When using `nf-core schema build` to make the JSON schema, not all entries have the "properties" key, but the code expects it to be there. I'm not sure if this an issue for the builder or for the checker.


### Command used and terminal output

```console
Error:

$ nextflow run main.nf -profile docker
Picked up JAVA_TOOL_OPTIONS:  -Xmx3435m
N E X T F L O W  ~  version 21.10.6
Launching `main.nf` [nasty_noyce] - revision: 31a6c995e0
JSONObject["properties"] not found.

 -- Check script 'subworkflows/fetchngs/main.nf' at line: 20 or see '.nextflow.log' file for more details
```


Relevant part of the JSON Schema (built using `nf-core schema build`)
```json
{
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "https://raw.githubusercontent.com/nf-core/fetchngs/master/nextflow_schema.json",
    "title": "nf-core/fetchngs pipeline parameters",
    "description": "Pipeline to fetch metadata and raw FastQ files from public databases",
    "type": "object",
    "definitions": {
        "input_output_options": {
            "title": "Input/output options",
            "type": "object",
            "fa_icon": "fas fa-terminal",
            "description": "Define where the pipeline should find input data and save output data.",
            "required": [
                "input",
                "outdir"
            ],
            "properties": {
                "input": {
                    "type": "string",
                    "format": "file-path",
                    "mimetype": "text/csv",
                    "pattern": "^\\S+\\.txt$",
                    "schema": "assets/schema_input.json",
                    "description": "Path to text file containing information about the samples in the experiment.",
                    "help_text": "You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. See [usage docs](https://nf-co.re//usage#samplesheet-input).",
                    "fa_icon": "fas fa-file-csv"
                },
                "outdir": {
                    "type": "string",
                    "description": "Path to the output directory where the results will be saved.",
                    "default": "./results",
                    "fa_icon": "fas fa-folder-open"
                },
                "email": {
                    "type": "string",
                    "description": "Email address for completion summary.",
                    "fa_icon": "fas fa-envelope",
                    "help_text": "Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run.",
                    "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$"
                }
            }
        },
        "reference_genome_options": {
            "title": "Reference genome options",
            "type": "object",
            "fa_icon": "fas fa-dna",
            "description": "Reference genome related files and options required for the workflow."
        },
```
`reference_genome_options` has no `properties` tag.


Discussion:
Hmm, probably a problem of the builder. Did you create this schema by (re)moving all of the parameters from the `reference_genome_options` group?
It was the default setting after running `nf-core schema build` and opening in the browser. I didn't do anything to it. 
which pipeline?
https://github.com/mahesh-panchal/test_nfcore_workflow_chain
Sorry, I can't reproduce your error (cloned your repo and ran the test profile). I don't see `reference_genome_options` in the schema of the linked repo.
The workflow works at the moment. I got rid of the "reference_genome_options" which was causing the error using the builder. To be honest, I'm not even sure why the "reference_genome_options" is there, but I assume because it pulled it from some subworkflow. 

But to recreate the issue, delete the `nextflow_schema.json` and then run `nf-core schema build`. That should recreate it. 
Okay, I think I found the problem. When building a schema it takes a skeleton schema from the template (which contains the `reference_genome_options` group containing `fasta` etc.) and then removes the unneeded parameters, but not the now empty group. This step should probably be part of the `remove_schema_notfound_configs()` function in `schema.py`.
@mashehu sounds like you've figured this one out - would you mind putting in a PR to fix it please?
To reproduce run:
1. `nf-core create` to create a fresh pipeline from the template
2. Delete [these 3 lines](https://github.com/nf-core/tools/blob/e0d00d3e09a996f59eb5ce6ebc4795452a5b1d39/nf_core/pipeline-template/nextflow.config#L17-L19) in the `nextflow.config` 
3. Delete [this line](https://github.com/nf-core/tools/blob/e0d00d3e09a996f59eb5ce6ebc4795452a5b1d39/nf_core/pipeline-template/main.nf#L20) in `main.nf`
4. Run `rm nextflow_schema.json && nf-core schema build`

This should update the scheme to remove all of the `params` in the `Reference genome options` group but will still leave the group in the schema without any `properties` which is causing the error.
Hmm, probably a problem of the builder. Did you create this schema by (re)moving all of the parameters from the `reference_genome_options` group?
It was the default setting after running `nf-core schema build` and opening in the browser. I didn't do anything to it. 
which pipeline?
https://github.com/mahesh-panchal/test_nfcore_workflow_chain
Sorry, I can't reproduce your error (cloned your repo and ran the test profile). I don't see `reference_genome_options` in the schema of the linked repo.
The workflow works at the moment. I got rid of the "reference_genome_options" which was causing the error using the builder. To be honest, I'm not even sure why the "reference_genome_options" is there, but I assume because it pulled it from some subworkflow. 

But to recreate the issue, delete the `nextflow_schema.json` and then run `nf-core schema build`. That should recreate it. 
Okay, I think I found the problem. When building a schema it takes a skeleton schema from the template (which contains the `reference_genome_options` group containing `fasta` etc.) and then removes the unneeded parameters, but not the now empty group. This step should probably be part of the `remove_schema_notfound_configs()` function in `schema.py`.
@mashehu sounds like you've figured this one out - would you mind putting in a PR to fix it please?
To reproduce run:
1. `nf-core create` to create a fresh pipeline from the template
2. Delete [these 3 lines](https://github.com/nf-core/tools/blob/e0d00d3e09a996f59eb5ce6ebc4795452a5b1d39/nf_core/pipeline-template/nextflow.config#L17-L19) in the `nextflow.config` 
3. Delete [this line](https://github.com/nf-core/tools/blob/e0d00d3e09a996f59eb5ce6ebc4795452a5b1d39/nf_core/pipeline-template/main.nf#L20) in `main.nf`
4. Run `rm nextflow_schema.json && nf-core schema build`

This should update the scheme to remove all of the `params` in the `Reference genome options` group but will still leave the group in the schema without any `properties` which is causing the error.

Resolving patch:
diff --git a/nf_core/schema.py b/nf_core/schema.py
--- a/nf_core/schema.py
+++ b/nf_core/schema.py
@@ -536,6 +536,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url):
             self.get_wf_params()
             self.make_skeleton_schema()
             self.remove_schema_notfound_configs()
+            self.remove_schema_empty_definitions()
             self.add_schema_found_configs()
             try:
                 self.validate_schema()
@@ -555,6 +556,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url):
         if not self.web_only:
             self.get_wf_params()
             self.remove_schema_notfound_configs()
+            self.remove_schema_empty_definitions()
             self.add_schema_found_configs()
             self.save_schema()
 
@@ -613,6 +615,25 @@ def get_wf_params(self):
                 )
             )
 
+    def remove_schema_empty_definitions(self):
+        """
+        Go through top-level schema remove definitions that don't have
+        any property attributes
+        """
+        # Identify and remove empty definitions from the schema
+        empty_definitions = []
+        for d_key, d_schema in list(self.schema.get("definitions", {}).items()):
+            if not d_schema.get("properties"):
+                del self.schema["definitions"][d_key]
+                empty_definitions.append(d_key)
+                log.warning(f"Removing empty group: '{d_key}'")
+
+        # Remove "allOf" group with empty definitions from the schema
+        for d_key in empty_definitions:
+            allOf = {"$ref": f"#/definitions/{d_key}"}
+            if allOf in self.schema.get("allOf", []):
+                self.schema["allOf"].remove(allOf)
+
     def remove_schema_notfound_configs(self):
         """
         Go through top-level schema and all definitions sub-schemas to remove
@@ -625,6 +646,7 @@ def remove_schema_notfound_configs(self):
             cleaned_schema, p_removed = self.remove_schema_notfound_configs_single_schema(definition)
             self.schema["definitions"][d_key] = cleaned_schema
             params_removed.extend(p_removed)
+
         return params_removed
 
     def remove_schema_notfound_configs_single_schema(self, schema):
@@ -772,6 +794,7 @@ def get_web_builder_response(self):
             log.info("Found saved status from nf-core schema builder")
             try:
                 self.schema = web_response["schema"]
+                self.remove_schema_empty_definitions()
                 self.validate_schema()
             except AssertionError as e:
                 raise AssertionError("Response from schema builder did not pass validation:\n {}".format(e))


Test patch: