Repository: clouditor/clouditor
Pull request #989 (2023-02-28 11:29:48)

Add example catalogs for testing
This PR updates the catalogs import. The catalogs are now read from a given folder. 
Updates: 
- changes orchestrator's catalogsFile to catalogsFolder ./catalogs,
- updates the loadEmbeddedCatalogs method to read several files form the folder ./catalogs.
- adds a README to ./catalog,
- adds a Demo Catalog to ./catalog and
- adds 1 example catalog to ./internal/testcatalogs for the tests.

codecov[bot]: # [Codecov](https://codecov.io/gh/clouditor/clouditor/pull/989?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=clouditor) Report
> Merging [#989](https://codecov.io/gh/clouditor/clouditor/pull/989?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=clouditor) (3d5514b) into [main](https://codecov.io/gh/clouditor/clouditor/commit/75f7e3685ac58b0b1919dbade6cc450e27b8e04e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=clouditor) (75f7e36) will **decrease** coverage by `0.10%`.
> The diff coverage is `n/a`.

```diff
@@            Coverage Diff             @@
##             main     #989      +/-   ##
==========================================
- Coverage   81.92%   81.83%   -0.10%     
==========================================
  Files          89       89              
  Lines        7573     7573              
==========================================
- Hits         6204     6197       -7     
- Misses       1050     1055       +5     
- Partials      319      321       +2     
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `81.83% <ø> (-0.10%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=clouditor#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/clouditor/clouditor/pull/989?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=clouditor) | Coverage Δ | |
|---|---|---|
| [api/streams.go](https://codecov.io/gh/clouditor/clouditor/pull/989?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=clouditor#diff-YXBpL3N0cmVhbXMuZ28=) | `87.23% <0.00%> (-2.84%)` | :arrow_down: |
| [service/assessment/assessment.go](https://codecov.io/gh/clouditor/clouditor/pull/989?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=clouditor#diff-c2VydmljZS9hc3Nlc3NtZW50L2Fzc2Vzc21lbnQuZ28=) | `88.80% <0.00%> (-0.79%)` | :arrow_down: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=clouditor). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=clouditor)


Patch:
diff --git a/catalogs/README.md b/catalogs/README.md
new file mode 100644
index 00000000..0ea47187
--- /dev/null
+++ b/catalogs/README.md
@@ -0,0 +1,11 @@
+# Demo catalog
+
+The folder 'catalog' contains a Demo Catalog for testing purposes and can be found [here](./demo_catalog.json). The flag **all in scope** is disabled fot the Demo Catalog and the **assurance levels** *low*, *medium* and *high* are used exemplarily.
+
+All catalogs may consist of controls, sub-controls and metrics.
+
+The flag **all in scope** can be enabled, which is necessary for catalogs that do not allow to select the scope of the controls and thus all controls are automatically in scope. 
+
+The controls of the catalog may contain different **assurance level** names, e.g., *low*, *medium* and *high*. These are used to decide which controls are used for the selected assurance level.
+
+**Note:** *all in scope* and the *assurance levels* are not mutually exclusive. It is possible to select the assurance level *low* in a catalog with *all in scope* enabled. That means that all controls with assurance level *low* must be used. 
diff --git a/catalogs/demo_catalog.json b/catalogs/demo_catalog.json
new file mode 100644
index 00000000..1d25dcb0
--- /dev/null
+++ b/catalogs/demo_catalog.json
@@ -0,0 +1,365 @@
+[
+    {
+        "id": "DemoCatalog",
+        "name": "A demo catalog for testing purposes",
+        "description": "A mock catalog for testing purposes",
+        "assurance_levels": ["low", "medium", "high"],
+        "all_in_scope": true,
+        "categories": [
+            {
+                "name": "Communication Security",
+                "description": "Secure all communications",
+                "controls": [
+                    {
+                        "id": "CommSec-01",
+                        "name": "Transport Encryption",
+                        "description": "The CSP should encrypt data transmissions",
+                        "controls": [
+                            {
+                                "id": "CommSec-01.1",
+                                "name": "The CSP shall use an up-to-date TLS version in all data transmissions",
+                                "metrics": [
+                                    {
+                                        "id": "TLSVersion"
+                                    },
+                                    {
+                                        "id": "TransportEncryptionAlgorihm"
+                                    }
+                                ],
+                                "parent_control_id": "CommSec-01",
+                                "parent_control_category_name": "Communication Security",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            },
+                            {
+                                "id": "CommSec-01.2",
+                                "name": "The CSP shall enforce encryption in all data transmissions",
+                                "metrics": [
+                                    {
+                                        "id": "TransportEncryptionEnforced"
+                                    }
+                                ],
+                                "parent_control_id": "CommSec-01",
+                                "parent_control_category_name": "Communication Security",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            },
+                            {
+                                "id": "CommSec-01.3",
+                                "name": "The CSP shall enable encryption in all data transmissions",
+                                "metrics": [
+                                    {
+                                        "id": "TransportEncryptionEnabled"
+                                    }
+                                ],
+                                "parent_control_id": "CommSec-01",
+                                "parent_control_category_name": "Communication Security",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    }
+                ]
+            },
+            {
+                "name": "System Hardening",
+                "description": "Harden all system components.",
+                "controls": [
+                    {
+                        "id": "SysHard-01",
+                        "name": "Transport",
+                        "description": "The CSP should encrypt all data transmissions",
+                        "controls": [
+                            {
+                                "id": "SysHard-01.1",
+                                "name": "The CSP shall use an up-to-date TLS version in all data transmissions",
+                                "metrics": [
+                                    {
+                                        "id": "TLSVersion"
+                                    },
+                                    {
+                                        "id": "TransportEncryptionAlgorihm"
+                                    }
+                                ],
+                                "parent_control_id": "SysHard-01",
+                                "parent_control_category_name": "System Hardening",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            },
+                            {
+                                "id": "SysHard-01.2",
+                                "name": "The CSP shall enforce encryption in all data transmissions.",
+                                "metrics": [
+                                    {
+                                        "id": "TransportEncryptionEnforced"
+                                    }
+                                ],
+                                "parent_control_id": "SysHard-01",
+                                "parent_control_category_name": "System Hardening",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            },
+                            {
+                                "id": "SysHard-01.3",
+                                "name": "The CSP shall enable encryption in all data transmissions",
+                                "metrics": [
+                                    {
+                                        "id": "TransportEncryptionEnabled"
+                                    }
+                                ],
+                                "parent_control_id": "SysHard-01",
+                                "parent_control_category_name": "System Hardening",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    },
+                    {
+                        "id": "SysHard-02",
+                        "name": "Firewall",
+                        "description": "The CSP should enable firewalls",
+                        "controls": [
+                            {
+                                "id": "SysHard-02.1",
+                                "name": "The CSP shall enable the Web Application Firewall (WAF).",
+                                "metrics": [
+                                    {
+                                        "id": "WebApplicationFireall"
+                                    }
+                                ],
+                                "parent_control_id": "SysHard-02",
+                                "parent_control_category_name": "System Hardening",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            },
+                            {
+                                "id": "SysHard-02.2",
+                                "name": "The CSP shall enable the L3 firewall and restrict specific ports.",
+                                "metrics": [
+                                    {
+                                        "id": "L3FirewallEnabled"
+                                    },
+                                    {
+                                        "id": "L3FirewallRestrictedPorts"
+                                    }
+                                ],
+                                "parent_control_id": "SysHard-02",
+                                "parent_control_category_name": "System Hardening",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    },
+                    {
+                        "id": "SysHard-03",
+                        "name": "Malware Protection",
+                        "description": "The CSP should provide malware protection.",
+                        "controls": [
+                            {
+                                "id": "SysHard-03.1",
+                                "name": "The CSP shall enable malware protection and store all log data.",
+                                "metrics": [
+                                    {
+                                        "id": "MalwareProtectionEnabled"
+                                    },
+                                    {
+                                        "id": "MalwareProtectionOutput"
+                                    }
+                                ],
+                                "parent_control_id": "SysHard-03",
+                                "parent_control_category_name": "System Hardening",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    },
+                    {
+                        "id": "SysHard-04",
+                        "name": "Automatic Updates",
+                        "description": "The CSP should enable firewalls",
+                        "controls": [
+                            {
+                                "id": "SysHard-04.1",
+                                "name": "The CSP shall enable automatic updates and execute in a specific time interall.",
+                                "metrics": [
+                                    {
+                                        "id": "AutomaticUpdatesEnabled"
+                                    },
+                                    {
+                                        "id": "AutomaticUpdatesInterval"
+                                    }
+                                ],
+                                "parent_control_id": "SysHard-04",
+                                "parent_control_category_name": "System Hardening",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    },
+                    {
+                        "id": "SysHard-05",
+                        "name": "Anomaly Detection",
+                        "description": "The CSP should provide anomaly detection mechanisms.",
+                        "controls": [
+                            {
+                                "id": "SysHard-05.1",
+                                "name": "The CSP shall enable anomaly detection and store all log data.",
+                                "metrics": [
+                                    {
+                                        "id": "AnomalyDetectionEnabled"
+                                    },
+                                    {
+                                        "id": "AnomalyDetectionOutput"
+                                    }
+                                ],
+                                "parent_control_id": "SysHard-05",
+                                "parent_control_category_name": "System Hardening",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    }, 
+                    {
+                        "id": "SysHard-06",
+                        "name": "Runtime",
+                        "description": "The CSP should provide state-of-the art runtime versions.",
+                        "controls": [
+                            {
+                                "id": "SysHard-06.1",
+                                "name": "The CSP shall use state-of-the-art runtime versions.",
+                                "metrics": [
+                                    {
+                                        "id": "RuntimeVersion"
+                                    }
+                                ],
+                                "parent_control_id": "SysHard-06",
+                                "parent_control_category_name": "System Hardening",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    }
+                ]
+            },
+            {
+                "name": "Logging",
+                "description": "Logging all activities.",
+                "controls": [
+                    {
+                        "id": "Log-01",
+                        "name": "Boot Logging",
+                        "description": "The CSP should log all boot activities.",
+                        "controls": [
+                            {
+                                "id": "Log-01.1",
+                                "name": "The CSP shall log all activies in a dedicated storage and retain the data for a specific time.",
+                                "metrics": [
+                                    {
+                                        "id": "BootLoggingEnabled"
+                                    },
+                                    {
+                                        "id": "BootLoggingOutput"
+                                    },
+                                    {
+                                        "id": "BootLoggingRetention"
+                                    }
+                                ],
+                                "parent_control_id": "Log-01",
+                                "parent_control_category_name": "Logging",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    },
+                    {
+                        "id": "Log-02",
+                        "name": "OS Logging",
+                        "description": "The CSP should log all OS activities.",
+                        "controls": [
+                            {
+                                "id": "Log-02.1",
+                                "name": "The CSP shall log all activies in a dedicated storage and retain the data for a specific time.",
+                                "metrics": [
+                                    {
+                                        "id": "OSLoggingEnabled"
+                                    },
+                                    {
+                                        "id": "OSLoggingOutput"
+                                    },
+                                    {
+                                        "id": "OSLoggingRetention"
+                                    }
+                                ],
+                                "parent_control_id": "Log-02",
+                                "parent_control_category_name": "Logging",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    },
+                    {
+                        "id": "Log-03",
+                        "name": "Protection Mechanisms",
+                        "description": "The CSP should log data from additional protection mechanisms.",
+                        "controls": [
+                            {
+                                "id": "Log-03.1",
+                                "name": "The CSP shall log malware protection log data.",
+                                "metrics": [
+                                    {
+                                        "id": "MalwareProtectionOutput"
+                                    }
+                                ],
+                                "parent_control_id": "Log-03",
+                                "parent_control_category_name": "Logging",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            },
+                            {
+                                "id": "Log-03.2",
+                                "name": "The CSP shall log anomaly detection log data.",
+                                "metrics": [
+                                    {
+                                        "id": "AnomalyDetectionOutput"
+                                    }
+                                ],
+                                "parent_control_id": "Log-03",
+                                "parent_control_category_name": "Logging",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    }
+                ]
+            },
+            {
+                "name": "Identity Security",
+                "description": "Identity and authentication security.",
+                "controls": [
+                    {
+                        "id": "IdSec-01",
+                        "name": "Identities",
+                        "description": "The CSP should check identities.",
+                        "controls": [
+                            {
+                                "id": "IdSec-01.1",
+                                "name": "The CSP shall check recent identity activities.",
+                                "metrics": [
+                                    {
+                                        "id": "IdentityRecentActivity"
+                                    }
+                                ],
+                                "parent_control_id": "IdSec-01",
+                                "parent_control_category_name": "Identity Security",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    },
+                    {
+                        "id": "IdSec-02",
+                        "name": "Password",
+                        "description": "The CSP should check user passwords.",
+                        "controls": [
+                            {
+                                "id": "IdSec-02.1",
+                                "name": "The CSP shall use password policies.",
+                                "metrics": [
+                                    {
+                                        "id": "IdentityPasswordPolicy"
+                                    }
+                                ],
+                                "parent_control_id": "IdSec-02",
+                                "parent_control_category_name": "Identity Security",
+                                "parent_control_category_catalog_id": "DemoCatalog"
+                            }
+                        ]
+                    }
+                ]
+            }
+        ]
+    }
+]
\ No newline at end of file
diff --git a/internal/testcatalogs/emptyTestFolder/.gitkeep b/internal/testcatalogs/emptyTestFolder/.gitkeep
new file mode 100644
index 00000000..e69de29b
diff --git a/internal/util/filesystem.go b/internal/util/filesystem.go
new file mode 100644
index 00000000..0830818b
--- /dev/null
+++ b/internal/util/filesystem.go
@@ -0,0 +1,52 @@
+// Copyright 2023 Fraunhofer AISEC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+//           $$\                           $$\ $$\   $$\
+//           $$ |                          $$ |\__|  $$ |
+//  $$$$$$$\ $$ | $$$$$$\  $$\   $$\  $$$$$$$ |$$\ $$$$$$\    $$$$$$\   $$$$$$\
+// $$  _____|$$ |$$  __$$\ $$ |  $$ |$$  __$$ |$$ |\_$$  _|  $$  __$$\ $$  __$$\
+// $$ /      $$ |$$ /  $$ |$$ |  $$ |$$ /  $$ |$$ |  $$ |    $$ /  $$ |$$ | \__|
+// $$ |      $$ |$$ |  $$ |$$ |  $$ |$$ |  $$ |$$ |  $$ |$$\ $$ |  $$ |$$ |
+// \$$$$$$\  $$ |\$$$$$   |\$$$$$   |\$$$$$$  |$$ |  \$$$   |\$$$$$   |$$ |
+//  \_______|\__| \______/  \______/  \_______|\__|   \____/  \______/ \__|
+//
+// This file is part of Clouditor Community Edition.
+
+package util
+
+import (
+	"fmt"
+	"os"
+	"strings"
+)
+
+// GetJSONFilenames returns all json files in the given folder
+func GetJSONFilenames(folder string) ([]string, error) {
+	var (
+		list []string
+	)
+
+	files, err := os.ReadDir(folder)
+	if err != nil {
+		return nil, err
+	}
+
+	for i := range files {
+		if strings.HasSuffix(files[i].Name(), ".json") {
+			list = append(list, fmt.Sprintf("%s/%s", folder, files[i].Name()))
+		}
+	}
+
+	return list, nil
+}
diff --git a/service/orchestrator/catalogs.go b/service/orchestrator/catalogs.go
index 8ae36d71..d481950e 100644
--- a/service/orchestrator/catalogs.go
+++ b/service/orchestrator/catalogs.go
@@ -5,9 +5,13 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
 
 	"clouditor.io/clouditor/api/orchestrator"
 	"clouditor.io/clouditor/internal/logging"
+	"clouditor.io/clouditor/internal/util"
 	"clouditor.io/clouditor/persistence"
 	"clouditor.io/clouditor/persistence/gorm"
 	"clouditor.io/clouditor/service"
@@ -220,23 +224,39 @@ func (svc *Service) loadCatalogs() (err error) {
 		log.Errorf("Error while saving catalog %v", err)
 	}
 
-	log.Debugf("Catalog loaded with id '%s'.", catalogs[0].GetId())
+	log.Debug("Catalogs successfully stored.")
 
 	return
 }
 
 func (svc *Service) loadEmbeddedCatalogs() (catalogs []*orchestrator.Catalog, err error) {
-	var b []byte
+	var (
+		b        []byte
+		fileList []string
+	)
 
-	log.Infof("Loading catalogs from %s", svc.catalogsFile)
-	b, err = f.ReadFile(svc.catalogsFile)
-	if err != nil {
-		return nil, fmt.Errorf("error while loading %s: %w", svc.catalogsFile, err)
-	}
+	// Get all filenames
+	fileList, err = util.GetJSONFilenames(svc.catalogsFolder)
 
-	err = json.Unmarshal(b, &catalogs)
-	if err != nil {
-		return nil, fmt.Errorf("error in JSON marshal: %w", err)
+	log.Infof("Loading catalogs from files %s.", strings.Join(fileList, ", "))
+
+	// Get catalog for each file
+	for i := range fileList {
+		var catalogsFromFile []*orchestrator.Catalog
+
+		b, err = os.ReadFile(filepath.Join(fileList[i]))
+		if err != nil {
+			log.Errorf("error while loading %s: %v", fileList[i], err)
+			continue
+		}
+
+		err = json.Unmarshal(b, &catalogsFromFile)
+		if err != nil {
+			log.Errorf("error in JSON marshal for file %s: %v", fileList[i], err)
+			continue
+		}
+
+		catalogs = append(catalogs, catalogsFromFile...)
 	}
 
 	// We need to make sure that sub-controls have the category_name and category_catalog_id of their parents set, otherwise we are failing a constraint.
@@ -250,5 +270,6 @@ func (svc *Service) loadEmbeddedCatalogs() (catalogs []*orchestrator.Catalog, er
 			}
 		}
 	}
+
 	return
 }
diff --git a/service/orchestrator/demo_catalogs.json b/service/orchestrator/demo_catalogs.json
deleted file mode 100644
index 2b3b0069..00000000
--- a/service/orchestrator/demo_catalogs.json
+++ /dev/null
@@ -1,35 +0,0 @@
-[
-    {
-        "id": "DemoCatalog",
-        "name": "A demo catalog for testing purposes",
-        "description": "A mock catalog for testing purposes",
-        "assurance_levels": ["low", "medium", "high"],
-        "categories": [
-            {
-                "name": "Communication Security",
-                "description": "Secure all communications",
-                "controls": [
-                    {
-                        "id": "CommSec-01",
-                        "name": "Transport Encryption",
-                        "description": "The CSP should encrypt data transmissions",
-                        "controls": [
-                            {
-                                "id": "CommSec-01.1",
-                                "name": "The CSP shall use an up-to-date TLS version in all data transmissions",
-                                "metrics": [
-                                    {
-                                        "id": "TLSVersion"
-                                    }
-                                ],
-                                "parent_control_id": "CommSec-01",
-                                "parent_control_category_name": "Communication Security",
-                                "parent_control_category_catalog_id": "DemoCatalog"
-                            }
-                        ]
-                    }
-                ]
-            }
-        ]
-    }
-]
\ No newline at end of file
diff --git a/service/orchestrator/orchestrator.go b/service/orchestrator/orchestrator.go
index 7cbb2f42..4df48780 100644
--- a/service/orchestrator/orchestrator.go
+++ b/service/orchestrator/orchestrator.go
@@ -49,8 +49,7 @@ import (
 var f embed.FS
 
 var DefaultMetricsFile = "metrics.json"
-
-var DefaultCatalogsFile = "demo_catalogs.json"
+var DefaultCatalogsFolder = "catalogs"
 
 var (
 	defaultMetricConfigurations map[string]*assessment.MetricConfiguration
@@ -83,7 +82,7 @@ type Service struct {
 	// loadMetricsFunc is a function that is used to initially load metrics at the start of the orchestrator
 	loadMetricsFunc func() ([]*assessment.Metric, error)
 
-	catalogsFile string
+	catalogsFolder string
 
 	// loadCatalogsFunc is a function that is used to initially load catalogs at the start of the orchestrator
 	loadCatalogsFunc func() ([]*orchestrator.Catalog, error)
@@ -116,10 +115,10 @@ func WithExternalMetrics(f func() ([]*assessment.Metric, error)) ServiceOption {
 	}
 }
 
-// WithCatalogsFile can be used to load a different catalogs file
-func WithCatalogsFile(file string) ServiceOption {
+// WithCatalogsFolder can be used to load catalog files from a different catalogs folder
+func WithCatalogsFolder(folder string) ServiceOption {
 	return func(s *Service) {
-		s.catalogsFile = file
+		s.catalogsFolder = folder
 	}
 }
 
@@ -154,9 +153,9 @@ func WithAuthorizationStrategy(authz service.AuthorizationStrategy) ServiceOptio
 func NewService(opts ...ServiceOption) *Service {
 	var err error
 	s := Service{
-		metricsFile:  DefaultMetricsFile,
-		catalogsFile: DefaultCatalogsFile,
-		events:       make(chan *orchestrator.MetricChangeEvent, 1000),
+		metricsFile:    DefaultMetricsFile,
+		catalogsFolder: DefaultCatalogsFolder,
+		events:         make(chan *orchestrator.MetricChangeEvent, 1000),
 	}
 
 	// Apply service options
@@ -317,6 +316,6 @@ func (svc *Service) RemoveCertificate(_ context.Context, req *orchestrator.Remov
 }
 
 // GetRuntimeInfo implements a method to retrieve runtime information
-func (svc *Service) GetRuntimeInfo(_ context.Context, _ *runtime.GetRuntimeInfoRequest) (res *runtime.Runtime, err error) {
+func (*Service) GetRuntimeInfo(_ context.Context, _ *runtime.GetRuntimeInfoRequest) (res *runtime.Runtime, err error) {
 	return service.GetRuntimeInfo()
 }


Test patch:
diff --git a/internal/util/filesystem_test.go b/internal/util/filesystem_test.go
new file mode 100644
index 00000000..c4fac90e
--- /dev/null
+++ b/internal/util/filesystem_test.go
@@ -0,0 +1,72 @@
+// Copyright 2023 Fraunhofer AISEC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+//           $$\                           $$\ $$\   $$\
+//           $$ |                          $$ |\__|  $$ |
+//  $$$$$$$\ $$ | $$$$$$\  $$\   $$\  $$$$$$$ |$$\ $$$$$$\    $$$$$$\   $$$$$$\
+// $$  _____|$$ |$$  __$$\ $$ |  $$ |$$  __$$ |$$ |\_$$  _|  $$  __$$\ $$  __$$\
+// $$ /      $$ |$$ /  $$ |$$ |  $$ |$$ /  $$ |$$ |  $$ |    $$ /  $$ |$$ | \__|
+// $$ |      $$ |$$ |  $$ |$$ |  $$ |$$ |  $$ |$$ |  $$ |$$\ $$ |  $$ |$$ |
+// \$$$$$$\  $$ |\$$$$$   |\$$$$$   |\$$$$$$  |$$ |  \$$$   |\$$$$$   |$$ |
+//  \_______|\__| \______/  \______/  \_______|\__|   \____/  \______/ \__|
+//
+// This file is part of Clouditor Community Edition.
+
+package util
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestGetJSONFilenames(t *testing.T) {
+	type args struct {
+		folder string
+	}
+	tests := []struct {
+		name    string
+		args    args
+		want    []string
+		wantErr bool
+	}{
+		{
+			name: "Empty input folder",
+			args: args{
+				folder: "",
+			},
+			want:    nil,
+			wantErr: true,
+		},
+		{
+			name: "Happy path",
+			args: args{
+				folder: "../../catalogs",
+			},
+			want:    []string{"../../catalogs/demo_catalog.json"},
+			wantErr: false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, err := GetJSONFilenames(tt.args.folder)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("GetJSONFilenames() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			if !reflect.DeepEqual(got, tt.want) {
+				t.Errorf("GetJSONFilenames() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}
diff --git a/service/orchestrator/catalogs_test.go b/service/orchestrator/catalogs_test.go
index c00b267e..ae38fb61 100644
--- a/service/orchestrator/catalogs_test.go
+++ b/service/orchestrator/catalogs_test.go
@@ -1,3 +1,28 @@
+// Copyright 2023 Fraunhofer AISEC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+//           $$\                           $$\ $$\   $$\
+//           $$ |                          $$ |\__|  $$ |
+//  $$$$$$$\ $$ | $$$$$$\  $$\   $$\  $$$$$$$ |$$\ $$$$$$\    $$$$$$\   $$$$$$\
+// $$  _____|$$ |$$  __$$\ $$ |  $$ |$$  __$$ |$$ |\_$$  _|  $$  __$$\ $$  __$$\
+// $$ /      $$ |$$ /  $$ |$$ |  $$ |$$ /  $$ |$$ |  $$ |    $$ /  $$ |$$ | \__|
+// $$ |      $$ |$$ |  $$ |$$ |  $$ |$$ |  $$ |$$ |  $$ |$$\ $$ |  $$ |$$ |
+// \$$$$$$\  $$ |\$$$$$   |\$$$$$   |\$$$$$$  |$$ |  \$$$   |\$$$$$   |$$ |
+//  \_______|\__| \______/  \______/  \_______|\__|   \____/  \______/ \__|
+//
+// This file is part of Clouditor Community Edition.
+
 package orchestrator
 
 import (
@@ -193,12 +218,12 @@ func TestService_ListCatalogs(t *testing.T) {
 		err                  error
 	)
 
-	orchestratorService := NewService()
+	orchestratorService := NewService(WithCatalogsFolder("catalogs"))
 	// 1st case: Default catalogs stored
 	listCatalogsResponse, err = orchestratorService.ListCatalogs(context.Background(), &orchestrator.ListCatalogsRequest{})
 	assert.NoError(t, err)
 	assert.NotNil(t, listCatalogsResponse.Catalogs)
-	assert.Equal(t, len(listCatalogsResponse.Catalogs), 1)
+	assert.Equal(t, 1, len(listCatalogsResponse.Catalogs))
 }
 
 func TestService_UpdateCatalog(t *testing.T) {
@@ -252,7 +277,7 @@ func TestService_RemoveCatalog(t *testing.T) {
 		err                  error
 		listCatalogsResponse *orchestrator.ListCatalogsResponse
 	)
-	orchestratorService := NewService()
+	orchestratorService := NewService(WithCatalogsFolder("internal/testcatalogs/emptyTestFolder"))
 
 	// 1st case: Empty catalog ID error
 	_, err = orchestratorService.RemoveCatalog(context.Background(), &orchestrator.RemoveCatalogRequest{CatalogId: ""})
@@ -269,21 +294,21 @@ func TestService_RemoveCatalog(t *testing.T) {
 	err = orchestratorService.storage.Create(mockCatalog)
 	assert.NoError(t, err)
 
-	// There are two catalogs in the db now (one default plus NewCatalog)
+	// There is one catalog in the db now (one default plus NewCatalog)
 	listCatalogsResponse, err = orchestratorService.ListCatalogs(context.Background(), &orchestrator.ListCatalogsRequest{})
 	assert.NoError(t, err)
 	assert.NotNil(t, listCatalogsResponse.Catalogs)
-	assert.Equal(t, 2, len(listCatalogsResponse.Catalogs))
+	assert.Equal(t, 1, len(listCatalogsResponse.Catalogs))
 
 	// Remove record
 	_, err = orchestratorService.RemoveCatalog(context.Background(), &orchestrator.RemoveCatalogRequest{CatalogId: mockCatalog.Id})
 	assert.NoError(t, err)
 
-	// There are two records left in the DB
+	// There is no record left in the DB
 	listCatalogsResponse, err = orchestratorService.ListCatalogs(context.Background(), &orchestrator.ListCatalogsRequest{})
 	assert.NoError(t, err)
 	assert.NotNil(t, listCatalogsResponse.Catalogs)
-	assert.Equal(t, 1, len(listCatalogsResponse.Catalogs))
+	assert.Equal(t, 0, len(listCatalogsResponse.Catalogs))
 }
 
 func TestService_GetCategory(t *testing.T) {
@@ -369,7 +394,7 @@ func TestService_GetCategory(t *testing.T) {
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
 				loadMetricsFunc:       tt.fields.loadMetricsFunc,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFile,
 				loadCatalogsFunc:      tt.fields.loadCatalogsFunc,
 				events:                tt.fields.events,
 			}
@@ -390,7 +415,7 @@ func TestService_GetControl(t *testing.T) {
 		storage               persistence.Storage
 		metricsFile           string
 		loadMetricsFunc       func() ([]*assessment.Metric, error)
-		catalogsFile          string
+		catalogsFolder        string
 		loadCatalogsFunc      func() ([]*orchestrator.Catalog, error)
 	}
 	type args struct {
@@ -457,7 +482,7 @@ func TestService_GetControl(t *testing.T) {
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
 				loadMetricsFunc:       tt.fields.loadMetricsFunc,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				loadCatalogsFunc:      tt.fields.loadCatalogsFunc,
 			}
 			gotRes, err := srv.GetControl(tt.args.ctx, tt.args.req)
@@ -476,20 +501,21 @@ func TestService_ListControls(t *testing.T) {
 		err                  error
 	)
 
-	orchestratorService := NewService()
+	orchestratorService := NewService(WithCatalogsFolder("internal/testcatalogs/emptyTestFolder"))
 	// 1st case: No Controls stored
 	listControlsResponse, err = orchestratorService.ListControls(context.Background(), &orchestrator.ListControlsRequest{})
 	assert.NoError(t, err)
 	assert.NotNil(t, listControlsResponse.Controls)
-	assert.NotEmpty(t, listControlsResponse.Controls)
+	assert.Empty(t, listControlsResponse.Controls)
 
-	// 2nd case: Two controls stored; note that we do not have to create an extra catalog/control since NewService above already loads the default catalogs/controls
+	// 2nd case: 30 controls stored; note that we do not have to create an extra catalog/control since NewService above already loads the default catalogs/controls
+	orchestratorService = NewService(WithCatalogsFolder("catalogs"))
 	listControlsResponse, err = orchestratorService.ListControls(context.Background(), &orchestrator.ListControlsRequest{})
 	assert.NoError(t, err)
 	assert.NotNil(t, listControlsResponse.Controls)
 	assert.NotEmpty(t, listControlsResponse.Controls)
-	// there are two default controls
-	assert.Equal(t, 2, len(listControlsResponse.Controls))
+	// there are 30 default controls
+	assert.Equal(t, 30, len(listControlsResponse.Controls))
 
 	// 3th case: List controls for a specific catalog and category.
 	listControlsResponse, err = orchestratorService.ListControls(context.Background(), &orchestrator.ListControlsRequest{
@@ -499,7 +525,7 @@ func TestService_ListControls(t *testing.T) {
 	assert.NoError(t, err)
 	assert.NotNil(t, listControlsResponse.Controls)
 	assert.NotEmpty(t, listControlsResponse.Controls)
-	assert.Equal(t, 2, len(listControlsResponse.Controls))
+	assert.Equal(t, 4, len(listControlsResponse.Controls))
 }
 
 func TestService_loadCatalogs(t *testing.T) {
@@ -508,7 +534,7 @@ func TestService_loadCatalogs(t *testing.T) {
 		storage               persistence.Storage
 		metricsFile           string
 		loadMetricsFunc       func() ([]*assessment.Metric, error)
-		catalogsFile          string
+		catalogsFolder        string
 		loadCatalogsFunc      func() ([]*orchestrator.Catalog, error)
 		events                chan *orchestrator.MetricChangeEvent
 	}
@@ -531,8 +557,8 @@ func TestService_loadCatalogs(t *testing.T) {
 		{
 			name: "storage error",
 			fields: fields{
-				catalogsFile: "demo_catalogs.json",
-				storage:      &testutil.StorageWithError{SaveErr: ErrSomeError},
+				catalogsFolder: "internal/testcatalogs/emptyTestFolder",
+				storage:        &testutil.StorageWithError{SaveErr: ErrSomeError},
 			},
 			wantResult: assert.NotEmpty,
 			wantErr: func(tt assert.TestingT, err error, i ...interface{}) bool {
@@ -554,8 +580,8 @@ func TestService_loadCatalogs(t *testing.T) {
 		{
 			name: "Happy path",
 			fields: fields{
-				catalogsFile: "demo_catalogs.json",
-				storage:      testutil.NewInMemoryStorage(t),
+				catalogsFolder: "catalogs",
+				storage:        testutil.NewInMemoryStorage(t),
 			},
 			wantResult: func(t assert.TestingT, i interface{}, i2 ...interface{}) bool {
 				svc, ok := i.(*Service)
@@ -579,7 +605,7 @@ func TestService_loadCatalogs(t *testing.T) {
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
 				loadMetricsFunc:       tt.fields.loadMetricsFunc,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				loadCatalogsFunc:      tt.fields.loadCatalogsFunc,
 				events:                tt.fields.events,
 			}
diff --git a/service/orchestrator/cloud_service_test.go b/service/orchestrator/cloud_service_test.go
index f760f968..5a201418 100644
--- a/service/orchestrator/cloud_service_test.go
+++ b/service/orchestrator/cloud_service_test.go
@@ -329,7 +329,7 @@ func TestService_ListCloudServices(t *testing.T) {
 		storage               persistence.Storage
 		metricsFile           string
 		loadMetricsFunc       func() ([]*assessment.Metric, error)
-		catalogsFile          string
+		catalogsFolder        string
 		loadCatalogsFunc      func() ([]*orchestrator.Catalog, error)
 		events                chan *orchestrator.MetricChangeEvent
 		authz                 service.AuthorizationStrategy
@@ -450,7 +450,7 @@ func TestService_ListCloudServices(t *testing.T) {
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
 				loadMetricsFunc:       tt.fields.loadMetricsFunc,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				loadCatalogsFunc:      tt.fields.loadCatalogsFunc,
 				events:                tt.fields.events,
 				authz:                 tt.fields.authz,
diff --git a/service/orchestrator/metrics_test.go b/service/orchestrator/metrics_test.go
index b534a33f..e43d43eb 100644
--- a/service/orchestrator/metrics_test.go
+++ b/service/orchestrator/metrics_test.go
@@ -57,7 +57,7 @@ func TestService_loadMetrics(t *testing.T) {
 		storage               persistence.Storage
 		metricsFile           string
 		loadMetricsFunc       func() ([]*assessment.Metric, error)
-		catalogsFile          string
+		catalogsFolder        string
 		loadCatalogsFunc      func() ([]*orchestrator.Catalog, error)
 		events                chan *orchestrator.MetricChangeEvent
 	}
@@ -105,7 +105,7 @@ func TestService_loadMetrics(t *testing.T) {
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
 				loadMetricsFunc:       tt.fields.loadMetricsFunc,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				loadCatalogsFunc:      tt.fields.loadCatalogsFunc,
 				events:                tt.fields.events,
 			}
@@ -546,7 +546,7 @@ func TestService_GetMetricImplementation(t *testing.T) {
 		storage               persistence.Storage
 		metricsFile           string
 		events                chan *orchestrator.MetricChangeEvent
-		catalogsFile          string
+		catalogsFolder        string
 	}
 	type args struct {
 		ctx context.Context
@@ -616,7 +616,7 @@ func TestService_GetMetricImplementation(t *testing.T) {
 				AssessmentResultHooks: tt.fields.AssessmentResultHooks,
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				events:                tt.fields.events,
 			}
 
@@ -639,7 +639,7 @@ func TestService_UpdateMetricImplementation(t *testing.T) {
 		AssessmentResultHooks []func(result *assessment.AssessmentResult, err error)
 		storage               persistence.Storage
 		metricsFile           string
-		catalogsFile          string
+		catalogsFolder        string
 		events                chan *orchestrator.MetricChangeEvent
 	}
 	type args struct {
@@ -719,7 +719,7 @@ func TestService_UpdateMetricImplementation(t *testing.T) {
 				AssessmentResultHooks: tt.fields.AssessmentResultHooks,
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				events:                tt.fields.events,
 			}
 			gotImpl, err := svc.UpdateMetricImplementation(tt.args.in0, tt.args.req)
@@ -738,7 +738,7 @@ func TestService_GetMetricConfiguration(t *testing.T) {
 		AssessmentResultHooks []func(result *assessment.AssessmentResult, err error)
 		storage               persistence.Storage
 		metricsFile           string
-		catalogsFile          string
+		catalogsFolder        string
 		events                chan *orchestrator.MetricChangeEvent
 		authz                 service.AuthorizationStrategy
 	}
@@ -830,7 +830,7 @@ func TestService_GetMetricConfiguration(t *testing.T) {
 				AssessmentResultHooks: tt.fields.AssessmentResultHooks,
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				events:                tt.fields.events,
 				authz:                 tt.fields.authz,
 			}
@@ -849,7 +849,7 @@ func TestService_ListMetricConfigurations(t *testing.T) {
 		AssessmentResultHooks []func(result *assessment.AssessmentResult, err error)
 		storage               persistence.Storage
 		metricsFile           string
-		catalogsFile          string
+		catalogsFolder        string
 		events                chan *orchestrator.MetricChangeEvent
 		authz                 service.AuthorizationStrategy
 	}
@@ -914,7 +914,7 @@ func TestService_ListMetricConfigurations(t *testing.T) {
 				AssessmentResultHooks: tt.fields.AssessmentResultHooks,
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				events:                tt.fields.events,
 				authz:                 tt.fields.authz,
 			}
@@ -940,7 +940,7 @@ func TestService_UpdateMetricConfiguration(t *testing.T) {
 		storage               persistence.Storage
 		metricsFile           string
 		loadMetricsFunc       func() ([]*assessment.Metric, error)
-		catalogsFile          string
+		catalogsFolder        string
 		loadCatalogsFunc      func() ([]*orchestrator.Catalog, error)
 		events                chan *orchestrator.MetricChangeEvent
 		authz                 service.AuthorizationStrategy
@@ -1239,7 +1239,7 @@ func TestService_UpdateMetricConfiguration(t *testing.T) {
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
 				loadMetricsFunc:       tt.fields.loadMetricsFunc,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				loadCatalogsFunc:      tt.fields.loadCatalogsFunc,
 				events:                tt.fields.events,
 				authz:                 tt.fields.authz,
diff --git a/service/orchestrator/orchestrator_test.go b/service/orchestrator/orchestrator_test.go
index 5868b754..feedb64b 100644
--- a/service/orchestrator/orchestrator_test.go
+++ b/service/orchestrator/orchestrator_test.go
@@ -31,7 +31,6 @@ import (
 	"os"
 	"reflect"
 	"runtime"
-	"strings"
 	"sync"
 	"testing"
 
@@ -106,7 +105,7 @@ func TestNewService(t *testing.T) {
 		{
 			name: "New service with catalogs file",
 			args: args{
-				opts: []ServiceOption{WithCatalogsFile("catalogsfile.json")},
+				opts: []ServiceOption{WithCatalogsFolder("catalogsFolder.json")},
 			},
 			want: func(tt assert.TestingT, i1 interface{}, i2 ...interface{}) bool {
 				s, ok := i1.(*Service)
@@ -114,7 +113,7 @@ func TestNewService(t *testing.T) {
 					return false
 				}
 
-				return assert.Equal(tt, "catalogsfile.json", s.catalogsFile)
+				return assert.Equal(tt, "catalogsFolder.json", s.catalogsFolder)
 			},
 		},
 		{
@@ -399,11 +398,10 @@ func Test_GetCertificate(t *testing.T) {
 
 			if tt.res != nil {
 				assert.NotEmpty(t, res.Id)
-				// Compare timestamp. We have to cut off the microseconds and seconds, otherwise an error can be returned.
-				t1 := strings.Split(tt.res.States[0].GetTimestamp(), ".")[0]
-				tt.res.States[0].Timestamp = t1[:len(t1)-3]
-				t2 := strings.Split(res.States[0].GetTimestamp(), ".")[0]
-				res.States[0].Timestamp = t2[:len(t2)-3]
+				// Check if timestamp is available and delete for comparison.
+				assert.NotEmpty(t, res.States[0].GetTimestamp())
+				res.States[0].Timestamp = ""
+				tt.res.States[0].Timestamp = ""
 				assert.True(t, proto.Equal(tt.res, res), "Want: %v\nGot : %v", tt.res, res)
 			}
 		})
diff --git a/service/orchestrator/toe_test.go b/service/orchestrator/toe_test.go
index 3561adcc..c1686c57 100644
--- a/service/orchestrator/toe_test.go
+++ b/service/orchestrator/toe_test.go
@@ -59,7 +59,7 @@ func TestService_CreateTargetOfEvaluation(t *testing.T) {
 		storage               persistence.Storage
 		metricsFile           string
 		loadMetricsFunc       func() ([]*assessment.Metric, error)
-		catalogsFile          string
+		catalogsFolder        string
 		loadCatalogsFunc      func() ([]*orchestrator.Catalog, error)
 		events                chan *orchestrator.MetricChangeEvent
 		authz                 service.AuthorizationStrategy
@@ -190,7 +190,7 @@ func TestService_CreateTargetOfEvaluation(t *testing.T) {
 				storage:               tt.fields.storage,
 				metricsFile:           tt.fields.metricsFile,
 				loadMetricsFunc:       tt.fields.loadMetricsFunc,
-				catalogsFile:          tt.fields.catalogsFile,
+				catalogsFolder:        tt.fields.catalogsFolder,
 				loadCatalogsFunc:      tt.fields.loadCatalogsFunc,
 				events:                tt.fields.events,
 				authz:                 tt.fields.authz,