Repository: opsdroid/opsdroid

Issue:
Update example config script
Our `configuration.yaml` file has suffered a few alterations and the script that should update the `example_configuration.yaml` file is really out of date. 

There are also a few bugs where the skill is added both to commented ones and active ones.

We should update this script in order to get out configuration files up to date. We should also update the `example_configuration.yaml` file.


Discussion:
Hi
Can I work on this issue?
Hello! That would be awesome!

Please let me know if you need any help whilst working on this issue 😄👍
Hi
I went through the `example_config.yaml` and the corresponding script that updates it, and I don't understand what needs to be updated.
Hello @bistaastha  Apologies for not replying to you sooner. 
You will need to do a few things to fix this issue:

1.  Update the script to remove the connectors/databases that have been included in the core
2. Make sure that the connectors/databases that have been included into core are shown in the example_config.yaml
3. Update the script to not include duplicate things - sometimes it would leave commented out skills and letter would show them commented as well
4. Update the example_config with the latest version.

I hope this made things more clear?
Yes, it does. Thank you :)
Hi.
I'm editing the example_config.yaml right now. There are skills which are both commented out and active. Which part has to be kept?
The default connectors/skills that should remain uncommented are:
connectors: websockets
skills: dance, hello, loudnoises, seen

All the others should remain commented out. Hope this helps
Actually loudnoises and seen were commented as well as uncommented.
Thanks, this helps.
Hi @bistaastha ,
I was going through the conversation.are you still working on the same?
I was thinking about helping out so let  me know if you need my help.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Since we haven't got any PR being raised in regards to this issue I am going to say that this issue is still open and we are accepting PR's to fix this 😄 
Hello,
can I take this issue? 
Sure @jos3p!
> Hello,
> can I take this issue?

Hello @jos3p if you are not working on this issue I would like to try, can you update me where you left off, or need help?
I'm still on it, will post a pullrequest soon

Resolving patch:
diff --git a/scripts/update_example_config/update_example_config.py b/scripts/update_example_config/update_example_config.py
--- a/scripts/update_example_config/update_example_config.py
+++ b/scripts/update_example_config/update_example_config.py
@@ -1,4 +1,4 @@
-from github import Github
+from github import Github, Repository
 from argparse import ArgumentParser
 import jinja2
 import base64
@@ -31,6 +31,21 @@ def render(tpl_path, context):
     )
 
 
+def get_core_modules():
+    """Get core module names of databases and connectors."""
+    core_modules = [
+        "database-" + name
+        for name in os.listdir("./opsdroid/database")
+        if os.path.isdir(os.path.join("./opsdroid/database", name))
+    ]
+    core_modules += [
+        "connector-" + name
+        for name in os.listdir("./opsdroid/connector")
+        if os.path.isdir(os.path.join("./opsdroid/connector", name))
+    ]
+    return core_modules
+
+
 def get_repos():
     """Get repository details for: skills, connectors, database."""
     return [
@@ -42,14 +57,25 @@ def get_repos():
     ]
 
 
-def get_readme(repo):
+def get_readme(module):
     """Get readme.md details from repository."""
-    readme_base64 = repo.get_readme().content
-    return base64.b64decode(readme_base64).decode("utf-8")
+    isRepo = isinstance(module, Repository.Repository)
+    if isRepo:
+        readme_base64 = module.get_readme().content
+        return base64.b64decode(readme_base64).decode("utf-8")
+    else:
+        if module[:9] == "connector":
+            mdfile = module[10:] + ".md"
+            subfolder = "connectors/"
+        else:
+            mdfile = module[9:] + ".md"
+            subfolder = "databases/"
+        core_readme = open("./docs/" + subfolder + mdfile, "rb").read().decode("utf-8")
+        return core_readme
 
 
 def get_config_details(readme):
-    """Gets all the configuration details located in the readme.md file,
+    """Gets all the configuration details located in the readme.md/<modulename>.md file of modules,
     under the title "Configuration".
 
     Note: Regex divided by multiline in order to pass lint.
@@ -62,20 +88,33 @@ def get_config_details(readme):
     return config
 
 
-def get_config_params(repo, readme):
+def get_config_params(module, readme):
     """Returns parameters to be used in the update."""
-    if repo.name[:5] == "skill":
-        raw_name = repo.name[6:]
-        repo_type = "skill"
-    elif repo.name[:9] == "connector":
-        raw_name = repo.name[10:]
-        repo_type = "connector"
+    isRepo = isinstance(module, Repository.Repository)
+
+    if isRepo:
+        if module.name[:5] == "skill":
+            raw_name = module.name[6:]
+            repo_type = "skill"
+        elif module.name[:9] == "connector":
+            raw_name = module.name[10:]
+            repo_type = "connector"
+        else:
+            raw_name = module.name[9:]
+            repo_type = "database"
     else:
-        raw_name = repo.name[9:]
-        repo_type = "database"
+        if module[:5] == "skill":
+            raw_name = module[6:]
+            repo_type = "skill"
+        elif module[:9] == "connector":
+            raw_name = module[10:]
+            repo_type = "connector"
+        else:
+            raw_name = module[9:]
+            repo_type = "database"
 
     name = raw_name.replace("-", " ").capitalize()
-    url = repo.html_url
+    url = module.html_url if isRepo else "core"
     config = get_config_details(readme)
 
     if config:
@@ -123,9 +162,9 @@ def get_parsers_details():
 def validate_yaml_format(mapping, error_strict):
     """Scans the configuration format and validates yaml formatting."""
     try:
-        yaml.load(mapping["config"])
+        yaml.load(mapping["config"], Loader=yaml.FullLoader)
     except (KeyError, TypeError):
-        yaml.load(mapping)
+        yaml.load(mapping, Loader=yaml.FullLoader)
     except yaml.scanner.ScannerError as e:
         if error_strict:
             raise e
@@ -137,7 +176,9 @@ def validate_yaml_format(mapping, error_strict):
 
 def triage_modules(g, active_modules, error_strict=False):
     """Allocate modules to their type and active/inactive status."""
+    core_modules = get_core_modules()
     repos = get_repos()
+    repos += core_modules
 
     skills = {"commented": [], "uncommented": []}
     connectors = {"commented": [], "uncommented": []}
@@ -156,19 +197,27 @@ def triage_modules(g, active_modules, error_strict=False):
         params = get_config_params(repo, readme)
         validate_yaml_format(params, error_strict)
 
-        if params["repo_type"] == "skill":
-            if params["raw_name"] in active_modules:
-                skills["uncommented"].append(params)
-            skills["commented"].append(params)
-
-        elif params["repo_type"] == "connector":
-            if params["raw_name"] in active_modules:
-                connectors["uncommented"].append(params)
-            connectors["commented"].append(params)
+        if (isinstance(repo, Repository.Repository)) and (
+            params["repo_type"] + "-" + params["raw_name"] in core_modules
+        ):
+            pass
         else:
-            if params["raw_name"] == active_modules:
-                databases["uncommented"].append(params)
-            databases["commented"].append(params)
+            if params["repo_type"] == "skill":
+                if params["raw_name"] in active_modules:
+                    skills["uncommented"].append(params)
+                else:
+                    skills["commented"].append(params)
+
+            elif params["repo_type"] == "connector":
+                if params["raw_name"] in active_modules:
+                    connectors["uncommented"].append(params)
+                else:
+                    connectors["commented"].append(params)
+            else:
+                if params["raw_name"] == active_modules:
+                    databases["uncommented"].append(params)
+                else:
+                    databases["commented"].append(params)
 
     return modules
 
@@ -218,7 +267,7 @@ def update_config(g, active_modules, config_path, error_strict=False):
     if args.active_modules:
         active_modules.append((args.active_skills.split(",")))
     else:
-        active_modules = ["dance", "hello", "seen", "loudnoises", "websocket", "shell"]
+        active_modules = ["dance", "hello", "seen", "loudnoises", "websocket"]
     if not args.output:
         base_path = "/".join(os.path.realpath(__file__).split("/")[:-3])
         config_path = base_path


Test patch: