Repository: Cog-Creators/Red-DiscordBot

Issue:
Trivia should have some validation checks for the trivia files
We should be checking the validity of the trivia list in `[p]trivia` command and `[p]triviaset custom upload` command. I don't know how much checks we need (could use schema verification, although that can result in poor errors sometimes), but our format looks like this (written in a form of a dictionary):
```py
# AUTHOR and CONFIG are special keys that are optional
{
    "AUTHOR": "some string",  # type: str
    "CONFIG": {
        # all keys in here are optional
        "max_score": 123,  # type: int
        "timeout": 123.0,  # type: float
        "delay": 123.0,  # type: float
        "bot_plays": False,  # type: bool
        "reveal_answer": False,  # type: bool
        "payout_multiplier": 123.0,  # type: float
    },
    "question 1 (string)": [
        # list of answers; can be a combination of any of the types below
        "answer 1",  # type: str
        123,  # type: int (gets implicitly converted by Trivia)
        True,  # type: bool (gets implicitly converted by Trivia;
               # this one can be a source of issues,
               # but it probably needs to remain to avoid breakage,
               # at least until next minor version)
        1.69,  # type: float (gets implicitly converted by Trivia;
               # this one can be a source of issues,
               # but it probably needs to remain to avoid breakage,
               # at least until next minor version)
    ],
}
```

As an example, this is one of the errors that can currently occur when starting the trivia with an invalid format, that happens to be valid YAML.
```
Exception in command 'trivia'
Traceback (most recent call last):
  File "/Users/choure/redenv/lib/python3.8/site-packages/discord/ext/commands/core.py", line 85, in wrapped
    ret = await coro(*args, **kwargs)
  File "/Users/choure/redenv/lib/python3.8/site-packages/redbot/cogs/trivia/trivia.py", line 320, in trivia
    trivia_dict.update(dict_)
ValueError: dictionary update sequence element #0 has length 1; 2 is required
```


Discussion:
Hi there, I am new to this project and I am interested in taking on this issue. My thinking is that JSON schema validation has a good chance of solving this. What are your concerns? 
> Hi there, I am new to this project and I am interested in taking on this issue. My thinking is that JSON schema validation has a good chance of solving this. What are your concerns?

The issue with json is that yaml was picked cause of user accessibility to be able to create them without any programming experience. Changing to json would make the gap to creating trivia's much harder as we don't have a way to interactively create them. 
My bad, I didn't mean changing from YAML to JSON I just meant that after parsing the YAML we could use a JSON schema validator such as this one (https://github.com/Julian/jsonschema) to verify that the YAML follows the desired format. 
I've found that some trivia lists (ex. hockey.yaml:1353) has a float listed as one of the answers. Should this be acceptable too? 
@leblancg It's the same situation as with `bool` type - it should be allowed (at least for now) as disallowing it would be a breaking change since it may break existing trivia lists (such as the hockey one).

I updated the issue, good catch!

Resolving patch:
diff --git a/redbot/cogs/trivia/trivia.py b/redbot/cogs/trivia/trivia.py
--- a/redbot/cogs/trivia/trivia.py
+++ b/redbot/cogs/trivia/trivia.py
@@ -3,7 +3,8 @@
 import math
 import pathlib
 from collections import Counter
-from typing import List, Literal
+from typing import Any, Dict, List, Literal
+from schema import Schema, Optional, Or, SchemaError
 
 import io
 import yaml
@@ -23,9 +24,23 @@
 from .log import LOG
 from .session import TriviaSession
 
-__all__ = ["Trivia", "UNIQUE_ID", "get_core_lists"]
+__all__ = ("Trivia", "UNIQUE_ID", "InvalidListError", "get_core_lists", "get_list")
 
 UNIQUE_ID = 0xB3C0E453
+TRIVIA_LIST_SCHEMA = Schema(
+    {
+        Optional("AUTHOR"): str,
+        Optional("CONFIG"): {
+            Optional("max_score"): int,
+            Optional("timeout"): Or(int, float),
+            Optional("delay"): Or(int, float),
+            Optional("bot_plays"): bool,
+            Optional("reveal_answer"): bool,
+            Optional("payout_multiplier"): Or(int, float),
+        },
+        str: [str, int, bool, float],
+    }
+)
 
 _ = Translator("Trivia", __file__)
 
@@ -267,6 +282,13 @@ async def trivia_upload(self, ctx: commands.Context):
                 _("There was an error parsing the trivia list. See logs for more info.")
             )
             LOG.exception("Custom Trivia file %s failed to upload", parsedfile.filename)
+        except SchemaError as e:
+            await ctx.send(
+                _(
+                    "The custom trivia list was not saved."
+                    " The file does not follow the proper data format.\n{schema_error}"
+                ).format(schema_error=box(e))
+            )
 
     @commands.is_owner()
     @triviaset_custom.command(name="delete", aliases=["remove"])
@@ -603,13 +625,7 @@ def get_trivia_list(self, category: str) -> dict:
         except StopIteration:
             raise FileNotFoundError("Could not find the `{}` category.".format(category))
 
-        with path.open(encoding="utf-8") as file:
-            try:
-                dict_ = yaml.safe_load(file)
-            except yaml.error.YAMLError as exc:
-                raise InvalidListError("YAML parsing failed.") from exc
-            else:
-                return dict_
+        return get_list(path)
 
     async def _save_trivia_list(
         self, ctx: commands.Context, attachment: discord.Attachment
@@ -671,9 +687,10 @@ async def _save_trivia_list(
                 return
 
         buffer = io.BytesIO(await attachment.read())
-        yaml.safe_load(buffer)
-        buffer.seek(0)
+        trivia_dict = yaml.safe_load(buffer)
+        TRIVIA_LIST_SCHEMA.validate(trivia_dict)
 
+        buffer.seek(0)
         with file.open("wb") as fp:
             fp.write(buffer.read())
         await ctx.send(_("Saved Trivia list as {filename}.").format(filename=filename))
@@ -697,3 +714,27 @@ def get_core_lists() -> List[pathlib.Path]:
     """Return a list of paths for all trivia lists packaged with the bot."""
     core_lists_path = pathlib.Path(__file__).parent.resolve() / "data/lists"
     return list(core_lists_path.glob("*.yaml"))
+
+
+def get_list(path: pathlib.Path) -> Dict[str, Any]:
+    """
+    Returns a trivia list dictionary from the given path.
+
+    Raises
+    ------
+    InvalidListError
+        Parsing of list's YAML file failed.
+    SchemaError
+        The list does not adhere to the schema.
+    """
+    with path.open(encoding="utf-8") as file:
+        try:
+            trivia_dict = yaml.safe_load(file)
+        except yaml.error.YAMLError as exc:
+            raise InvalidListError("YAML parsing failed.") from exc
+
+    try:
+        TRIVIA_LIST_SCHEMA.validate(trivia_dict)
+    except SchemaError as exc:
+        raise InvalidListError("The list does not adhere to the schema.") from exc
+    return trivia_dict


Test patch:
diff --git a/tests/cogs/test_trivia.py b/tests/cogs/test_trivia.py
--- a/tests/cogs/test_trivia.py
+++ b/tests/cogs/test_trivia.py
@@ -1,33 +1,27 @@
+import textwrap
+
 import yaml
+from schema import SchemaError
 
 
 def test_trivia_lists():
-    from redbot.cogs.trivia import get_core_lists
+    from redbot.cogs.trivia import InvalidListError, get_core_lists, get_list
 
     list_names = get_core_lists()
     assert list_names
     problem_lists = []
     for l in list_names:
-        with l.open(encoding="utf-8") as f:
-            try:
-                dict_ = yaml.safe_load(f)
-            except yaml.error.YAMLError as e:
-                problem_lists.append((l.stem, "YAML error:\n{!s}".format(e)))
+        try:
+            get_list(l)
+        except InvalidListError as exc:
+            e = exc.__cause__
+            if isinstance(e, SchemaError):
+                problem_lists.append((l.stem, f"SCHEMA error:\n{e!s}"))
             else:
-                for key in list(dict_.keys()):
-                    if key == "CONFIG":
-                        if not isinstance(dict_[key], dict):
-                            problem_lists.append((l.stem, "CONFIG is not a dict"))
-                    elif key == "AUTHOR":
-                        if not isinstance(dict_[key], str):
-                            problem_lists.append((l.stem, "AUTHOR is not a string"))
-                    else:
-                        if not isinstance(dict_[key], list):
-                            problem_lists.append(
-                                (l.stem, "The answers for '{}' are not a list".format(key))
-                            )
+                problem_lists.append((l.stem, f"YAML error:\n{e!s}"))
+
     if problem_lists:
         msg = ""
-        for l in problem_lists:
-            msg += "{}: {}\n".format(l[0], l[1])
+        for name, error in problem_lists:
+            msg += f"- {name}:\n{textwrap.indent(error, '    ')}"
         raise TypeError("The following lists contain errors:\n" + msg)