Repository: repobee/repobee

Issue:
Refactor how default plugins are resolved
In implementing the "installed" version of RepoBee, there's a need to provide default plugins that are only loaded when RepoBee is actually installed. In other words, there are default plugins, and kind of default plugins. This can be resolved statically, but it's easier to resolve it at runtime, and also makes for easier testing. Here's the changes that need to be made.

* Move all default plugins int `src/_repobee/ext/defaults`
* Create a function `get_default_plugins(include_dist_plugins: bool)` in `plugin.py`
* Put future dist plugins into `src/_repobee/ext/dist`

Another benefit of this is that it won't be possible for users to specify a default plugin with thi `-p` option, as it does not look in subdirectories.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/repobee/repobee/pull/452?src=pr&el=h1) Report
> Merging [#452](https://codecov.io/gh/repobee/repobee/pull/452?src=pr&el=desc) into [master](https://codecov.io/gh/repobee/repobee/commit/6a150d68d4b1468155bb2ec7d14bbeb276bfe173&el=desc) will **decrease** coverage by `4.00%`.
> The diff coverage is `91.30%`.

[![Impacted file tree graph](https://codecov.io/gh/repobee/repobee/pull/452/graphs/tree.svg?width=650&height=150&src=pr&token=5vOKgkphZe)](https://codecov.io/gh/repobee/repobee/pull/452?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #452      +/-   ##
==========================================
- Coverage   92.69%   88.69%   -4.01%     
==========================================
  Files          40       40              
  Lines        2164     2158       -6     
  Branches      369      373       +4     
==========================================
- Hits         2006     1914      -92     
- Misses        125      208      +83     
- Partials       33       36       +3     
```


| [Impacted Files](https://codecov.io/gh/repobee/repobee/pull/452?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/\_repobee/constants.py](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree#diff-c3JjL19yZXBvYmVlL2NvbnN0YW50cy5weQ==) | `100.00% <ø> (ø)` | |
| [src/\_repobee/ext/defaults/configwizard.py](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree#diff-c3JjL19yZXBvYmVlL2V4dC9kZWZhdWx0cy9jb25maWd3aXphcmQucHk=) | `100.00% <ø> (ø)` | |
| [src/\_repobee/ext/defaults/genreviews.py](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree#diff-c3JjL19yZXBvYmVlL2V4dC9kZWZhdWx0cy9nZW5yZXZpZXdzLnB5) | `91.66% <ø> (ø)` | |
| [src/\_repobee/ext/defaults/github.py](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree#diff-c3JjL19yZXBvYmVlL2V4dC9kZWZhdWx0cy9naXRodWIucHk=) | `90.29% <ø> (ø)` | |
| [src/\_repobee/plugin.py](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree#diff-c3JjL19yZXBvYmVlL3BsdWdpbi5weQ==) | `92.18% <85.71%> (-1.09%)` | :arrow_down: |
| [src/\_repobee/cli/mainparser.py](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree#diff-c3JjL19yZXBvYmVlL2NsaS9tYWlucGFyc2VyLnB5) | `96.44% <100.00%> (+0.02%)` | :arrow_up: |
| [src/\_repobee/ext/gitlab.py](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree#diff-c3JjL19yZXBvYmVlL2V4dC9naXRsYWIucHk=) | `63.86% <100.00%> (-28.56%)` | :arrow_down: |
| [src/\_repobee/main.py](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree#diff-c3JjL19yZXBvYmVlL21haW4ucHk=) | `91.66% <100.00%> (+3.91%)` | :arrow_up: |
| [src/\_repobee/git.py](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree#diff-c3JjL19yZXBvYmVlL2dpdC5weQ==) | `96.62% <0.00%> (-3.38%)` | :arrow_down: |
| ... and [10 more](https://codecov.io/gh/repobee/repobee/pull/452/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/repobee/repobee/pull/452?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/repobee/repobee/pull/452?src=pr&el=footer). Last update [6a150d6...427cfc3](https://codecov.io/gh/repobee/repobee/pull/452?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/docs/code.rst b/docs/code.rst
index e0a8edf..dbae3b5 100644
--- a/docs/code.rst
+++ b/docs/code.rst
@@ -59,39 +59,39 @@ util
     :members:
 
 
-Core plugins
-============
+Default plugins
+===============
+These plugins are loaded by default.
 
-defaults
---------
+configwizard
+------------
+
+.. automodule:: _repobee.ext.defaults.configwizard
+    :members:
 
-.. automodule:: _repobee.ext.defaults
+genreviews
+----------
+
+.. automodule:: _repobee.ext.defaults.genreviews
     :members:
 
+
 github
 ------
 
-.. automodule:: _repobee.ext.github
+.. automodule:: _repobee.ext.defaults.github
     :members:
 
+Additional plugins
+==================
+These plugins ship with RepoBee, but are not loaded by default.
+
 gitlab
 ------
 
 .. automodule:: _repobee.ext.gitlab
    :members:
 
-
-
-pairwise
---------
-
-.. automodule:: _repobee.ext.pairwise
-    :members:
-
-
-Extension plugins
-=================
-
 javac
 -----
 
@@ -104,11 +104,12 @@ pylint
 .. automodule:: _repobee.ext.pylint
     :members:
 
-config-wizard
--------------
 
-.. automodule:: _repobee.ext.configwizard
-   :members:
+pairwise
+--------
+
+.. automodule:: _repobee.ext.pairwise
+    :members:
 
 query
 -----
diff --git a/src/_repobee/cli/mainparser.py b/src/_repobee/cli/mainparser.py
index 2523bd1..a2e2668 100644
--- a/src/_repobee/cli/mainparser.py
+++ b/src/_repobee/cli/mainparser.py
@@ -122,7 +122,7 @@ def create_parser_for_docs() -> argparse.ArgumentParser:
     """
     daiquiri.setup(level=logging.FATAL)
     # load default plugins
-    plugin.initialize_plugins([_repobee.constants.DEFAULT_PLUGIN])
+    plugin.initialize_default_plugins()
     ext_commands = plug.manager.hook.create_extension_command()
     return create_parser(
         show_all_opts=True,
@@ -158,6 +158,7 @@ def create_parser(
             _versioned_plugin_name(p)
             for p in plug.manager.get_plugins()
             if isinstance(p, types.ModuleType)
+            and not plugin.is_default_plugin(p)
         ]
     )
 
diff --git a/src/_repobee/constants.py b/src/_repobee/constants.py
index 0b1a1a9..5a55a27 100644
--- a/src/_repobee/constants.py
+++ b/src/_repobee/constants.py
@@ -40,5 +40,3 @@ CONFIGURABLE_ARGS = set(ORDERED_CONFIGURABLE_ARGS)
 
 TOKEN_ENV = "REPOBEE_TOKEN"
 TOKEN_ENV_OLD = "REPOBEE_OAUTH"
-
-DEFAULT_PLUGIN = "defaults"
diff --git a/src/_repobee/ext/defaults/__init__.py b/src/_repobee/ext/defaults/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/src/_repobee/ext/configwizard.py b/src/_repobee/ext/defaults/configwizard.py
similarity index 100%
rename from src/_repobee/ext/configwizard.py
rename to src/_repobee/ext/defaults/configwizard.py
diff --git a/src/_repobee/ext/defaults.py b/src/_repobee/ext/defaults/genreviews.py
similarity index 66%
rename from src/_repobee/ext/defaults.py
rename to src/_repobee/ext/defaults/genreviews.py
index 4ba1c9b..797d4ff 100644
--- a/src/_repobee/ext/defaults.py
+++ b/src/_repobee/ext/defaults/genreviews.py
@@ -1,17 +1,7 @@
-"""The defaults plugin contains all default hook implementations.
+"""The default implementation of generate_review_allocations.
 
-The goal is to make core parts of repobee pluggable using hooks that only
-return the first result that is not None. The standard behavior will be
-provided by the default plugin (this one), which implements all of the required
-hooks.  The default plugin will always be run last, so any user-defined hooks
-will run before it and therefore effectively override the default hooks.
-
-Currently, only the peer review related generate_review_allocations hook has a
-default implementation.
-
-.. module:: defaults
-    :synopsis: Plugin that provides the default behavior for core repobee
-        functionality.
+.. module:: genreviews
+    :synopsis: The default implementation of generate_review_allocations
 
 .. moduleauthor:: Simon Larsén
 """
@@ -21,10 +11,6 @@ from typing import List
 
 import repobee_plug as plug
 
-# import other default plugins
-from _repobee.ext.github import DefaultAPIHooks  # noqa: F401
-from _repobee.ext.configwizard import create_extension_command  # noqa: F401
-
 
 @plug.repobee_hook
 def generate_review_allocations(
diff --git a/src/_repobee/ext/github.py b/src/_repobee/ext/defaults/github.py
similarity index 100%
rename from src/_repobee/ext/github.py
rename to src/_repobee/ext/defaults/github.py
diff --git a/src/_repobee/ext/gitlab.py b/src/_repobee/ext/gitlab.py
index 8bad8fd..a1059a2 100644
--- a/src/_repobee/ext/gitlab.py
+++ b/src/_repobee/ext/gitlab.py
@@ -25,7 +25,7 @@ import requests.exceptions
 import repobee_plug as plug
 
 from _repobee import exception
-from _repobee.ext.github import DEFAULT_REVIEW_ISSUE
+from _repobee.ext.defaults.github import DEFAULT_REVIEW_ISSUE
 
 LOGGER = daiquiri.getLogger(__file__)
 
diff --git a/src/_repobee/main.py b/src/_repobee/main.py
index fe9a164..8b994a6 100644
--- a/src/_repobee/main.py
+++ b/src/_repobee/main.py
@@ -17,7 +17,6 @@ import _repobee.cli.parsing
 import _repobee.cli.preparser
 from _repobee import plugin
 from _repobee import exception
-from _repobee import constants
 from _repobee import config
 from _repobee.cli.preparser import separate_args
 
@@ -46,19 +45,19 @@ def main(sys_args: List[str]):
         )
         config_file = parsed_preparser_args.config_file
 
-        if parsed_preparser_args.no_plugins:
-            LOGGER.info("Non-default plugins disabled")
-            plugin.initialize_plugins([constants.DEFAULT_PLUGIN])
-        else:
+        # IMPORTANT: the default plugins must be loaded before user-defined
+        # plugins to ensure that the user-defined plugins override the defaults
+        # in firstresult hooks
+        LOGGER.debug("Initializing default plugins")
+        plugin.initialize_default_plugins()
+
+        if not parsed_preparser_args.no_plugins:
+            LOGGER.debug("Initializing user plugins")
             plugin_names = (
                 parsed_preparser_args.plug
                 or config.get_plugin_names(config_file)
             ) or []
-            # IMPORTANT: the default plugin MUST be loaded last to ensure that
-            # any user-defined plugins override the firstresult hooks
-            plugin.initialize_plugins(
-                plugin_names + [constants.DEFAULT_PLUGIN]
-            )
+            plugin.initialize_plugins(plugin_names)
 
         config.execute_config_hooks(config_file)
         ext_commands = plug.manager.hook.create_extension_command()
diff --git a/src/_repobee/plugin.py b/src/_repobee/plugin.py
index ee2d5de..4133e7e 100644
--- a/src/_repobee/plugin.py
+++ b/src/_repobee/plugin.py
@@ -11,8 +11,7 @@ import collections
 import contextlib
 import shutil
 import tempfile
-
-import types
+import pkgutil
 import pathlib
 import importlib
 from types import ModuleType
@@ -21,6 +20,7 @@ from typing import List, Optional, Iterable, Mapping, Union
 import daiquiri
 
 import _repobee
+import _repobee.ext.defaults
 from _repobee import exception
 
 import repobee_plug as plug
@@ -38,7 +38,9 @@ def _external_plugin_qualname(plugin_name):
     )
 
 
-def load_plugin_modules(plugin_names: Iterable[str]) -> List[ModuleType]:
+def load_plugin_modules(
+    plugin_names: Iterable[str], allow_qualified: bool = False
+) -> List[ModuleType]:
     """Load the given plugins. Plugins are loaded such that they are executed
     in the same order that they are specified in the plugin_names list.
 
@@ -54,9 +56,18 @@ def load_plugin_modules(plugin_names: Iterable[str]) -> List[ModuleType]:
         # import nr 2
         from repobee_javac import javac
 
-    Args:
-        plugin_names: A list of plugin names. Overrides the config file.
+    If ``allow_qualified`` is True, an additional import using the provided
+    plugin names as-is is also attempted.
+
+    .. code-block:: python
+
+        # import nr 3 (only if allow_qualified)
+        import javac
 
+    Args:
+        plugin_names: A list of plugin names.
+        allow_qualified: If True, attempts to import modules using the plugin
+            names as qualified paths.
     Returns:
         a list of loaded modules.
     """
@@ -64,9 +75,11 @@ def load_plugin_modules(plugin_names: Iterable[str]) -> List[ModuleType]:
     LOGGER.debug("Loading plugins: " + ", ".join(plugin_names))
 
     for name in plugin_names:
-        plug_mod = _try_load_module(
-            _plugin_qualname(name)
-        ) or _try_load_module(_external_plugin_qualname(name))
+        plug_mod = (
+            _try_load_module(_plugin_qualname(name))
+            or _try_load_module(_external_plugin_qualname(name))
+            or (allow_qualified and _try_load_module(name))
+        )
         if not plug_mod:
             msg = "failed to load plugin module " + name
             raise exception.PluginLoadError(msg)
@@ -203,20 +216,26 @@ def try_register_plugin(
 
 
 def initialize_plugins(
-    plugin_names: List[str] = None,
+    plugin_names: List[str] = None, allow_qualified: bool = False,
 ) -> List[Union[ModuleType, type]]:
     """Load and register plugins.
 
     Args:
         plugin_names: An optional list of plugin names that overrides the
-        configuration file's plugins.
+            configuration file's plugins.
+        allow_qualified: Allows the plugin names to be qualified.
     Returns:
         A list of registered modules and classes.
+    Raises:
+        :py:class:`_repobee.exception.PluginLoadError`
     """
+    if not allow_qualified:
+        _check_no_qualified_names(plugin_names)
+
     registered_plugins = plug.manager.get_plugins()
     plug_modules = [
         p
-        for p in load_plugin_modules(plugin_names=plugin_names)
+        for p in load_plugin_modules(plugin_names, allow_qualified)
         if p not in registered_plugins
     ]
     registered = register_plugins(plug_modules)
@@ -224,7 +243,15 @@ def initialize_plugins(
     return registered
 
 
-def resolve_plugin_version(plugin_module: types.ModuleType,) -> Optional[str]:
+def _check_no_qualified_names(names: List[str]):
+    qualified_names = [name for name in names if "." in name]
+    if qualified_names:
+        raise exception.PluginLoadError(
+            f"Qualified names not allowed: {qualified_names}"
+        )
+
+
+def resolve_plugin_version(plugin_module: ModuleType) -> Optional[str]:
     """Return the version of the top-level package containing the plugin, or
     None if it is not defined.
     """
@@ -235,6 +262,53 @@ def resolve_plugin_version(plugin_module: types.ModuleType,) -> Optional[str]:
     )
 
 
+def is_default_plugin(module: ModuleType) -> Optional[str]:
+    """Check if the provided module is a default module.
+
+    Args:
+        module: A Python module.
+    Returns:
+        True iff the provided module is a default plugin.
+    """
+    return module.__package__ == _repobee.ext.defaults.__name__
+
+
+def initialize_default_plugins() -> None:
+    """Initialize the default plugin modules."""
+    default_plugin_qualnames = get_qualified_module_names(
+        _repobee.ext.defaults
+    )
+    initialize_plugins(default_plugin_qualnames, allow_qualified=True)
+
+
+def get_qualified_module_names(pkg: ModuleType) -> List[str]:
+    """Return a list of all python modules in the given package. Only considers
+    the modules directly in this package, and not in subpackages.
+
+    Args:
+        pkg: The package to resolve modules in.
+    Returns:
+        All modules in the given package.
+    """
+    return [f"{pkg.__name__}.{name}" for name in get_module_names(pkg)]
+
+
+def get_module_names(pkg: ModuleType) -> List[str]:
+    """Get the unqualified module names from the given package.
+
+    Args:
+        pkg: The package to resolve modules in.
+    Returns:
+        All modules in the given package.
+    """
+    return [
+        name
+        for file_finder, name, _ in pkgutil.iter_modules(pkg.__path__)
+        # only include modules (i.e. files), not subpackages
+        if (pathlib.Path(file_finder.path) / (name + ".py")).is_file()
+    ]
+
+
 def execute_clone_tasks(
     repo_names: List[str], api: plug.API, cwd: Optional[pathlib.Path] = None
 ) -> Mapping[str, List[plug.Result]]:


Test patch:
diff --git a/tests/unit_tests/repobee/conftest.py b/tests/unit_tests/repobee/conftest.py
index 52c8a11..8d004ed 100644
--- a/tests/unit_tests/repobee/conftest.py
+++ b/tests/unit_tests/repobee/conftest.py
@@ -199,3 +199,14 @@ def config_mock(empty_config_mock, students_file):
     )
     empty_config_mock.write(config_contents)
     yield empty_config_mock
+
+
+@pytest.fixture
+def load_default_plugins():
+    """Load the default plugins."""
+    default_plugin_names = _repobee.plugin.get_qualified_module_names(
+        _repobee.ext.defaults
+    )
+    _repobee.plugin.initialize_plugins(
+        default_plugin_names, allow_qualified=True
+    )
diff --git a/tests/unit_tests/repobee/plugin_tests/test_configwizard.py b/tests/unit_tests/repobee/plugin_tests/test_configwizard.py
index c748f9a..fe7f5ce 100644
--- a/tests/unit_tests/repobee/plugin_tests/test_configwizard.py
+++ b/tests/unit_tests/repobee/plugin_tests/test_configwizard.py
@@ -9,7 +9,7 @@ from unittest.mock import patch
 import pytest
 
 import _repobee.constants
-from _repobee.ext import configwizard
+from _repobee.ext.defaults import configwizard
 
 
 @pytest.fixture
diff --git a/tests/unit_tests/repobee/plugin_tests/test_defaults.py b/tests/unit_tests/repobee/plugin_tests/test_genreviews.py
similarity index 83%
rename from tests/unit_tests/repobee/plugin_tests/test_defaults.py
rename to tests/unit_tests/repobee/plugin_tests/test_genreviews.py
index b1eb76f..ef9560a 100644
--- a/tests/unit_tests/repobee/plugin_tests/test_defaults.py
+++ b/tests/unit_tests/repobee/plugin_tests/test_genreviews.py
@@ -1,10 +1,10 @@
-"""Tests for the defaults plugin."""
+"""Tests for the genreviews plugin."""
 import itertools
 import collections
 
 import pytest
 
-from _repobee.ext import defaults
+from _repobee.ext.defaults import genreviews
 
 import constants
 
@@ -16,7 +16,7 @@ class TestGenerateReviewAllocations:
     def test_raises_when_too_many_reviews(self, num_students, num_reviews):
         teams = list(constants.STUDENTS[:num_students])
         with pytest.raises(ValueError) as exc_info:
-            defaults.generate_review_allocations(teams, num_reviews)
+            genreviews.generate_review_allocations(teams, num_reviews)
 
         assert "num_reviews must be less than len(teams)" in str(
             exc_info.value
@@ -24,7 +24,7 @@ class TestGenerateReviewAllocations:
 
     def test_raises_when_too_few_reviews(self):
         with pytest.raises(ValueError) as exc_info:
-            defaults.generate_review_allocations(list(constants.STUDENTS), 0)
+            genreviews.generate_review_allocations(list(constants.STUDENTS), 0)
         assert "num_reviews must be greater than 0" in str(exc_info.value)
 
     @pytest.mark.parametrize(
@@ -37,7 +37,7 @@ class TestGenerateReviewAllocations:
         students = list(constants.STUDENTS[:num_students])
         assert len(students) == num_students, "pre-test assert"
 
-        allocations = defaults.generate_review_allocations(
+        allocations = genreviews.generate_review_allocations(
             students, num_reviews
         )
 
@@ -60,7 +60,9 @@ class TestGenerateReviewAllocations:
         teams = list(constants.STUDENTS[:num_students])
         expected_reviewed_teams = list(teams)
 
-        allocations = defaults.generate_review_allocations(teams, num_reviews)
+        allocations = genreviews.generate_review_allocations(
+            teams, num_reviews
+        )
 
         assert sorted(expected_reviewed_teams) == sorted(
             [alloc.reviewed_team for alloc in allocations]
diff --git a/tests/unit_tests/repobee/plugin_tests/test_github.py b/tests/unit_tests/repobee/plugin_tests/test_github.py
index 84ead19..a8a6b47 100644
--- a/tests/unit_tests/repobee/plugin_tests/test_github.py
+++ b/tests/unit_tests/repobee/plugin_tests/test_github.py
@@ -7,11 +7,9 @@ import github
 
 import repobee_plug as plug
 
-import _repobee
-import _repobee.ext
-import _repobee.ext.github
+import _repobee.ext.defaults.github as github_plugin
 from _repobee import exception
-from _repobee.ext.github import REQUIRED_TOKEN_SCOPES
+from _repobee.ext.defaults.github import REQUIRED_TOKEN_SCOPES
 
 import constants
 import functions
@@ -314,20 +312,20 @@ def issues(repos):
 
 @pytest.fixture(scope="function")
 def api(happy_github, organization, no_teams):
-    return _repobee.ext.github.GitHubAPI(BASE_URL, TOKEN, ORG_NAME, USER)
+    return github_plugin.GitHubAPI(BASE_URL, TOKEN, ORG_NAME, USER)
 
 
 class TestInit:
     def test_raises_on_empty_user_arg(self):
         with pytest.raises(TypeError) as exc_info:
-            _repobee.ext.github.GitHubAPI(BASE_URL, TOKEN, ORG_NAME, "")
+            github_plugin.GitHubAPI(BASE_URL, TOKEN, ORG_NAME, "")
 
         assert "argument 'user' must not be empty" in str(exc_info.value)
 
     @pytest.mark.parametrize("url", ["https://github.com", constants.HOST_URL])
     def test_raises_when_url_is_bad(self, url):
         with pytest.raises(plug.PlugError) as exc_info:
-            _repobee.ext.github.GitHubAPI(url, TOKEN, ORG_NAME, USER)
+            github_plugin.GitHubAPI(url, TOKEN, ORG_NAME, USER)
 
         assert (
             "invalid base url, should either be https://api.github.com or "
@@ -338,7 +336,7 @@ class TestInit:
         "url", ["https://api.github.com", constants.BASE_URL]
     )
     def test_accepts_valid_urls(self, url):
-        api = _repobee.ext.github.GitHubAPI(url, TOKEN, ORG_NAME, USER)
+        api = github_plugin.GitHubAPI(url, TOKEN, ORG_NAME, USER)
 
         assert isinstance(api, plug.API)
 
@@ -756,7 +754,7 @@ def team_to_repos(api, no_repos, organization):
 class TestAddReposToReviewTeams:
     def test_with_default_issue(self, team_to_repos, organization, api):
         num_teams = len(team_to_repos)
-        default_issue = _repobee.ext.github.DEFAULT_REVIEW_ISSUE
+        default_issue = github_plugin.DEFAULT_REVIEW_ISSUE
         assert num_teams, "pre-test assert"
         team_repo_tuples = [
             (team, *repos) for team, repos in team_to_repos.items()
@@ -829,7 +827,7 @@ class TestVerifySettings:
 
     def test_happy_path(self, happy_github, organization, api):
         """Tests that no exceptions are raised when all info is correct."""
-        _repobee.ext.github.GitHubAPI.verify_settings(
+        github_plugin.GitHubAPI.verify_settings(
             USER, ORG_NAME, BASE_URL, TOKEN
         )
 
@@ -837,7 +835,7 @@ class TestVerifySettings:
         self, happy_github, monkeypatch, api
     ):
         with pytest.raises(exception.BadCredentials) as exc_info:
-            _repobee.ext.github.GitHubAPI.verify_settings(
+            github_plugin.GitHubAPI.verify_settings(
                 USER, ORG_NAME, BASE_URL, ""
             )
 
@@ -845,7 +843,7 @@ class TestVerifySettings:
 
     def test_incorrect_info_raises_not_found_error(self, github_bad_info, api):
         with pytest.raises(exception.NotFoundError):
-            _repobee.ext.github.GitHubAPI.verify_settings(
+            github_plugin.GitHubAPI.verify_settings(
                 USER, ORG_NAME, BASE_URL, TOKEN
             )
 
@@ -853,14 +851,14 @@ class TestVerifySettings:
         type(happy_github).oauth_scopes = PropertyMock(return_value=["repo"])
 
         with pytest.raises(exception.BadCredentials) as exc_info:
-            _repobee.ext.github.GitHubAPI.verify_settings(
+            github_plugin.GitHubAPI.verify_settings(
                 USER, ORG_NAME, BASE_URL, TOKEN
             )
         assert "missing one or more access token scopes" in str(exc_info.value)
 
     def test_not_owner_raises(self, happy_github, organization, api):
         with pytest.raises(exception.BadCredentials) as exc_info:
-            _repobee.ext.github.GitHubAPI.verify_settings(
+            github_plugin.GitHubAPI.verify_settings(
                 NOT_OWNER, ORG_NAME, BASE_URL, TOKEN
             )
 
@@ -883,7 +881,7 @@ class TestVerifySettings:
         happy_github.get_user.side_effect = lambda _: User(login=None)
 
         with pytest.raises(exception.UnexpectedException) as exc_info:
-            _repobee.ext.github.GitHubAPI.verify_settings(
+            github_plugin.GitHubAPI.verify_settings(
                 USER, ORG_NAME, BASE_URL, TOKEN
             )
 
@@ -907,7 +905,7 @@ class TestVerifySettings:
         ]
 
         with pytest.raises(exception.UnexpectedException) as exc_info:
-            _repobee.ext.github.GitHubAPI.verify_settings(
+            github_plugin.GitHubAPI.verify_settings(
                 USER, ORG_NAME, BASE_URL, TOKEN
             )
 
diff --git a/tests/unit_tests/repobee/test_cli.py b/tests/unit_tests/repobee/test_cli.py
index 21a64a2..f8479f3 100644
--- a/tests/unit_tests/repobee/test_cli.py
+++ b/tests/unit_tests/repobee/test_cli.py
@@ -13,7 +13,7 @@ import _repobee
 import _repobee.cli.dispatch
 import _repobee.cli.parsing
 import _repobee.ext
-import _repobee.ext.github
+import _repobee.ext.defaults.github
 import _repobee.ext.query
 import _repobee.constants
 import _repobee.plugin
@@ -77,7 +77,7 @@ VALID_PARSED_ARGS = dict(
 
 @pytest.fixture(autouse=True)
 def api_instance_mock():
-    instance_mock = mock.MagicMock(spec=_repobee.ext.github.GitHubAPI)
+    instance_mock = mock.MagicMock(spec=_repobee.ext.defaults.github.GitHubAPI)
 
     def _get_repo_urls(repo_names, org_name=None, teams=None):
         return [generate_repo_url(name, org_name) for name in repo_names]
@@ -91,18 +91,19 @@ def api_instance_mock():
 
 @pytest.fixture(autouse=True)
 def api_class_mock(mocker, api_instance_mock):
-    class_mock = mocker.patch("_repobee.ext.github.GitHubAPI", autospec=True)
+    class_mock = mocker.patch(
+        "_repobee.ext.defaults.github.GitHubAPI", autospec=True
+    )
     class_mock.return_value = api_instance_mock
     return class_mock
 
 
 @pytest.fixture(autouse=True)
-def load_default_plugins(api_instance_mock):
-    """Load the default plugins after mocking the GitHubAPI."""
-    loaded = _repobee.plugin.load_plugin_modules(
-        [_repobee.constants.DEFAULT_PLUGIN]
-    )
-    _repobee.plugin.register_plugins(loaded)
+def load_default_plugins_auto(load_default_plugins):
+    """The only point of this fixture is to make load_default_plugins
+    autouse=True.
+    """
+    yield
 
 
 @pytest.fixture
@@ -709,7 +710,7 @@ class TestExtensionCommands:
         command.
         """
         yield mock.MagicMock(
-            spec=_repobee.ext.configwizard.create_extension_command
+            spec=_repobee.ext.defaults.configwizard.create_extension_command
         )
 
     @pytest.fixture
diff --git a/tests/unit_tests/repobee/test_command.py b/tests/unit_tests/repobee/test_command.py
index f5185e8..1bd7849 100644
--- a/tests/unit_tests/repobee/test_command.py
+++ b/tests/unit_tests/repobee/test_command.py
@@ -7,7 +7,7 @@ from unittest.mock import patch, MagicMock, call, PropertyMock
 import pytest
 
 import _repobee
-import _repobee.ext.github
+import _repobee.ext.defaults.github
 from _repobee import command
 from _repobee import git
 from _repobee import util
@@ -147,8 +147,10 @@ def api_mock(request, mocker):
     def url_from_repo_info(repo_info):
         return generate_repo_url(repo_info.name)
 
-    mock = MagicMock(spec=_repobee.ext.github.GitHubAPI)
-    api_class = mocker.patch("_repobee.ext.github.GitHubAPI", autospec=True)
+    mock = MagicMock(spec=_repobee.ext.defaults.github.GitHubAPI)
+    api_class = mocker.patch(
+        "_repobee.ext.defaults.github.GitHubAPI", autospec=True
+    )
     api_class.return_value = mock
 
     mock.get_repo_urls.side_effect = partial(
@@ -158,7 +160,7 @@ def api_mock(request, mocker):
         map(url_from_repo_info, repo_infos)
     )
     mock.get_issues = MagicMock(
-        spec="_repobee.ext.github.GitHubAPI.get_issues",
+        spec="_repobee.ext.defaults.github.GitHubAPI.get_issues",
         side_effect=_get_issues,
     )
     type(mock).token = PropertyMock(return_value=TOKEN)
@@ -701,11 +703,11 @@ def test_purge_review_teams(master_names, students, api_mock):
 # TODO add more test cases
 class TestAssignPeerReviewers:
     @pytest.fixture(autouse=True)
-    def load_default_plugins(self):
-        modules = plugin.load_plugin_modules(
-            [_repobee.constants.DEFAULT_PLUGIN]
-        )
-        plugin.register_plugins(modules)
+    def load_default_plugins_auto(self, load_default_plugins):
+        """The only point of this fixture is to make load_default_plugins
+        autouse=True.
+        """
+        yield
 
     @pytest.mark.parametrize(
         "num_students, num_reviews",
diff --git a/tests/unit_tests/repobee/test_main.py b/tests/unit_tests/repobee/test_main.py
index 4b818ce..5056633 100644
--- a/tests/unit_tests/repobee/test_main.py
+++ b/tests/unit_tests/repobee/test_main.py
@@ -1,18 +1,19 @@
 import argparse
 import tempfile
 import pathlib
-from unittest.mock import MagicMock, patch
+from unittest.mock import MagicMock, patch, call
 from collections import namedtuple
 
 import pytest
 
 import _repobee.cli.preparser
+import _repobee.ext.defaults
 from functions import raise_
 import _repobee.constants
 from _repobee.cli import mainparser
 from _repobee import main
 from _repobee import plugin
-from _repobee.ext import configwizard
+from _repobee.ext.defaults import configwizard
 
 import constants
 
@@ -41,6 +42,7 @@ CLONE_ARGS = "clone --mn week-2 -s slarse".split()
 module = namedtuple("module", ("name",))
 
 DEFAULT_EXT_COMMANDS = [configwizard.create_extension_command()]
+DEFAULT_PLUGIN_NAMES = plugin.get_qualified_module_names(_repobee.ext.defaults)
 
 
 @pytest.fixture
@@ -55,7 +57,7 @@ def api_instance_mock(mocker):
 
 @pytest.fixture
 def init_plugins_mock(mocker):
-    def init_plugins(plugs=None):
+    def init_plugins(plugs=None, allow_qualified=False):
         list(map(module, plugs or []))
 
     return mocker.patch(
@@ -89,7 +91,6 @@ def dispatch_command_mock(mocker):
 def test_happy_path(
     api_instance_mock, handle_args_mock, dispatch_command_mock, no_config_mock
 ):
-
     sys_args = ["just some made up arguments".split()]
 
     main.main(sys_args)
@@ -156,8 +157,12 @@ def test_plugins_args(
 
     main.main(sys_args)
 
-    init_plugins_mock.assert_called_once_with(
-        ["javac", "pylint", _repobee.constants.DEFAULT_PLUGIN]
+    init_plugins_mock.assert_has_calls(
+        [
+            call(["javac", "pylint"]),
+            call(DEFAULT_PLUGIN_NAMES, allow_qualified=True),
+        ],
+        any_order=True,
     )
     handle_args_mock.assert_called_once_with(
         CLONE_ARGS,
@@ -178,7 +183,7 @@ def test_no_plugins_arg(
     main.main(sys_args)
 
     init_plugins_mock.assert_called_once_with(
-        [_repobee.constants.DEFAULT_PLUGIN]
+        DEFAULT_PLUGIN_NAMES, allow_qualified=True
     )
     handle_args_mock.assert_called_once_with(
         CLONE_ARGS,
@@ -199,7 +204,7 @@ def test_no_plugins_with_configured_plugins(
     main.main(sys_args)
 
     init_plugins_mock.assert_called_once_with(
-        [_repobee.constants.DEFAULT_PLUGIN]
+        DEFAULT_PLUGIN_NAMES, allow_qualified=True
     )
     handle_args_mock.assert_called_once_with(
         CLONE_ARGS,
@@ -216,8 +221,12 @@ def test_configured_plugins_are_loaded(
 
     main.main(sys_args)
 
-    init_plugins_mock.assert_called_once_with(
-        [*constants.PLUGINS, _repobee.constants.DEFAULT_PLUGIN]
+    init_plugins_mock.assert_has_calls(
+        [
+            call(["javac", "pylint"]),
+            call(DEFAULT_PLUGIN_NAMES, allow_qualified=True),
+        ],
+        any_order=True,
     )
     handle_args_mock.assert_called_once_with(
         CLONE_ARGS,
@@ -234,8 +243,12 @@ def test_plugin_with_subparser_name(
 
     main.main(sys_args)
 
-    init_plugins_mock.assert_called_once_with(
-        ["javac", "clone", _repobee.constants.DEFAULT_PLUGIN]
+    init_plugins_mock.assert_has_calls(
+        [
+            call(["javac", "clone"]),
+            call(DEFAULT_PLUGIN_NAMES, allow_qualified=True),
+        ],
+        any_order=True,
     )
     handle_args_mock.assert_called_once_with(
         CLONE_ARGS,
@@ -263,17 +276,19 @@ def test_invalid_plug_options(dispatch_command_mock, init_plugins_mock):
 
     Note that the default plugins must be loaded for this test to work.
     """
-    # load default plugins
-    loaded = plugin.load_plugin_modules([_repobee.constants.DEFAULT_PLUGIN])
-    plugin.register_plugins(loaded)
+    plugin.initialize_plugins(
+        plugin.get_qualified_module_names(_repobee.ext.defaults),
+        allow_qualified=True,
+    )
 
     sys_args = ["repobee", "-p", "javac", "-f", *CLONE_ARGS]
 
     with pytest.raises(SystemExit):
         main.main(sys_args)
 
-    init_plugins_mock.assert_called_once_with(
-        ["javac", _repobee.constants.DEFAULT_PLUGIN]
+    init_plugins_mock.assert_has_calls(
+        [call(["javac"]), call(DEFAULT_PLUGIN_NAMES, allow_qualified=True)],
+        any_order=True,
     )
     assert not dispatch_command_mock.called
 
diff --git a/tests/unit_tests/repobee/test_plugin.py b/tests/unit_tests/repobee/test_plugin.py
index 2e207f0..3a2f2ba 100644
--- a/tests/unit_tests/repobee/test_plugin.py
+++ b/tests/unit_tests/repobee/test_plugin.py
@@ -16,6 +16,7 @@ import pytest
 import repobee_plug as plug
 
 import _repobee.constants
+import _repobee.ext.defaults
 from _repobee import plugin
 from _repobee import exception
 
@@ -37,23 +38,34 @@ def unregister_plugins():
 class TestLoadPluginModules:
     """Tests for load_plugin_modules."""
 
-    def test_load_all_bundled_plugins(self):
-        """Test load the bundled plugins, i.e. the ones listed in
-        constants.PLUGINS.
-        """
-        plugin_names = [*PLUGINS, _repobee.constants.DEFAULT_PLUGIN]
-        plugin_qualnames = list(map(plugin._plugin_qualname, plugin_names))
+    def test_load_default_plugins(self):
+        default_plugin_qualnames = plugin.get_qualified_module_names(
+            _repobee.ext.defaults
+        )
+
+        modules = plugin.load_plugin_modules(
+            default_plugin_qualnames, allow_qualified=True
+        )
 
-        modules = plugin.load_plugin_modules(plugin_names)
         module_names = [mod.__name__ for mod in modules]
+        assert module_names == default_plugin_qualnames
 
-        assert module_names == plugin_qualnames
+    def test_load_bundled_plugins(self):
+        """Test load the bundled plugins that are not default plugins."""
+        bundled_plugin_qualnames = plugin.get_qualified_module_names(
+            _repobee.ext
+        )
+        bundled_plugin_names = plugin.get_module_names(_repobee.ext)
+
+        modules = plugin.load_plugin_modules(bundled_plugin_names)
+
+        module_names = [mod.__name__ for mod in modules]
+        assert module_names == bundled_plugin_qualnames
 
     def test_load_no_plugins(self, no_config_mock):
         """Test calling load plugins when no plugins are specified results in
         no plugins being loaded."""
         modules = plugin.load_plugin_modules([])
-
         assert modules == []
 
     def test_raises_when_loading_invalid_module(self, empty_config_mock):
@@ -69,6 +81,19 @@ class TestLoadPluginModules:
             exc_info.value
         )
 
+    def test_raises_when_loading_default_plugins_without_allow_qualified(self):
+        """Default plugins can only be loaded by their qualified names, and it
+        should only be allowed if allow_qualify is True.
+        """
+        default_plugin_qualnames = plugin.get_qualified_module_names(
+            _repobee.ext.defaults
+        )
+
+        with pytest.raises(exception.PluginLoadError) as exc_info:
+            plugin.load_plugin_modules(default_plugin_qualnames)
+
+        assert "failed to load plugin module" in str(exc_info.value)
+
 
 class TestRegisterPlugins:
     """Tests for register_plugins."""
@@ -214,3 +239,10 @@ class TestInitializePlugins:
             plugin.initialize_plugins([mod_name])
 
         assert warning_mock.called
+
+    def test_raises_on_qualified_names_by_default(self):
+        qualname = "_repobee.ext.query"
+        with pytest.raises(exception.PluginLoadError) as exc_info:
+            plugin.initialize_plugins([qualname])
+
+        assert "Qualified names not allowed" in str(exc_info.value)