Package: localstack-plugin-loader
localstack-plugin-loaderlocalstack-plugin-loader is the dynamic code loading framework used inLocalStack.OverviewThe localstack-plugin-loader builds a higher-level plugin mechanism aroundPython's entry point mechanism.
It provides tools to load plugins from entry points at run time, and to discover entry points from plugins at build time (so you don't have to declare entry points statically in yoursetup.py).Core conceptsPluginSpec: describes aPlugin. Each plugin has a namespace, a unique name in that namespace, and aPluginFactory(something that createsPluginthe spec is describing.
In the simplest case, that can just be the Plugin's class).Plugin: an object that exposes ashould_loadandloadmethod.
Note that it does not function as a domain object (it does not hold the plugins lifecycle state, like initialized, loaded, etc..., or other metadata of the Plugin)PluginFinder: finds plugins, either at build time (by scanning the modules usingpkgutilandsetuptools) or at run time (reading entrypoints of the distribution usingstevedore)PluginManager: manages the run time lifecycle of a Plugin, which has three states:resolved: the entrypoint pointing to the PluginSpec was imported and thePluginSpecinstance was createdinit: thePluginFactoryof thePluginSpecwas successfully invokedloaded: theloadmethod of thePluginwas successfully invokedLoading PluginsAt run time, aPluginManageruses aPluginFinderthat in turn uses stevedore to scan the available entrypoints for things that look like aPluginSpec.
WithPluginManager.load(name: str)orPluginManager.load_all(), plugins within the namespace that are discoverable in entrypoints can be loaded.
If an error occurs at any state of the lifecycle, thePluginManagerinforms thePluginLifecycleListenerabout it, but continues operating.Discovering entrypointsAt build time (e.g., withpython setup.py develop/install/sdist), a specialPluginFindercollects anything that can be interpreted as aPluginSpec, and creates from it setuptools entrypoints.
In thesetup.pywe can use theplugin.setuptools.load_entry_pointsmethod to collect a dictionary for theentry_pointsvalue ofsetup().fromplugin.setuptoolsimportload_entry_pointssetup(entry_points=load_entry_points(exclude=("tests","tests.*",)))Note thatload_entry_pointswill try to resolve a cached version ofentry_points.txtfrom the.egg-infodirectory,
to avoid resolving the entry points when building the package from a source distribution.ExamplesTo build something using the plugin framework, you will first want to introduce a Plugin that does something when it is loaded.
And then, at runtime, you need a component that uses thePluginManagerto get those plugins.One class per pluginThis is the way we went withLocalstackCliPlugin. Every plugin class (e.g.,ProCliPlugin) is essentially a singleton.
This is easy, as the classes are discoverable as plugins.
Simply create a Plugin class with a name and namespace and it will be discovered by the build timePluginFinder.# abstract case (not discovered at build time, missing name)classCliPlugin(Plugin):namespace="my.plugins.cli"defload(self,cli):self.attach(cli)defattach(self,cli):raiseNotImplementedError# discovered at build time (has a namespace, name, and is a Plugin)classMyCliPlugin(CliPlugin):name="my"defattach(self,cli):# ... attach commands to cli objectnow we need aPluginManager(which has a generic type) to load the plugins for us:cli=# ... needs to come from somewheremanager:PluginManager[CliPlugin]=PluginManager("my.plugins.cli",load_args=(cli,))plugins:List[CliPlugin]=manager.load_all()# todo: do stuff with the plugins, if you want/need#  in this example, we simply use the plugin mechanism to run a one-shot function (attach) on a load argumentRe-usable pluginsWhen you have lots of plugins that are structured in a similar way, we may not want to create a separate Plugin class
for each plugin. Instead we want to use the samePluginclass to do the same thing, but use several instances of it.
ThePluginFactory, and the fact thatPluginSpecinstances defined at module level are discoverable (inpired
bypluggy), can be used to achieve that.classServicePlugin(Plugin):def__init__(self,service_name):self.service_name=service_nameself.service=Nonedefshould_load(self):returnself.service_nameinconfig.SERVICESdefload(self):module=importlib.import_module("localstack.services.%s"%self.service_name)# suppose we define a convention that each service module has a Service class, like moto's `Backend`self.service=module.Service()defservice_plugin_factory(name)->PluginFactory:defcreate():returnServicePlugin(name)returncreate# discoverables3=PluginSpec("localstack.plugins.services","s3",service_plugin_factory("s3"))# discoverabledynamodb=PluginSpec("localstack.plugins.services","dynamodb",service_plugin_factory("dynamodb"))# ... could be simplified with convenience framework code, but the principle will stay the sameThen we could use thePluginManagerto build a SupervisorclassSupervisor:manager:PluginManager[ServicePlugin]defstart(self,service_name):plugin=manager.load(service_name)service=plugin.serviceservice.start()Functions as pluginswith the@plugindecorator, you can expose functions as plugins. They will be wrapped by the framework
intoFunctionPlugininstances, which satisfy both the contract of a Plugin, and that of the function.frompluginimportplugin@plugin(namespace="localstack.configurators")defconfigure_logging(runtime):logging.basicConfig(level=runtime.config.loglevel)@plugin(namespace="localstack.configurators")defconfigure_somethingelse(runtime):# do other stuff with the runtime objectpassWith a PluginManager viaload_all, you receive theFunctionPlugininstances, that you can call like the functionsruntime=LocalstackRuntime()forconfiguratorinPluginManager("localstack.configurators").load_all():configurator(runtime)Installpip install localstack-plugin-loaderDevelopCreate the virtual environment, install dependencies, and run testsmake venv
make testRun the code formattermake formatUpload the pypi package using twinemake upload