Package: otumat
Otumat: Maintainer Tools & UtilitiesOtumat (pronounced "Automate") is a suite of maintainer tools and utilities for pip packages.The following features are currently supported.Usage TrackingHave you ever wondered:How many users are actually using my Python package?How often are they using my Python package?Which features/methods are most used and which ones are less interesting to the community?Is there a better alternative to track and measure usage data than using anonymous download data available from PyPi's download logs in Google BigQuery? SeeAnalyzing PyPI Package Downloadfor more details on thisSince a reasonable solution could not be found, I introduced this feature that provides the mechanism and building blocks to have usage tracking data as granular as you need it.There are a few pre-requisites or assumptions:User clients will have at least some periodic internet connection to upload usage logs to a centralized, remote usage-data aggregating host4 HTTP routes need to be implemented on your remote usage-data hostGETGUI-based authenticated route to register package installations with a user. You may use it to collect consent, have your user complete a survey, etc.POSTauthenticated API route to accept the form submission of the above GUI route. AninstallIdshould be returned along with other details to ensure an 'open' connection.POSTauthenticated API route that accepts user's event data and will store in an medium of your choice.POSTstandard OAuth2.0 route that will allow refreshingaccess_token's andrefresh_token's.PKCE flow) currently implemented).Specific request/response details for the above 4 routes to follow soon.Once your remote server is ready, simply add the following to your package:Includeotumatas arequirementsdependencyIn your__init__.py, intantiate anUsageAgentthat your package can refer to. For example:fromotumat.usageimportUsageAgentas_UsageAgentusage_agent=_UsageAgent(author='DataJoint',data_directory='datajoint-python',package_name=__name__,host='https://datajoint.io',install_route='/user/usage-install',event_route='/api/usage-event',refresh_route='/auth/token',response_timeout=300,upload_frequency='12h')Therefore, the first time your package is imported on the client's machine, it will trigger the usage tracking installation enrollment. User's will need to opt-in though the default isnotto collect usage data.Log any interesting event within your package using the instantiatedUsageAgent. For example, we can log imports by including the following also in our__init__.py:usage_agent.log(event_type='import')Events will be buffered locally until the upload interval arrives. Caches are then unloaded. Daemon service runs cross-platform for Windows, MACOS, Linux and activates on startup.Specific example of what an implemented flow looks like to follow soon.Disable Usage Tracking RegistrationThere are some cases where it is undesirable to have the usage tracking flow triggered. For instance, if you'd like to depend on a package (e.g.datajoint) which does have the usage tracking flow enabled but would rather not trigger it within your package. For such a case, you could do the following in your package's__init__.pybefore your first import fromdatajoint. It will effectively disable usage tracking checks, flows, and prompts in your package:importotumatas_otumat_otumat.DISABLE_USAGE_TRACKING_PACKAGES=(['datajoint']+_otumat.DISABLE_USAGE_TRACKING_PACKAGES)# first import from package with usage tracking enabledimportdatajointFile WatchingThis feature allows you to run a given script whenever a selected file is modified.To watch a file, installotumatusingpip install otumat, then run the command:otumat watch [-h] -f WATCH_FILE [-i WATCH_INTERVAL] -s  WATCH_SCRIPT [watch_args ...]ArgumentsHelp:-h, --helpRequired named arguments:-f WATCH_FILE: Path to file to be watched.-s WATCH_SCRIPT: Path to script to run on file change.Optional named arguments:-i WATCH_INTERVAL: Interval in seconds between polls.Defaults to 5 seconds.watch_args: Arguments providing state between runs.Defaults to no arguments.Validation of Trusted PluginsThis package also includes a setuptools extension that provides new keyword argumentsprivkey_pathandpubkey_path.By specifying theprivkey_path, setuptools will generate the git hash (SHA1) of the module directory and sign the output based on the PEM key path passed in. The resulting signature will be stored as egg metadata{{module_name}}.sigaccessible viapkg_resourcesmodule.If passingpubkey_path, this will simply be copied in as egg metadata{{module_name}}.pub.This provides a solution to determining the 'trust-worthiness' of plugins or extensions that may be developed by the community for a given pip package if the public key file is available for the RSA keypair. The choice of what to do for failed verification is up to you.UseExtensible Package e.g.basesetuptools.setup(...setup_requires=['otumat'],pubkey_path='./pubkey.pem',...Plugin Package e.g.plugin1setuptools.setup(...setup_requires=['otumat'],privkey_path='~/keys/privkey.pem',...Verifying Contentsimportpkg_resourcesfrompathlibimportPathfromotumatimporthash_pkg,verifybase_name='base'plugin_name='plugin1'base_meta=pkg_resources.get_distribution(base_name)plugin_meta=pkg_resources.get_distribution(plugin_name)data=hash_pkg(pkgpath=str(Path(plugin_meta.module_path,plugin_name)))signature=plugin_meta.get_metadata('{}.sig'.format(plugin_name))pubkey_path=str(Path(base_meta.egg_info,'{}.pub'.format(base_name)))verify(pubkey_path=pubkey_path,data=data,signature=signature)Compatibility withgitandopensslCLIFor reference, certificates may also be generated and verified usinggitandopensslby the following process:Generate$cd{{/path/to/local/repo/dir}}$gitadd.--all
$GIT_HASH=$(gitls-files-s{{/pip/package/dir}}|githash-object--stdin)$printf$GIT_HASH|openssldgst-sha256-sign{{/path/to/privkey/pem}}-out{{pip_package_name}}.sigbin-sigoptrsa_padding_mode:pss
$opensslenc-base64-in{{pip_package_name}}.sigbin-out{{pip_package_name}}.sig
$rm{{pip_package_name}}.sigbin
$gitresetVerify$cd{{/path/to/local/repo/dir}}$gitadd.--all
$GIT_HASH=$(gitls-files-s{{/pip/package/dir}}|githash-object--stdin)$opensslenc-base64-d-in{{pip_package_name}}.sig-out{{pip_package_name}}.sigbin
$printf$GIT_HASH|openssldgst-sha256-verify{{/path/to/pubkey/pem}}-signature{{pip_package_name}}.sigbin-sigoptrsa_padding_mode:pss
$rm{{pip_package_name}}.sigbin
$gitreset