Package: cognite-extractor-manager
cogexcogexis a tool for managing extractors for Cognite Data Fusion written in Python. It provides
utilities for initializing a new extractor project and building self-contained executables of Python
based extractors.Important note for users runningpyenvpyenvis a neat tool for managing Python installations.Sincecogexuses PyInstaller to build executables, we need Python to be installed with a shared
instance oflibpython, whichpyenvdoes not do by default. To fix this, make sure to add the--enable-sharedflag when installing new Python versions withpyenv, like so:envPYTHON_CONFIGURE_OPTS="--enable-shared"pyenvinstall3.9.0You can read more about it in thePyInstaller documentationOverview of featuresStart a new extractor projectTo start a new extractor project, move to the desired directory and runcogexinitYou will first be prompted for some information, beforecogexwill initialize a new project.Add dependenciesExtractor projects initiated withcogexwill usepoetryfor managing dependencies. Runningcogex initwill automatically install the Cognite SDK and extractor-utils framework, but if your
extractor needs any other dependency, simply add them usingpoetry, like so:poetryaddrequestsType checking and code styleIt is recommended that you run code checkers on your extractor, in particular:blackis an opinionated code style checker that will enforce a consistent code style throughout
your project. This is useful to avoid unecessary changes and minimizing PR diffs.isortis a tool that sorts your imports, also contributing to a consistent code style and
minimal PR diffs.mypyis a static type checker for Python which ensures that you are not making any type errors
in your code that would go unnoticed before suddently breaking your extractor in production.cogexwill install all of these, and automatically run them on every commit. If you for some
reason need to perform a commit despite one of these failing, you can rungit commit --no-verify,
although this is not recommended.Build and package an extractor projectPackaging a binary of your extractorIt is not always an option to rely on a Python installation at the machine your extractor will be
deployed at. For those scenarios it is useful to package the extractor, including its dependencies
and the Python runtime, into a single self-contained executable. To do this, runcogexbuildThis will create a new executable (for the operating system you rancogex buildfrom) in thedistdirectory.Making docker imagesTo build a docker image, you first need to add a[tools.cogex.docker]section to your pyproject
file. The required fields aretags: A list of tags to tag the resulting image with. These support some simple templating, if
you include{version}in your tag, it will be replaced with the current version of the
extractor.{major}will be replaced with the current major version.If your[tool.poetry.scripts]includes multiple entries, you need to specify which one to use
in the docker image with theentrypointfieldIn addition, you have some additional fields:base-image: Which base image to use. By default, thedebian-slimbased python image for the
python version currently running with be chosen.install-dirif you want to specify where in the image the extractor should be installedpreamblewhich can contain additional dockerimage statements to run in the beginning of the
dockerfile.Minimal example:[tool.cogex.docker]tags=["cognite/my-extractor:{version}"]Larger example (from the DB Extractor):[tool.cogex.docker]base-image="python:3.10"preamble="""RUN apt-get update\&& apt-get dist-upgrade -y dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server\&& gpgconf gpgsm gpgv libssl-dev libssl1.1 opensslRUN apt-get install -y apt-utils build-essentialRUN apt-get install -y unixodbc-dev unixodbc"""tags=["eu.gcr.io/cognite-registry/db-extractor-base:latest","eu.gcr.io/cognite-registry/db-extractor-base:{version}","cognite/db-extractor-base:{version}",]You can now build and tag docker images withcogex build --dockerimageIf you just want to see the generated dockerfile, instead runcogex build --dockerfileCreating a new version of your extractorTo keep track of which version of the code base is running at a given deployment it is very useful
to version your extractor. When releasing a new version, runpoetryversion[patch/minor/major]To automatically bump the corresponding version number. Note that this only updates the version
number inpyproject.toml. When runningcogex buildthis new version number will be propagated
through the rest of the code base.Any extractor project should follow semantic versioning, which means you should bumppatchfor any minor bug fixes or improvementsminorfor new features or bigger improvements thatdoesn'tbreak compatabilitymajorfor new feature or improvements that breaks compatability with previous versions, in
other words for those scenarios where the new version is not a drop-in replacement for an old
version. For example:When adding a new required config fieldWhen removing a config fieldWhen changing defaults in a way that could break existing deployments