Package: simple-strop
Simple Python PackageThis project is a simple python package that prompts the user for input, then performs a few simple operations on it, then prints it out. More interesting than the actual operations that this code performs are the concepts that it illustrates. We have a package (with multiple files), unit tests, coverage, dependency management, and configuration for a CI/CD pipeline.What does this package do?You can install this package from the python package managerpipwithpip install simple-strop. Once installed, the package has 2 modules (that aren't tests):operations.pyandutils.py. An example usage is below:>>>importsimple_stropasss>>>sample_string='This is my sample string'>>>ss.reverse(sample_string)'gnirts elpmas ym si sihT'>>>ss.piglatin(sample_string)'Isthay isway myay amplesay ingstray'>>>ss.is_capitalized(sample_string)True>>>ss.is_all_caps(sample_string)False>>>ss.is_vowel('y')False>>>ss.is_consonant('q')TrueThat's really all there is to it. Note that there are a few more functions inoperations.pythat you could use, but these are mostly just helper functions forpiglatin.Getting StartedBefore you can follow this demo, you'll need to do a few things:Install python version 3.9Fork this repositoryAdd CI/CD variablesInstall PythonTo install python, followthis linkor the one above and download the version for 3.9. If you are able to use the installer, make sure that you check the box for adding python to your path, otherwise you'll need to manually add python to your path.Fork the RepoTo "fork" the repository, simply click the fork button in the top right corner of the screen next to the star and notication buttons. Forking a repository makes a copy that you own and that you can edit and manage however you want. Forks can be merged back into the original project, or they might diverge and become their own projects.You can read more about forks here, but I won't talk about it more in this demo.Once you've forked the repo, you can clone from your fork. The URL to clone is available from the Clone dropdown below the fork button. You can clone to your local machine withgit clone <url>.Add CI/CD VariablesAs is often the case, there are some things that occur at build or release time that should be secret to all except for me (or you, after you fork). In this case, the secrets are login credentials for PyPI as those are needed in the pipeline to actually push this package to the public registry after it is built.To add the variables we need, on the left side of the screen hover over the settings button, then click "CI/CD". You will be presented with a list of collabsible sections. Scroll down to the one called "Variables" and click the "Add Variable" button. For the key, enterTEST_PYPI_USERNAMEand put in the username of your account on test.pypi.org. Before clicking "Add variable", make sure to check both the "Protect variable" and "Mask variable" boxes. This will prevent people from being able to reach the value of this variable when examining your pipelines or creating branches and running modified pipelines of their own. Repeat this process for the variables:TEST_PYPI_PASSWORD: the password for your account on test.pypi.orgPYPI_USERNAME: (optional) the username for your account on pypi.orgPYPI_PASSWORD: (optional) the password for your account on pypi.orgThe optional variables are only necessary if you intend to run the "release to pypi" stage, which would publish your package for all the world to see.Dependency Management (python specific)One of the most frequent issues when it comes to porting software from one machine to another is ensuring that the two machinesagree completelyon what the environment for running that code is. When I say "environment", I'm talking about the variables that are set on the system as well as the specific versions of packages that are installed on the system.If you're running a Linux distribution and I'm running on Windows and our mutual friend is running on MacOS, then there is no chance that we can haveallof our environment variables match (without some sort of virtualization... foreshadowing), but we can ensure that our package versions are identical by using a virtual environment manager likepipenv. Pipenv is an easy tool for ensuring that not only do you have the same packages installed, but that you have the same minute versions of those packages installed. It takes a little bit of getting used to, but once you've adopted it the benefits are immense!Once you start using pipenv, you also have the benefit that your list of globally installed packages (the packages that you install when you just runpip install ...) will become much shorter.Pipenv is just another python package and can be installed withpip install pipenv.Short Intro to PipenvThere are a few bread-and-butter commands, so I'll cover those first:CommandDescriptionpipenv shellThis enters your shell into the virtual environment or creates an empty one if the directory you're in doesn't contain a Pipfile.pipenv install <package_name>This installs a package into your environment and creates a new environment if the directory you're in doesn't container a Pipfile. If a Pipfile exists but there is no virtual environment stored on your machine, pipenv creates a virtual environment and populates it with packages from the Pipfile.lock.pipenv lockThis resolves all of the dependencies of all of the packages you've installed and chooses the most modern versions that don't conflict with other package dependencies. It can't be used unless you have a Pipfile. You can also use it to generate yourrequirements.txtpipenv uninstall <package_name>Remove a pacakge from your environment. Can't be used unless you have a Pipfile (with whatever package you're trying to remove)pipenv --rmDeletes the virtual environment for this directory, but keeps the Pipfile and Pipfile.lock. Useful if you've installed extraneous packages into this environment without using pipenv (likepip install ...inside of the pipenv shell) or if you are done working on a project for the time being and want to free up some space.So to get started, you go to the top level of whatever git repository you're using for your code and runpipenv shell. This will create and enter a virtual environment that is completely distinct from your base environment. You can install any packages you like without cluttering up your home environment withpipor you can add pacakges to your project withpipenv install.For this project, I added 5 packages: 2 build-time dependencies - build, twine; and 3 development-dependencies - pytest, pytest-cov, coverage. To install run/build time dependencies just usepipenv install ..., but if you want to install dependencies that are manually enabled, usepipenv install --dev .... The added benefit is when generating requirements.txt files, your dev packages won't be included (by default).After you install packages with pipenv, it will automatically generate aPipfileand aPipfile.lock(although you can tell pipenv not to lock). The Pipfile is basically just a meta description of what your project needs. It says things like what version of python the project is running in, what the packages are, what the dev dependencies are, and then some information about where pipenv was installed from. The Pipfile.lock is more specific and contains a hash of all of the packages for your project, hashes for verifying the installation of those packages, the specific version numbers of the packages, the list of dependencies for each installed package, hashes for those, their dependencies, etc.YOU SHOULD NEVER ATTEMPT TO MANUALLY EDIT YOUR PIPFILE.LOCK. In the worst case, it might make your environment uninstallable and you will need to remove the lock file (which will forget all of your pinned versions) in order to correct this. Then you might have to deal with situations where new versions of your dependencies were released that break your code... it's bad. Let pipenv handle the lock file and get familiar with commands arguments for the install, uninstall, and lock commands.Unit TestingFor this demo, I'll be focusing in on the unit tests for this package. As you can see in the directory, the tests are right along side the normal code in this package. This was a choice I made out of convenience because I wanted to do the minimal amount of setup with pytest and coverage and if I were to try and separate the tests from the source, it would create import complications and the coverage would be wrong (by default) and it would just be a whole mess.You can run the unit tests yourself locally. First install the environment via pipenv:pipenv install --dev, then enter the virtual environment withpipenv shell, then you can just run thepytestcommand. If you don't include the--devflag, you won't install the packages for unit testing and it won't work.Pytest is configured to also run a code coverage report and save that into a directory calledhtmlcov. After running the unit tests, open the filehtmlcov/index.htmlwith a web browser and you'll be able to interactively click through each file, see which lines are covered (spoiler alert, it's all of them), and see what the coverage of the whole project is.You can get familiar with how coverage is working by commenting out some of the functions in thesrc/simple_strop/test_...files then runningpytestagain to see the coverage re-calculated.Building the Package FilesOnce you've installed the virtual environment, go ahead and runpython -m buildfrom the top level of the repository. This will run for a few seconds and when it's done, you'll have a few new directories:build/,dist/,src/simple_strop.egg-info/. Each of these serves its own purpose, but thedistdirectory is how we might actuallydistributeour package.Inside ofdistthere should be two files:simple-strop-<version>.tar.gzsimple_strop-<version>-py3-none-any.whlThe first file is just a compressed archive of the files in this repository that are actually meaningful to the distribution of your project. If you'd like, you can open this file (on linux and mac machines natively, windows if you have gnu installed) withtar xf <filename>and then check out the new directory it created with the same name as the tarball.The second file is a "wheel". Wheel is python's format for creating installable packages. It's a binary file that contains the source code and instructions to pip (or pipenv) on how to actually install the package, what additional needs it has (like adding a file to the user's path like pipenv does), etc.You can actually install packages directly from wheel files withpip install <path_to_wheel>DistributingOnce you've built the package intodist, you can upload it to a package directory like PyPI! Unfortunately, for this demo, you won't be able to just directly do that, but you could edit the files here to make that possible.The command to upload ispython -m twine upload -r testpypi dist/*, but there are a few things to keep in mind:This is an effectively useless package and we probably shouldn't clutter up the global python directory with dozens of copies of this packageThere is actually a test version of PyPI that developers can use to play around or make sure that their package is actually all set for distributionbeforethey publicly publish it.You can find TestPyPI here.I've already published this package, so if you tried to as well, PyPI would complain that a file with that name already exists on their servers.You'll need to create an account with TestPyPI or PyPI to be able to upload any packages.So with those in mind, how do you edit this source code to upload it yourself? Well first I'll direct youto this guide. Before making this tutorial, I hadn't actually made a python package from scratch but I knew this would be a good example of CI/CD. This guide is published by python themselves and I strayed from it very little (and the places where I did I've documented here).You'll need to edit the filessetup.cfgto change the metadata of the project andLICENSEto change the directory to your legal intellectual property (provided you change anything else too according to the license I chose). After that, you should be good to go! Now you can change this repo to your heart's content, re-build, re-distribute and watch the magic!When you updatesetup.cfg, you should change the author name and email to be your own. You'll also need to change the version number each time you want to redistribute. PyPI doesn't allow you to overrite previous versions (this could cause strange errors with nested dependencies), so every change must be a new version. We are using "SemVer" or "Semantic Versioning" here, but you are free to identify versions in whatever way you like.You can read more about SemVer here. If you would like to know what the other values of thesetup.cfgare,you can read more about these configuration files here.PipelinesThe last component of this repository is the CI/CD pipeline. Pipelines are a sequence of steps organized into aDAG (directed acyclic graph)for managing builds, tests, releases, and anything else that a project might need to go from source code to distributable. You can see the configuration for this repository's pipeline in the file.gitlab-ci.ymlin the top level of the repository. The.ymlextension identifies this file as a "yaml" file, almost all pipelines are defined in yaml files as they are easy to read and are quite flexible.You can read more about yamls here.At a minimum, a pipeline should have 2 stages: build and deploy (but test is also highly recommended!!!). In this repo, we actually have 4 because we need to build two times: first to build the dependencies that our pipeline will need for testing and second to actually build our source code into the wheel file.The stages of the pipeline are defined at the top:stages:-build-test-package-deployEach of these will fulfill one part of the process and the final one should actually make our package public. As we go through the pipeline, I'll be referring to the content of the.gitlab-ci.ymlfile as if you can see it, so I recommend that you follow along as you read. For anything in the yaml that I don't explain, you can find explanationin GitLab's documentation here.TerminologyBefore we go into each stage, we should make sure we have the right terminology because I'm going to be throwing a lot at you:TermDefinitionArtifactStrictly speaking, an artifact is any file generated by a pipeline during its run. Artifacts are (by default) passed from stage to stage so that every job in a stage after an artifact is created may access it. In our project, we have a few artifacts: the built package dependencies, the html coverage report, and the built package in thedistdirectory.CacheJust like in your browser or most of the applications you use, caches are a locally saved version of some information that you use regularly, but that doesn't regularly change. In our pipeline, after we install the dependencies of our package from pip, we aren't making any more changes to these dependencies, but we'll need them in every subsequent job. That is a perfect scenario for a cache.Cache PolicyIn GitLab pipelines specifically, you are provided with a way to specify what level of access to a cache each job needs. These levels are push-pull, pull, and push. These do what you would roughly expect from their shared language with a git repo: pull allows the job to read from the cache, push allows the job to edit/overrite the cache, and pull-push allows both.With these ideas in mind, let's take a look at the pipeline.ThebuildStageOur build stage only has 1 job in it. A job is a single, focused collection of commands to accomplish a specific goal. It is more focused than stages. This job is calledbuild-dependenciesand it is responsible for installing the packages we need for the later stages.In this stage, we see the first occurrence of a "yaml anchor". Anchors can get pretty complicated, but the easiest way to think about them are as variables but for yaml files: you define a variable with&variable_nameand you can then reference it with*variable_name. So when we see:.cache:&cachekey:$CI_COMMIT_REF_SLUGpolicy:pullpaths:-$PIP_CACHE_DIR-$PYTHON_PACKAGE_DIRbuild-dependencies:stage:buildcache:<<:*cachepolicy:pull-pushwe are defining the anchorcacheat the top to be the object containing three keys:key,policy, andpathswherepathsis an array with two indices. We then use ourcacheanchor inside of thebuild-dependenciesobject with the "merge operator"<<: *variable_name.This merge operator is saying, "I want to put the value of this anchor in this position as if it were defined here". So after the merge operator, if we expanded this file it would look like this:.cache:key:$CI_COMMIT_REF_SLUGpolicy:pullpaths:-$PIP_CACHE_DIR-$PYTHON_PACKAGE_DIRbuild-dependencies:stage:buildcache:key:$CI_COMMIT_REF_SLUGpolicy:pullpaths:-$PIP_CACHE_DIR-$PYTHON_PACKAGE_DIRpolicy:pull-pushNotice that the key "policy" is repeated. In this instance, we want to override it from what is defined in our anchor, so we can just define it again and yaml will forget the previous value.Inside of thebuild-dependenciesjob, we can see that we have defined it to be in the stage "build". We also define how we want the cache to work. Defining a cache isn't necessary, but it is useful for speeding up repeated builds. In this stage, however, we don't want to use the cache, we want to overwrite the cache so that it is fresh for this build. Here, we identify a reusable cache by the git commit the pipeline was run for (CI_COMMIT_REF_SLUG) and say that we want to be able to push to this cache.We also define our "artifacts". Artifacts are files, strings, or directories that are the output of a job and are needed for another job or just for later reference. In this case, after we install our packages, we want to keep the directories where we installed these packages for use in the future stages, so we define those directories as artifacts.Finally, we define our scripts. Before executing the standard script, we make sure to delete our old cache and add our cache to our PYTHONPATH environment variable. This tells python how to find these packages later. Then we run twopip installcommands to install the packages that we defined in our requirements files. Note the usage of variables with${VARIABLE_NAME}. This allows us to change these values easily in one spot (the variables section) and not have to worry about tracking down all of the places in the pipeline we used them. GitLab's pipelines support variables in almost any value for key value pairs and as environment variables in scripts.ThetestStagetest:stage:testcache:<<:*cacheartifacts:expire_in:1 daypaths:-htmlcov-.coveragecoverage:'/Totalcoverage:\d+.\d+%$/'script:-python -m pytestNow that we've built our dependencies, we're ready to run our unit tests. You can see that most of the configuration here is very similar to the build stage: we pull in our cache anchor (without overriding the default policy), we define the coverage output as artifacts, and we run a script. The script here is justpython -m pytestbecause we already defined.coveragercandpytest.inito run our tests with all of the command line flags and configurations that we need. Note: when we were running locally, we ran with justpytest. The command here is logically identical, but a little more specific and helps us avoid issues with caching in pipelines.Note the inclusion of thecoverageline. That is how we tell gitlab to parse the output of this job with the regular expression so that the pipeline can report our code coverage at the end.This stage is relatively simple because of our configuration, so I'll leave the reader with this question: how might we have a problem if we needed access to some external service during our unit tests? For example a database or a third-party API?ThepackageStagebuild-package:stage:packagecache:<<:*cacheartifacts:expire_in:1 daypaths:-distscript:-python -m buildThis is pretty similar to the previous stages as well, but here we are defining ourdistdirectory as an artifact. We want to make sure that in the final stage, we access it to deploy. We have this stage defined on its own for a few reasons: first, we want it to benefit from the cache because it would be very annoying to have to rebuild this late in the pipeline; second, we don't want to spend the time actually building until we've tested and know that everything is working correctly.ThedeployStage.release:stage:deploywhen:manualcache:<<:*cachedependencies:-build-packagerelease to testpypi:extends:.releasescript:-python -m twine upload -u ${TEST_PYPI_USERNAME} -p ${TEST_PYPI_PASSWORD} -r testpypi dist/*release to pypi:extends:.releasescript:-python -m twine upload -u ${PYPI_USERNAME} -p ${PYPI_PASSWORD} dist/*Finally, we've built and tested our package and we're ready to hand it off the the world! It's time to introduce one final concept of pipelines: templating. Templates and anchors share the same basic goal to reduce repetition, but they accomplish it in different ways. Anchors are single-fileonly. If I defined a pipeline where one file pulled configuration from another, then I couldn't reuse the anchors across files. Also, it looks a little cleaner to use templates. Whereas before I needed 3 or 4 paragraphs to explain what<<:was doing, here you see "extends" and already have an idea.When we say "extends" here, what we're really saying is "give me everything from this job". So inrelease to testpypi, even though I didn't say it there it isas ifI had definedstage: deployandwhen: manual. Note, I still used an anchor inside of my template and I could even have my template extend from somewhere else.Though I didn't include it in this repository, I could have set up each of these stages to be in their own files called something like.build.yml,.test.yml, etc. Then I could haveincludedthose files in my.gitlab-ci.ymlpipeline and used them as if they were defined at the top. In that situation, however, I couldn't have used mycacheanchor so if I wanted to share that, I could have broken that out into its own job and had each of these other jobs extend from cache. In this small example, that's not really worth it. But if I had a huge project with a really big pipeline, then I might consider refactoring my pipeline in that way.Additionally, if I was working in an organization, I probably won't want to write a pipeline for every project that is more or less the same. I could make a repo that just holds pipeline templates and include templates from that repo instead! Saves time and ensures that if I ever find a bug in one, I can patch all of my pipelines at once.Now with our releases to TestPyPI and PyPI defined, we are ready to deploy! But hang on, let's say that we have something to deploy that might break the version of our code that our users are already using? Generally speaking, it's a bad idea for pipelines to push all the way to production without having some sort of user input so we have also defined thewhenvalue in.releaseto be "manual". This will run all of the jobs of the pipeline except for this one, then wait for an authorized user's input before continuing.