Package: ltd-conveyor
LSST the Docs: ConveyorConveyoris a common Python library and command-line app for LSST the Docs (LTD) services and clients.Key features:ltdcommand-line app.Theltd uploadsubcommand makes it easy to integrate documentation uploads with continuous integration jobs, particularly Travis CI.TheltdconveyorPython package provides APIs for working with LSST the Docs.ltdconveyor.s3subpackage provides S3 object management services.
All documentation hosted by LTD is served through theFastly CDN, which uses S3 object metadata to purge caches.
Conveyor ensures that this metadata is properly set by all clients and microservices that upload, copy, or delete objects from S3.ltdconveyor.fastlyprovides Fastly cache management APIs.ltdconveyor.keeperprovides APIs for working with LTD Keeper’s REST API.You can learn more about LTD in ourSQR-006technote.Read the docs:https://ltd-conveyor.lsst.ioChange log0.8.3 (2024-02-22)Improved error reporting from LTD Keeper API responses for common scenarios and direct the user to contact dm-docs-support on the LSSTC Slack.0.8.2 (2024-02-22)Removed usage ofpkg_resourcesfor better compatibility with Python 3.12 environments.Fixed internal typing issues.Updated the GitHub Actions workflows.0.8.1 (2021-09-27)Fix parsing of theGITHUB_HEAD_REFenvironment variable in GitHub Actions.0.8.0 (2021-09-15)Theltd uploadcommand now works in pull request workflows on GitHub Actions.0.7.0 (2020-09-01)This release focuses on infrastructure improvements, and should not have a significant impact on users.InfrastructureThis tool is officially tested and supported on Python 3.7 and 3.8.
Python 3.6 is no longer officially supported.Removed pins on third-party dependencies, including boto3, requests, uritemplate, and click.
In doing so, LTD Conveyor is easier to integrate into your existing Python environment.We have migrated from Travis CI to GitHub Actions for testing, linting, documentation builds and deployment, and deployments to PyPI.
In addition to on-demand testing, GitHub Actions tests the package on a daily basis to ensure compatibility with third-party dependencies.Internally, we use tox for running tests, linters, and to build documentation.
tox makes it possible for local test runs to use the same set up as CI.The codebase is now automatically formatted with black and isort.
By using pre-commit, we ensure that formatting is always applied.The codebase now has type annotations, which are tested with mypy.
Type annotations help ensure that APIs are explicitly defined and correctly used.0.6.1 (2020-02-23)FixedThe--ghoption forltd uploadnow supports tag and PR events, in addition to branches.0.6.0 (2020-02-20)AddedAdded a--ghoption to theltd uploadcommand to support usage inGitHub Actions.0.5.0 (2020-02-05)AddedLTD Conveyor can now upload new builds to S3 using Amazon’s S3’s presigned POST URL feature.
This means that clients no longer need credentials for S3 — the LTD Keeper API server generates presigned POST URLs as part of thePOST/products/<product>/builds/build registration step.Theltdcommand now uses the presigned POST-based URLs.
The--aws-idand--aws-secretoptions have been removed, but the--userand--passwordoptions remain.If you use environment variables, there should be no change in how you useltd upload(aside from not needing the$LTD_AWS_IDand$LTD_AWS_SECRETenvironment variables).FixedIn INFO-level and higher logging, the module path isn’t displayed.
Module paths are only shown in DEBUG-level logging where it’s most useful.InfrastructurePackaging is now done through asetup.cfgfile and apyproject.tomlfile (PEP 518).Updated Travis CI configuration to test with Python 3.8; also refactored the CI Pipeline with Travis CI stages.0.4.2 (2018-10-09)FixedFixed a bug whereltdconveyor.s3.delete_dir, since 0.4.1, would raise aTypeErrorwhile deleting an empty directory (no objects in the S3 prefix).0.4.1 (2018-10-08)FixedFixed a bug whereltdconveyor.s3.delete_dirwould fail if there are more than 1000 objects under a path prefix that is being deleted.Fixed title of the project in the documentation by updating to use aDocumenteer-based Sphinx set up.0.4.0 (2018-04-17)AddedAdded a new command-line app,ltd, that provides subcommands for clients to work with the LSST the Docs API.
This app is implemented withClick, and its documentation is automatically generated from the command-line help withsphinx-click.The first subcommand isltd upload, which lets a client upload a site to LTD as a new build.
This command includes special features for using the client from Travis CI (ltd upload--travisoption) to populate the version information from the Travis environment.
Other flags allowltd uploadto become a no-op under certain circumstances (for example, skip uploads on pull requests) or arbitrarily (setltd upload--skiporexport LTD_SKIP_UPLOAD=true).NoteWith this feature, LTD Conveyor effectively deprecatesLTD Mason.
LTD Mason was the original uploader for LSST the Docs, but it was also designed around the idea of building theLSST Science Pipelinesdocumentation as well (hence the term “mason”).
Over time, we realized it is better to have a general purpose client and uploader for LSST the Docs (LTD Conveyor) and a dedicated tool for assembling the multi-package Sphinx documentation site (Documenteer).Added theltdconveyor.keepersubpackage that widens the scope of LTD Conveyor to be a full-service library for building LTD clients, not just an S3 upload client.Theltdconveyor.keeper.login.get_keeper_tokenfunction lets you obtain a temporary auth token for the LTD Keeper API.Theltdconveyor.keeper.buildmodule includes functions for performing the build upload handshake with the LTD Keeper API.ChangedRemoved Python 2.7 and 3.4 compatibility.
Now the codebase is entirely Python 3 (3.5, 3.6)-oriented.Completely reorganized Python namespace.
Now all S3 APIs are inltdconveyor.s3.
Fastly APIs are available fromltdconveyor.fastly.Switched to usingsetuptools_scmto generate version strings for releases.Switched to usingextras_requirefor development dependencies (pip install-e".[dev]").
This lets us exclusively coordinate dependencies insetup.py.Enable testing viapython setup.py test.
Also upgrade the testing stack topytest3.5 andpytest-flake81.0.Compatibility updates to the Sphinx documentation infrastructure.Default toacl=Noneto support more AWS IAM users.
It turns out that not all IAM users withPutObjectpermissions also have permissions to set theACLfor an object.
We want to make it possible for many lightweight IAM users to upload to restricted sub-directories of thelsst-the-docsbucket, but it seems hard to make these users ACL grantees too.
We now seek an alternative:Thelsst-the-docsbucket now has a PublicReadbucketpolicyNo ACL is set on individual objects.FixedChanged assertions toRuntimeErrors.
Assertions shouldn’t be used to raise exceptions in production code.0.3.1 (2017-03-27)AddedAddopen_bucketfunction.
This provides a convenient API for getting a boto3 bucket resource, particularly for clients that use the upload_object and upload_file APIs directly that take a bucket only.0.3.0 (2017-02-20)AddedAddedcontent_typeparameter toupload_objectso that a user can specifytext/htmlfor an HTML upload.
Theupload_file/upload_dirfunctions avoid this problem by using Python’smimetypeslibrary to guess the encoding, butupload_objectis lower-level and warrants having the user explicitly provide the content type.Refactored new functioncreate_dir_redirect_object.
This code used to be insideupload_dir, but inLTD Dasherit seems necessary to upload files one at a time, and thus it’s necessary to directly create these directory redirect objects.FixedFix issue where'..'shows up in directory names (seen when doingupload_dirwithLTD Dasherasset directories).0.2.0 (2017-02-02)AddedPorted Fastlypurge_keyfunction fromLTD Keeper.
Any LTD application that uploads objects to S3 might also need to purge Fastly CDN caches.
This purge capability is presented as a simple function.0.1.0 (2017-01-25)AddedPort S3 codebase fromLTD Mason.
The purpose of LTD Conveyor is to provide a set of common S3 APIs that can be used from both client (uploading docs from CI) and server (LTD Keeper) settings.MIT LicenseCopyright 2016-2020 Association of Universities for Research in Astronomy, Inc. (AURA)Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.