Package: nypl-py-utils
PythonUtilsThis package contains common Python utility classes and functions.ClassesPushing records to KinesisSetting and retrieving a resource in S3Decrypting values with KMSEncoding and decoding records using a given Avro schemaConnecting to and querying a MySQL databaseConnecting to and querying a PostgreSQL databaseConnecting to and querying a PostgreSQL database using a connection poolConnecting to and querying RedshiftMaking requests to the Oauth2 authenticated APIs such as NYPL Platform API and SierraFunctionsReading a YAML config file and putting the contents in os.environ -- seeconfig/sample.yamlfor an example of how the config file should be formattedCreating a logger in the appropriate formatObfuscating a value using bcryptParsing/building Research Catalog identifiersUsage# test_file.pyfromnypl_py_utils.classes.kinesis_clientimportKinesisClientfromnypl_py_utils.functions.config_helperimportload_env_fileload_env_file(...)kinesis_client=KinesisClient(...)# requirements.txt# Do not use any version below 1.0.0# All available optional dependencies can be found in pyproject.toml.# See the "Managing dependencies" section below for more details.nypl-py-utils[kinesis-client,config-helper]==1.1.2Developing locallyIn order to use the local version of the package instead of the global version, use a virtual environment. To set up a virtual environment and install all the necessary dependencies, run:python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install .
pip install '.[development]'
deactivate && source .venv/bin/activateManaging dependenciesIn order to prevent dependency bloat, this package has no required dependencies. Instead, each class and helper file has its own optional dependency set. For instance, if an app needs to use the KMS client and the obfuscation helper, it should addnypl-py-utils[kms-client, obfuscation-helper]to the app's requirements. This way, only the required dependencies are installed.When a new client or helper file is created, a new optional dependency set should be added topyproject.toml. Thedevelopmentdependency set, which includes all the dependencies required by all of the classes and tests, should also be updated.The optional dependency sets also give the developer the option to manually list out the dependencies of the clients rather than relying upon what the package thinks is required, which can be beneficial in certain circumstances. For instance, AWS lambda functions come withboto3andbotocorepre-installed, so it's not necessary to include these (rather hefty) dependencies in the lambda deployment package.TroubleshootingUsing PostgreSQLClient in an AWS LambdaBecausepsycopgrequires a statically linked version of thelibpqlibrary, thePostgreSQLClientcannot be installed as-is in an AWS Lambda function. Instead, it must be packaged as follows:pipinstall--target./packagenypl-py-utils[postgresql-client]==1.1.2

pipinstall\--platformmanylinux2014_x86_64\--target=./package\--implementationcp\--python3.9\--only-binary=:all:--upgrade\'psycopg[binary]'Using PostgreSQLClient locallyIf using thePostgreSQLClientproduces the following error locally:ImportError: no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: No module named 'psycopg_binary'
- couldn't import psycopg 'python' implementation: dlsym(0x7f8620446f40, PQsslInUse): symbol not foundthen try running:pipuninstallpsycopg
pipinstall"psycopg[c]"Git workflowThis repo uses theMain-QA-Productiongit workflow.mainhas the latest and greatest commits,qahas what's in our QA environment, andproductionhas what's in our production environment.Ideal WorkflowCut a feature branch off ofmainCommit changes to your feature branchFile a pull request againstmainand assign a reviewer (who must be an owner)Include relevant updates to pyproject.toml and READMEIn order for the PR to be accepted, it must pass all unit tests, have no lint issues, and update the CHANGELOG (or contain theSkip-Changeloglabel in GitHub)After the PR is accepted, merge intomainMergemain>qaDeploy app to QA on GitHub and confirm it worksMergeqa>productionDeploy app to production on GitHub and confirm it worksDeploymentThe utils repo is deployed as a PyPI packagehereand as a Test PyPI package for QA purposeshere. In order to be deployed, the version listed inpyproject.tomlmust be updated. To deploy to Test PyPI, create a new release in GitHub and tag itqa-vX.X.X. The GitHub Actions deploy-qa workflow will then build and publish the package. To deploy to production PyPI, create a release and tag itproduction-vX.X.X.