Package: utilities-hki
Global Utilitieslast modified 8 March 2023 by Colleen TreadoTheutilities-hkirepository contains the common utilities required by multiple otherhumankind-datasciencerepositories. Unlike the oldutilitiesrepo, this package contains no encrypted files, and credentials are now passed into the utility functions as input arguments.Current statusThis repository is the code repository for theutilities-hkipip package, which, together with the newcredentialsrepository, replaces the currentutilitiessubmodule in the other repositories used for data science at Humankind. The package was created by followingthis guideand the package can be found onPyPI. Most of our repositories have been updated to import the newutilities-hkipip package and call the updated utility functions, passing in the credentials from the newcredentialsrepo, instead. The repositories that still need to be updated aredaily_volume_predictvolume_predict (but this is not in use)facebook_adsInstallation and setupFor first-time setup, clone the repository into a fresh work area:# cloning via ssh is preferred but requires an ssh key connection in your accountgitclonegit@github.com:humankind-datascience/utilities-hki.gitThe code requires a number of Python packages to run, which should be installed inside of a dedicated virtual environment. The preferred virtual environment tool isvirtualenvwrapper.To install the required packages in a new virtual environment, run the following command from the top-level directory of the git repository:pipinstall-rrequirements.txtIf additional packages need to be installed upon changes to the code, add them to therequirements-top-level.txtfile. Then run the below commands to install (and upgrade) the top-level dependencies and update therequirements.txtfile for future use.pipinstall-rrequirements-top-level.txt--upgrade
pipfreeze-rrequirements-top-level.txt>requirements.txtAdditionally, the AWS Command Line Interface (AWS CLI) is required for use of the botocore library, which is used in database utilitify functions to read from and write to the AWS RDS databases. See theAWS CLI documentationfor installation instructions.Now you can run the top-level scripts:python<utilities-script.py>Theutilities-hkirepository contains only testing top-level scripts, designed to test the utility functions during package development.Code updatesWhen making changes to the code, followGitHub flow, i.e. create a new branch, make changes on that branch, frequently committing and pushing those changes to that branch, and then create a pull request to merge those changes into master upon review and approval.Utility code overviewTheutilities-hkirepository contains common utility functions used across repositories in the Humankind Data Science code base. The utility functions are grouped by type into separate modules, as outlined below.analy_utils: analysis utility functions, including cleaning procedures for and assignment of engagement types to the visit-level data;db_utils: database utility functions;email_utils: email utility functions;fb_utils: Facebook Ads utility functions.Standard cleaning of the visit-level data should be implemented at the start of any analysis and can be achieved by calling theanaly_utils.clean_visitsfunction (see the docstrings for more details.Sample code for applying the trained clustering model and assigning the letter/numeric grades to the engagement types for each visit is provided below, wherevisitis a DataFrame. Read the docstrings for assign_cluster() and get_cluster_grades() for more details.from utilities-hki import analy_utils
# assumes visit data has already been pulled or loaded into visit

engagement = analy_utils.assign_cluster(visit)
engagement = engagement.reset_index().merge(
    analy_utils.get_cluster_grades(), how='left', on='engagement_type')
visit = visit.merge(engagement, how='inner', on='visit_id')