Package: risknet
Welcome to Risknet! This is a downloadable Pip package where you can access and run an XGBoost pipeline.Folder/File Layout Layoutsrc/risknetconfig: holdsconf.yamlconf.yaml: helps with setuphandlers.pydata: empty folder where user can store FM datasetjobscloud_etl.py: helps save files to cloudproc: contains preprocessing steps like feature encoding, label prep, and train-test splitsencoder.py: feature engineering/encoding categoricalslabel_prep.py: defines default, progress on loanreducer.py: reduces features based on importance, also train/test/val splits
-run: contains files for running pipelinemain.py: currently empty. WIP define pipeline as a function and call heremodel.py: defines the model class + functionspipeline.py: calls functions to execute the pipelinesys: contains files to set up system environment and logginglog.py: sets up loggermanagers.py: sets up the dask managersystem.py: defines creating and removing files via thesyspackage
-main.py: logs start, stop time for running the program (including downloading packages from setup.cfg)tests: store tests here
-test_stub.py: currently only asserts True == True. No tests added yet.Running The CodeCurrently, this code is hosted on testpypi, a Test version of the Python Packaging Index. You can see package documentationhere.As described in the website above, you can access the code by runningpip install -i https://test.pypi.org/simple/ riskneton your local computer. Ensure pip is updated.Note: this code might not work on your computer as many dependencies of the package like pandas are not available for download via testpypiTo run a specific part of the code, use risknet.utils.{feature_name} as is standard practice when accessing Python packages.Reproducibility Information:Accessing DataYou may want to access our base dataset for reference purposes. Here's how you can do that:In this study we will use the Freddie Mac Single-Family Loan Dataset to run code. Specifically, we will use the 2009_Q1 dataset.Go to this linkhere. This will redirect you to the Freddie Mac site.Scroll down until you reach a table that says "Loan-Level Dataset Files". Download thestandard quarterlydataset.Submit necessary education credentials including name, email, and reason for accessing the files. There should not be a payment step. The site will email you a username and password.Reload the page and log in with your new email/password credentials.Download the Quarter 1 data from the year 2009. You will receive a .zip file in your Downloads folder.If you unzip the file, you will see multiple files including a "date_time" file and a "data" file.
6. Save these files into the src/data folder in a local copy of this repositoryDependenciesYou can find a list of this package's dependencies inside the file calledsetup.cfg.In summary, the downloads needed for this code are:numpy==1.26.1pandas==2.1.2dask[complete]==2023.10.1xgboost==2.0.1PyYAMLtypes-PyYAMLpyarrowfastparquetpytestpytest-covmypyflake8Steps to Update Version on TestPyPiTo update the version:Reset the code from the previous version (if necessary)rm -rf dist buildto remove build foldermanually remove "egg-info" folder. This will changesrctosrc/risknet.Update setup.cfg's version number depending on if major, minor, or bug changeRerunpython3 -m build(you should get a new dist folder + egg folder in \src)THIS SHOULD CREATE A NEW binary file where version is UPDATEDMake sure you're in the same directory as your setup.cfg when you run this command.Rerunpython3 -m twine upload --repository testpypi dist/*Username:__token__Password: [testpypi password starting with pypi]If you did NOT update the version # before runningbuildthen you will get an errorWhen Uploading to PyPi:Repeat steps above with these important differences:Usepython3 -m twine upload dist/*to upload to PyPi. You do not need to specify--repository testpypiwhen uploading to PyPi.Login username will be the same. However, remember to use PyPi's login password/API token,notTestPyPi's token for the password.Package Version History Documentation:0.0.1: Ran into problems with installing pytest-cov0.0.2: Got error:ERROR: Could not find a version that satisfies the requirement dask[complete] (from risknet) (from versions: none) ERROR: No matching distribution found for dask[complete]0.0.3: Tried moving dask into [options.extras_require] not install_requires. Got error for flake80.0.4: Commented out more packages.
Got error:error: package directory 'lib3/yaml' does not existandmetadata-generation-failed.0.0.5: Moved types-YAML into options, still get'lib3/yaml does not exist'error :/0.0.6: we're removing YAML as a test
Got errorERROR: Could not find a version that satisfies the requirement typing (from risknet) (from versions: none) ERROR: No matching distribution found for typing0.0.10: reverting to version 0.0.6, checking if it works --success! But still errorNo matching distribution found for typing0.0.11: what happens when we remove typing (since it's part of stdlib in python >3.5)?0.0.12: commented out all packages added by EC0.0.13: Got new error:
`ERROR: Cannot install pandas==1.3.4 and risknet==0.0.12 because these package versions have conflicting dependencies.The conflict is caused by:
risknet 0.0.12 depends on numpy
pandas 1.3.4 depends on numpy>=1.17.3; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10"`Solution: will try setting python > 3.10SUCCESS! Version 0.0.13 can download from emily's (base)Caveats:Probably can only work because it has all dependencies already installed in the env (it threw an error when I tried to run it in risknet_test)Can only importrisknet.utils.label_prepsincerisknet.utils.encoder, etc. have local imports to different .py files which Python can't read (??)
But Running>>> import risknet.utils.label_prep as label_prep, >>> label_prep.label_proc(fm_root, data)works!!0.0.14: change setup.py toif __name__ == "__main__: setup().0.0.16: try compiling on base environment (python 3.12, pip 23.2)