Package: data-loader-plugin
Data Loader Plugin - PythonTable of Content (ToC)Data Loader Plugin - PythonTable of Content (ToC)OverviewReferencesPython modulePython virtual environmentsInstallationClone this Git repositoryPython environmentUsageInstall thedata-loader-pluginmoduleInstall in the Python user spaceInstallation in a dedicated Python virtual environmentUsedata-loader-pluginas a module from another Python programDevelopment / ContributionTest the data loader plugin Python moduleTable of contents generated with markdown-tocOverviewThedata loader plugin,
aims at supporting running programs (e.g., API service backends) when
downloading data from cloud services such asAWS S3. It provides a base Python library,
namelydata-loader-plugin,
offering a few methods to download data files from AWS S3.ReferencesPython moduleGitHub:https://github.com/cloud-helpers/python-plugin-data-loader/tree/master/data_loader_pluginPyPi:https://pypi.org/project/data-loader-plugin/Read the Docs (RTD):https://readthedocs.org/projects/data-loader-plugin/Python virtual environmentsPyenv and pipenv:http://github.com/machine-learning-helpers/induction-python/tree/master/installation/virtual-envInstallationClone this Git repository$mkdir-p~/dev/infra&&\gitclonegit@github.com:cloud-helpers/python-plugin-data-loader.git~/dev/infra/python-plugin-data-loader
$cd~/dev/infra/python-plugin-data-loaderPython environmentIf not already done so, installpyenv, Python 3.9 and,pipandpipenvPyEnv:$gitclonehttps://github.com/pyenv/pyenv.git${HOME}/.pyenv
$cat>>~/.profile2<< _EOF# Pythoneval "\$(pyenv init --path)"_EOF$cat>>~/.bashrc<< _EOF# Pythonexport PYENV_ROOT="\${HOME}/.pyenv"export PATH="\${PYENV_ROOT}/bin:\${PATH}". ~/.profile2if command -v pyenv 1>/dev/null 2>&1theneval "\$(pyenv init -)"fiif command -v pipenv 1>/dev/null 2>&1theneval "\$(pipenv --completion)"fi_EOF$.~/.bashrcPython 3.9:$pyenvinstall3.9.8&&pyenvlocal3.9.8pip:$python-mpipinstall-Upippipenv:$python-mpipinstall-UpipenvUsageInstall thedata-loader-pluginmoduleThere are at least two ways to install thedata-loader-pluginmodule,
in the Python user space withpipand in a dedicated virtual environment
withpipenv.Both options may be installed in parallelThe Python user space (typically,/usr/local/opt/python@3.9on MacOS
or~/.pyenv/versions/3.9.8on Linux) may already have many other modules
installed, parasiting a fine-grained control over the versions of every
Python dependency. If all the versions are compatible, then that option
is convenient as it is available from the whole user space, not just
from this sub-directoryIn the remainder of thatUsage section, it will be assumed
that thedata-loader-pluginmodule has been installed and readily
available from the environment, whether that environment is virtual
or not.
In other words, to adapt the documentation for the case wherepipenvis used, just addpipenv runin front of every Python-related command.Install in the Python user spaceInstall and use thedata-loader-pluginmodule in the user space
(withpip):$python-mpipuninstalldata-loader-plugin
$python-mpipinstall-Udata-loader-pluginInstallation in a dedicated Python virtual environmentInstall and use thedata-loader-pluginmodule in a virtual environment:$pipenvshell(python-...-JwpAHotb)✔python-mpipinstall-Udata-loader-plugin(python-...-JwpAHotb)✔python-mpipinstall-Udata-loader-plugin(python-...-JwpAHotb)✔exitUsedata-loader-pluginas a module from another Python programCheck the data file with the AWS command-line (CLI):$awss3ls--humans3://nyc-tlc/trip\data/yellow_tripdata_2021-07.csv--no-sign-request2021-10-2920:44:34249.3MiByellow_tripdata_2021-07.csvModule import statements:>>>importimportlib>>>fromtypesimportModuleType>>>fromdata_loader_plugin.baseimportDataLoaderBaseCreate an instance of the DataLoaderBase Python class:>>>plugin:ModuleType=importlib.import_module("data_loader_plugin.copyfile")>>>data_loader:DataLoaderBase=plugin.DataLoader(local_path='/tmp/yellow_tripdata_2021-07.csv',external_url='s3://nyc-tlc/trip\ data/yellow_tripdata_2021-07.csv',)>>>data_load_success,message=data_loader.load()Development / ContributionBuild the source distribution and Python artifacts (wheels):$rm-rf_skbuild/build/dist/.tox/__pycache__/.pytest_cache/MANIFEST*.egg-info/
$pipenvrunpythonsetup.pysdistbdist_wheelUpload to Test PyPi (no Linux binary wheel can be uploaded on PyPi):$PYPIURL="https://test.pypi.org"$pipenvruntwineupload-u__token__--repository-url${PYPIURL}/legacy/dist/*
Uploadingdistributionstohttps://test.pypi.org/legacy/
Uploadingdata_loader_plugin-0.0.1-py3-none-any.whl100%|███████████████████████████████████████|23.1k/23.1k[00:02<00:00,5.84kB/s]Uploadingdata-loader-plugin-0.0.1.tar.gz100%|███████████████████████████████████████|23.0k/23.0k[00:01<00:00,15.8kB/s]Viewat:
https://test.pypi.org/project/data-loader-plugin/0.0.1/Upload/release the Python packages onto thePyPi repository:Register the authentication token for access to PyPi:$PYPIURL="https://upload.pypi.org"$pipenvrunkeyringset${PYPIURL}/__token__
Passwordfor'__token__'in'${PYPIURL}/':Register the authentication token for access to PyPi:$pipenvruntwineupload-u__token__--repository-url${PYPIURL}/legacy/dist/*
Uploadingdistributionstohttps://upload.pypi.org/legacy/
Uploadingdata_loader_plugin-0.0.1-py3-none-any.whl100%|███████████████████████████████████████|23.1k/23.1k[00:02<00:00,5.84kB/s]Uploadingdata-loader-plugin-0.0.1.tar.gz100%|███████████████████████████████████████|23.0k/23.0k[00:01<00:00,15.8kB/s]Viewat:
https://pypi.org/project/data-loader-plugin/0.0.1/Note that the documentation is built automatically by ReadTheDocs (RTD)The documentation is available fromhttps://data-loader-plugin.readthedocs.io/en/latest/The RTD project is setup onhttps://readthedocs.org/projects/data-loader-plugin/Build the documentation manually (withSphinx):$pipenvrunpythonsetup.pybuild_sphinx
runningbuild_sphinx
RunningSphinxv4.3.0[autosummary]generatingautosummaryfor:README.md
mystv0.15.2:...,words_per_minute=200)building[mo]:targetsfor0pofilesthatareoutofdate
building[html]:targetsfor1sourcefilesthatareoutofdate
updatingenvironment:[newconfig]1added,0changed,0removed
readingsources...[100%]README
...
lookingfornow-outdatedfiles...nonefound
picklingenvironment...donecheckingconsistency...donepreparingdocuments...donewritingoutput...[100%]README
...
buildsucceeded.

TheHTMLpagesareinbuild/sphinx/html.Re-generate the Python dependency files (requirements.txt)
for the CI/CD pipeline (currently Travis CI):$pipenv--rm;rm-fPipfile.lock;pipenvinstall;pipenvinstall--dev
$gitaddPipfile.lock
$pipenvlock-r>ci/requirements.txt
$pipenvlock--dev-r>ci/requirements-dev.txt
$gitaddci/requirements.txtci/requirements-dev.txt
$gitcommit-m"[CI] Upgraded the Python dependencies for the Travis CI pipeline"Test the data loader plugin Python moduleEnter into thepipenvShell:$pipenvshell(python-...-iVzKEypY)✔python-V
Python3.9.8Uninstall any previously installeddata-loader-pluginmodule/library:(python-...-iVzKEypY)✔python-mpipuninstalldata-loader-pluginLaunch a simple test withpytest(python-iVzKEypY)✔python-mpytesttests===================testsessionstarts==================platformdarwin--Python3.9.8,pytest-6.2.5,py-1.11.0,pluggy-1.0.0
rootdir:~/dev/infra/python-plugin-data-loader
plugins:cov-3.0.0
collected3items

tests/test_copyfile.py.[33%]tests/test_s3.py..[100%]======================3passedin1.22s==================Exit thepipenvShell:(python-...-iVzKEypY)✔exit