Package: baskref
BaskRef (Basketball Scraper)BaskRef is a tool to scrape basketball Data from the web.The goal of this project is to provide a data collection utility for
NBA basketball data. The collection strategy is to scrape data fromhttps://www.basketball-reference.com.
The data can then be saved into a csv to be used by a different utility.About the PackageWhat data are we collecting?games & game stats (in depth stats of the games)players game statsAll datasets are available to be collected:by day (all games in one day)by whole season (regular + playoffs)by playoffsFuture Collections (Not yet implemented)players meta data (Not Implemented)game logs (Not Implemented)How to Install & Run the Package?Install the projectpipinstallbaskref# optional set logging level. Default value is INFOexportLOG_LEVEL=DEBUG# INFO, DEBUG, ERRORScrape Game DataScrape all games for the 7th of January 2022.baskref-tg-d2022-01-07-fpdatasets# python -c "from baskref import run_baskref; run_baskref()" -t g -d 2022-01-07 -fp datasetsScrape all games for the 2006 NBA season (regular season + playoffs).baskref-tgs-y2006-fpdatasets# python -c "from baskref import run_baskref; run_baskref()" -t gs -y 2006 -fp datasetsScrape all games for the 2006 NBA playoffs.baskref-tgp-y2006-fpdatasets# if you don't install the package# python -c "from baskref import run_baskref; run_baskref()" -t gp -y 2006 -fp datasetsScrape Game URLs only# simply add "u" to any of the three scraping types:# g -> gu, gs -> gsu, gp -> gpubaskref-tgu-d2022-01-07-fpdatasetsScrape Player Stats Data# simply add "pl" to any of the three scraping types:# g -> gpl, gs -> gspl, gp -> gpplbaskref-tgpl-d2022-01-07-fpdatasetsScrape Using a ProxyUse proxy for scraping.baskref-tg-d2022-01-07-fpdatasets-phttp://someproxy.comHow to Use the Package?Install requirementspipinstall-rrequirements.txtData Collection UtilityThis refers to the scraping functionalities.For any mode of collection first you need to import and initialize
the below classes.frombaskref.data_collectionimport(BaskRefUrlScraper,BaskRefDataScraper,)url_scraper=BaskRefUrlScraper()data_scraper=BaskRefDataScraper()# optionally you can set a proxyproxy_url_scraper=BaskRefUrlScraper("http://someproxy.com")proxy_data_scraper=BaskRefDataScraper("http://someproxy.com")The BaskRefDataScraper.get_games_data returns a list of dictionaries.Collect games for a specific dayfromdatetimeimportdategame_urls=url_scraper.get_game_urls_day(date(2022,1,7))game_data=data_scraper.get_games_data(game_urls)Collect games for a specific season (regular + playoffs)game_urls=url_scraper.get_game_urls_year(2006)game_data=data_scraper.get_games_data(game_urls)Collect games for a specific postseasongame_urls=url_scraper.get_game_urls_playoffs(2006)game_data=data_scraper.get_games_data(game_urls)Collect player stats for for a specific dayfromdatetimeimportdategame_urls=url_scraper.get_game_urls_day(date(2022,1,7))pl_stats_data=data_scraper.get_player_stats_data(game_urls)Data Saving PackageThis refers to the saving of the data.Save a list of dictionaries to a CSV file.importosfrombaskref.data_saving.file_saverimportsave_file_from_listsave_path=os.path.join('datasets','file_name.csv')save_file_from_list(game_data,save_path)How to Run Tests?Run all tests with PytestpytestRun all tests with coveragecoverage run --source=baskref -m pytest
coverage report --omit="*/test*" -m --skip-emptyCode FormatingThe code base uses black for automatic formating.
the configuration for black is stored in pyproject.toml file.# run black over the entire code baseblack.LintingThe code base uses pylint and mypy for code linting.Pylintthe configuration for pylint is stored in .pylintrc file.# run pylint over the entire code basepylint--recursive=y./MyPythe configuration for mypy is stored in pyproject.toml file.# run mypy over the entire code basemypybaskrefBonusPrepare project for developmentCreate Virtual EnvironmentYou might want to use a virtual environment for executing the project.this is an optional step (if skipping go straight to step 2)Create a new virtual environemntpython -m venv venv  # The second parameter is a path to the virtual env.Activate the new virtual environment# Windows
.\venv\Scripts\activate

# Unix
source venv/bin/activateLeaving the virtual environmentdeactivateInstall all the dev requirementspip install -r requirements_dev.txt

# uninstall all packages Windows
pip freeze > unins && pip uninstall -y -r unins && del unins

# uninstall all packages linux
pip freeze | xargs pip uninstall -yInstall the pre-commit hookpre-commit installPrepare a new VersionThis section describes some of the steps when preparing a new baskref version.adjust the pyproject.toml fileversiondependenciesinstall project locally and test itpython -m build
pip install .publish project to test.pypipip install --upgrade twine
twine upload --repository testpypi dist/*
# install from test.pypi
pip install --index-url https://test.pypi.org/simple/ baskrefpublish a new versiontwine upload dist/*ContributorsDominik Zulovec Sajovic