Package: dawson-college-pyscrapper
Dawson College PyScrapper v1.1.1A Python module which contains useful functions to help scrape data fromDawson Collegewhich is a CEGEP in Montreal Quebec Canada.FeaturesGet information on all the programs offered by Dawson College (ex: Computer Science, Computer Engineering, etc.)Get an estimate of the total number of students enrolledGet the total number of faculty membersGet the general metrics of Dawson College (ex: total number of programs offered, number of programs, number of profiles, number of disciplines, number of special studies, number of general studies, etc.)UsageInstallationpip install git+ssh://git@github.com/jdboisvert/dawson-college-pyscrapperUsing the core functionalityGetting program details for a specific programfromdawson_college_pyscrapper.scrapperimportget_program_detailsprogram_url="https://www.dawsoncollege.qc.ca/programs/program-name"# Get the BeautifulSoup Tag object of the program that is listed on the programs page.listed_program=BeautifulSoup(requests.get(PROGRAMS_LISTING_URL).text.strip(),"html.parser").find("tr")# Get the details of the program at the given URL.program_details=get_program_details(program_url=program_url,listed_program=listed_program)print(program_details)Get details of all programsfromdawson_college_pyscrapper.scrapperimportget_programsprograms=get_programs()forprograminprograms:print(f"Program Name:{program.name}")print(f"Modified Date:{program.modified_date}")print(f"Program Type:{program.program_type}")print(f"Program URL:{program.url}")print("\n")Get the total number of students enrolledfromdawson_college_pyscrapper.scrapperimportget_total_number_of_studentstotal_number_of_students=get_total_number_of_students()print(f"Total number of students:{total_number_of_students}")Get the total number of faculty membersfromdawson_college_pyscrapper.scrapperimportget_total_number_of_faculty_memberstotal_number_of_faculty_members=get_total_number_of_faculty_members()print(f"Total number of faculty members:{total_number_of_faculty_members}")Get the general metrics of Dawson Collegefromdawson_college_pyscrapper.scrapperimportscrapegeneral_metrics=scrape()print(f"Total programs offered:{general_metrics.total_programs_offered}")print(f"Number of programs:{general_metrics.number_of_programs}")print(f"Number of profiles:{general_metrics.number_of_profiles}")print(f"Number of disciplines:{general_metrics.number_of_disciplines}")print(f"Number of special studies:{general_metrics.number_of_special_studies}")print(f"Number of general studies:{general_metrics.number_of_General_studies}")print("\n")print("Year count:")foryear,countingeneral_metrics.total_year_counts.items():print(f"{year}:{count}")print("\n")print("Programs:")forprogramingeneral_metrics.programs:print(f"Program Name:{program.name}")print(f"Modified Date:{program.modified_date}")print(f"Program Type:{program.program_type}")print(f"Program URL:{program.url}")print("\n")More examplesCheck out the examples in the tests directory.DevelopmentGetting started# install pyenv (if necessary)brewinstallpyenvpyenv-virtualenvecho"""export PYENV_VIRTUALENV_DISABLE_PROMPT=1eval "$(pyenvinit--path)"eval "$(pyenvinit-)"eval "$(pyenvvirtualenv-init-)"""">~/.zshrcsource~/.zshrc# create a virtualenvpyenvinstall3.11.1
pyenvvirtualenv3.11.1dawson_college_pyscrapper
pyenvactivatedawson_college_pyscrapper# install dependenciespipinstall-Upip
pipinstall-e".[dev]"Pre-commitA number of pre-commit hooks are set up to ensure all commits meet basic code quality standards.If one of the hooks changes a file, you will need togit addthat file and re-rungit commitbefore being able to continue.To Install:
pre-commit installTestingpytestandtoxare used for testing. Tox is configured to try testing against both Python 3.8 and Python 3.9 if you have them available. If one is missing, Tox will skip it rather than fail out.# just the unit tests against your current python version
pytest

# just the unit tests with a matching prefix
pytest -k test_some_function

# full test suite and code coverage reporting
toxCreditsJeffrey Boisvert (jdboisvert)info.jeffreyboisvert@gmail.com