Package: road-data-scraper
Road Data ScraperThe Road Data Scraper is a comprehensive Python tool designed to extract and process data from the WebTRIS Traffic Flow API. It is a complete rewrite of theONS Road Data Pipelineoriginally written in R. You can refer to the documentation of the ONS Road Data Pipelinehereand to the WebTRIS Traffic Flow APIhere.Developer UsageTo get started with the Road Data Scraper, ensure Python 3.9 is installed on your machine. If you're using Anaconda or Miniconda, you can create a virtual environment with Python 3.9 using:conda create --name py39 python=3.9Clone the repository:git clone https://github.com/dombean/road_data_scraper.gitNavigate into the cloned repository:cd road_data_scraper/Install the package in editable mode:pip install -e .Change directory into the package folder:cd src/road_data_scraper/Adjust the config.ini file according to your requirementsExecute the script:python main.pyorpython3 main.pyProject StructureThe Road Data Scraper project has the following structure:├── config.ini
├── main.py
├── setup.cfg
├── setup.py
├── pyproject.toml
├── api_main.py
├── Dockerfile
├── src
│ ├── road_data_scraper
│ │ ├── steps
│ │ │ ├── download.py
│ │ │ ├── file_handler.py
│ │ │ └── metadata.py
│ │ └── report
│ │ ├── report.py
│ │ └── road_data_report_template.ipynb
├── tests
├── requirements.txt
├── requirements_dev.txt
├── tox.ini
└── README.mdThe project directory contains the following components:config.ini: Configuration file for the Road Data Scraper pipeline.main.py: Main script to run the Road Data Scraper pipeline.setup.cfg&setup.py&pyproject.toml: Configuration file for the Python package.api_main.py: Main script for running the Road Data Scraper as a FastAPI application.Dockerfile: Dockerfile for building a Docker image of the Road Data Scraper.src: Directory containing the source code of the Road Data Scraper.road_data_scraper: Package directory.steps: Module directory containing the main modules for data scraping.download.py: Module for scraping data from the WebTRIS Highways England API.file_handler.py: Module for handling files and directories in the data scraping process.metadata.py: Module for generating metadata for the road traffic sensor data scraping pipeline.report: Module directory for generating HTML reports.report.py: Module for generating HTML reports based on a template Jupyter notebook.road_data_report_template.ipynb: Template Jupyter notebook for generating the HTML report.requirements.txt: File listing the required Python packages for the project.requirements_dev.txt: File listing additional development-specific requirements for the project.tox.ini: Configuration file for running tests using the Tox testing tool.tests: Directory containing test files for the project.README.md: Documentation file providing an overview and instructions for using the Road Data Scraper.The main functionality of the Road Data Scraper resides in thesrc/road_data_scraper/stepsdirectory, where the core modules for data scraping, file handling, metadata generation, and report generation are located. Theroad_data_report_template.ipynbfile, which serves as the template for generating HTML reports, is placed inside thesrc/road_data_scraper/reportdirectory.The additional component,Dockerfile, is located in the root directory. It is used for building a Docker image of the Road Data Scraper, allowing for easy deployment and containerization of the application.Adjusting the Config File (config.ini)There are several configurable options in the config.ini file:start_date: Specify a start date in the format%Y-%m-%d, e.g, "2021-01-01".end_date: Specify an end date in the format%Y-%m-%d, e.g, "2021-01-31".test_run: Set toTruefor testing the pipeline (runs on a subset of available URLs) andFalsefor a complete data download.generate_report: Set toTrueto generate a HTML report showcasing the Active and Inactive IDs for each road sensor -- MIDAS, TMU, and TAME.output_path: Provide a path to save the outputs generated by the Road Data Scraper Pipeline; for example, "/home/user/Documents/"rm_dir: Set toTrueif you're using a Google Cloud VM Instance and you don't want to store the data on the VM (assuming you setgcp_storage=True).Google Cloud (GCP) Storage OptionsTo save output data to a Google Cloud bucket, adjust the following settings:gcp_storage: Set toTrueto save the data generated by the pipeline to a Google Cloud bucket.gcp_credentials: Provide the path to your GCP credentials json file, e.g., "/home/user/gcp_credentials.json".gcp_bucket_name: Provide the name of your GCP bucket, e.g., "road_data_scraper_bucket".gcp_blob_name: Provide the name of the folder in the GCP bucket where you want the pipeline to save the data, e.g., "landing_zone".Google Cloud VM Instance SetupFollow the below steps to set up the Road Data Scraper on a Google Cloud VM instance:Login toGoogle Cloud Platformand click onCompute Enginein the left side-bar.Then, in the left side-bar, click onMarketplaceand search forUbuntu 20.04 LTS (Focal), then, clickLAUNCH.Name the instance appropriately; clickCOMPUTE-OPTIMISED(note: leave the defaults -- 4 vCPU, 16 GB memory); underFirewall, clickAllow HTTPS traffic; and finallyCREATEthe VM instance.SSH into the VM instance.Run the following commands:sudo apt-get update && sudo apt-get dist-upgrade -y && sudo apt-get install python3-pip -y && sudo apt-get install wget -yPip install the road_data_scraper Package using the command:pip install road_data_scraperUpload GCP json credentials file.Download theconfig.inifile using the command:wget https://raw.githubusercontent.com/dombean/road_data_scraper/main/src/road_data_scraper/config.iniDownload therunner.pyfile using the command:wget https://raw.githubusercontent.com/dombean/road_data_scraper/main/runner.pyOpenrunner.pyand put in the absolute path to theconfig.inifile.Change config.ini parameters accordingly, see README section:Adjusting the Config File (config.ini).Run the Road Data Scraper Pipeline using the command:python3 runner.pyLogin toGoogle Cloud Platformand click onCompute Enginein the left side-bar.Click onMarketplaceand search forUbuntu 20.04 LTS (Focal). Click LAUNCH__.Name your instance, selectCOMPUTE-OPTIMISED(default settings are recommended), enable HTTPS traffic underFirewall, andCREATEthe VM instance.SSH into the created VM instance.Update your instance and install necessary packages:sudo apt-get update && sudo apt-get dist-upgrade -y && sudo apt-get install python3-pip -y && sudo apt-get install wget -yInstall the road_data_scraper Package:pip install road_data_scraperUpload your GCP json credentials file.Download theconfig.inifile:wget https://raw.githubusercontent.com/dombean/road_data_scraper/main/src/road_data_scraper/config.iniDownload therunner.pyfile:wget https://raw.githubusercontent.com/dombean/road_data_scraper/main/runner.pyUpdate the path to theconfig.inifile inrunner.py.Adjust the parameters in theconfig.inifile as per your requirements. Refer to the README section onAdjusting the Config Filefor more information.Run the Road Data Scraper Pipeline:python3 runner.pyGoogle Cloud Run SetupEnsure Docker and Google Cloud SDK are installed locally. You will also need to authenticate Google Cloud and Docker.Login to Google Cloud on the command line:gcloud auth loginConfigure Google Cloud Project on the command line:gcloud config set project <project-name>Configure Docker and Google Cloud Credentials:gcloud auth configure-dockerClone the repository:git clone https://github.com/dombean/road_data_scraper.gitChange directory into the cloned repository:cd road_data_scraper/Download your Google CloudJSON Credentialsinto the repository.Build the Docker Image:docker build -t road-data-scraper -f Dockerfile .Test the Docker Image:docker run -it --env PORT=80 -p 80:80 road-data-scraperTag the Docker Image:docker tag road-data-scraper eu.gcr.io/<project-name>/road-data-scraperPush the Docker Image:docker push eu.gcr.io/<project-name>/road-data-scraperDeploy the Docker Image on Google Cloud Run:gcloud run deploy road-data-scraper --image eu.gcr.io/<project-name>/road-data-scraper --platform managed --region europe-west2 --timeout "3600" --cpu "4" --memory "16Gi" --max-instances "3"