Package: scrapd
Extract data fromAPD news site.ScrAPD is a small utility designed to help organizations retrieving traffic fatality data in a friendly manner.InstallationScrAPD requires Python 3.7+ to work.pip install scrapdQuickstartCollect all the data as CSV:scrapd --format csvBy default, scrapd does not display anything until it is done collecting the data. If you want to get some feedback
about the process, you can enable logging, by adding the-vto the command you want to use. Multiple-voptions increase the verbosity. The maximum is 3 (-vvv):scrapd -v --format csvTo save the results to a file, use the shell redirection:scrapd -v --format csv > results.csvNoteThe logs are displayed tostderrand will not appear in the result file generated by the redirection. If you want to
include this information add&2>1.ExamplesRetrieve the traffic fatalities that happened between January 15th 2019 and January 18th 2019, and output the results
injson:scrapd --from "Jan 15 2019" --to "Jan 18 2019" --format json

[
  {
    "case": "19-0161105",
    "crash": 2,
    "date": "2019-01-16",
    "fatalities": [
      {
        "age": 58,
        "dob": "1960-02-15",
        "ethnicity": "White",
        "first": "Ann",
        "gender": "Female",
        "generation": "",
        "last": "Bottenfield-Seago",
        "middle": ""
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-2-3",
    "location": "West William Cannon Drive and Ridge Oak Road",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed.",
    "time": "15:42:00"
  },
  {
    "case": "19-0150158",
    "crash": 1,
    "date": "2019-01-15",
    "fatalities": [
      {
        "age": 31,
        "dob": "1987-07-09",
        "ethnicity": "White",
        "first": "David",
        "gender": "Male",
        "generation": "",
        "last": "Sell",
        "middle": "Hilburn"
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-1-4",
    "location": "10500 block of N IH 35 SB",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.",
    "time": "06:20:00"
  }
]Do the same research but output as CSV:scrapd --from "Jan 15 2019" --to "Jan 18 2019" --format csv


crash,case,date,time,location,first,middle,last,generation,ethnicity,gender,dob,age,link,notes
2,19-0161105,2019-01-16,15:42:00,West William Cannon Drive and Ridge Oak Road,Ann,,Bottenfield-Seago,,White,Female,1960-02-15,58,http://austintexas.gov/news/traffic-fatality-2-3,"The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed."
1,19-0150158,2019-01-15,06:20:00,10500 block of N IH 35 SB,David,Hilburn,Sell,,White,Male,1987-07-09,31,http://austintexas.gov/news/traffic-fatality-1-4,The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.Retrieve all the traffic fatalities from 2019 (as of Jan 20th 2019) in json, and enabling the logging to follow the progress
of the process:scrapd -v --from "1 1 2019" --to "Jan 20 2019" --format json

Fetching page 1...
Fetching page 2...
Fetching page 3...
Fetching page 4...
Fetching page 5...
Fetching page 6...
Fetching page 7...
Fetching page 8...
Fetching page 9...
Fetching page 10...
Fetching page 11...
Total: 3
[
  {
    "case": "19-0081623",
    "crash": 3,
    "date": "2019-01-08",
    "fatalities": [
      {
        "age": 15,
        "dob": "2003-02-18",
        "ethnicity": "Hispanic",
        "first": "Jesus",
        "gender": "Male",
        "generation": "",
        "last": "Servantez",
        "middle": ""
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-3-4",
    "location": "3600 block of South Capital of Texas Highway SB",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that the driver of a silver, 2018 KIA was traveling in the center lane of the 3600 block of South Capital of Texas Highway SB when the car collided with Jesus Servantez, a pedestrian in the roadway. The driver remained at the scene and told investigators that he did not see Servantez prior to the crash.\n\n\tJesus Servantez was transported to Saint David\u2019s South Austin Medical Center where he succumbed to his injuries on January 21, 2019. No charges are expected to be filed.",
    "time": "21:37:00"
  },
  {
    "case": "19-0161105",
    "crash": 2,
    "date": "2019-01-16",
    "fatalities": [
      {
        "age": 58,
        "dob": "1960-02-15",
        "ethnicity": "White",
        "first": "Ann",
        "gender": "Female",
        "generation": "",
        "last": "Bottenfield-Seago",
        "middle": ""
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-2-3",
    "location": "West William Cannon Drive and Ridge Oak Road",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed.",
    "time": "15:42:00"
  },
  {
    "case": "19-0150158",
    "crash": 1,
    "date": "2019-01-15",
    "fatalities": [
      {
        "age": 31,
        "dob": "1987-07-09",
        "ethnicity": "White",
        "first": "David",
        "gender": "Male",
        "generation": "",
        "last": "Sell",
        "middle": "Hilburn"
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-1-4",
    "location": "10500 block of N IH 35 SB",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.",
    "time": "06:20:00"
  }
]The “Case” field is assigned by the Austin Police Department, while the “ID” field is a key assigned by ScrAPD to label a separate record for each deceased person.Speed and accuracyScrAPD executes all the requests in an asynchronous manner. As a result it goes very fast.It parses the information using both the text of the report itself and the Twitter tweet stored in the page metadata.
Combining these two methods provides a high degree of confidence in the parsing and allows us to reach more than95% of success rate.Some statistics:99% of entries detected95% of entries correctly parsed5% of entries failed the parsing onlypartially0 complete failureprocessing time: ~2m00s for ~140 entriesWho uses ScrAPD?The AustinPedestrian Advisory Councilused ScrAPD to compile a detailed presentation of the status of the traffic deaths in Austin, TX:2018 PAC retrospective presentationThe following organizations also use ScrAPD:Walk AustinFarm&CityBicycle Advisory CouncilSee more athttp://docs.scrapd.org/who-is-using-us/Related resourcesThe Statesman published an article aboutpedestrian deaths reaching a record high level in 2018.NPR’s view on the same subjectPedestrian Deaths Reach Highest Level In Decades