Package: partridge
PartridgePartridge is a Python 3.6+ library for working withGTFSfeeds usingpandasDataFrames.Partridge is heavily influenced by our experience atRemixanalyzing and debugging every GTFS feed we could find.At the core of Partridge is a dependency graph rooted attrips.txt. Disconnected data is pruned away according to this graph when reading the contents of a feed.Feeds can also be filtered to create a view specific to your needs. It’s most common to filter a feed down to specific dates (service_id) or routes (route_id), but any field can be filtered.PhilosophyThe design of Partridge is guided by the following principles:As much as possibleFavor speedAllow for extensionSucceed lazily on expensive pathsFail eagerly on inexpensive pathsAs little as possibleDo anything other than efficiently read GTFS files into DataFramesTake an opinion on the GTFS specInstallationpip install partridgeGeoPandas supportpip install partridge[full]UsageSetupimportpartridgeasptginpath='path/to/caltrain-2017-07-24/'ExamplesThe following is a collection of gists containing Jupyter notebooks with transformations to GTFS feeds that may be useful for intake into software applications.Find the busiest week in a feed and reduce its file sizeCombine routes by route_short_nameMerge GTFS with shapefile geometriesMerge multiple agencies into oneRewrite a feed to clean up formatting issuesIf a feed has stop_code, replace the contents of stop_id with the contents of stop_codeDiff the number of service hours in two feedsInvestigate the the distance in meters of each stop to the closest point on a shapeConvert frequencies.txt to an equivalent trips.txtCalculate headway for a stopInspecting the calendarThe date with the most tripsdate,service_ids=ptg.read_busiest_date(inpath)#  datetime.date(2017, 7, 17), frozenset({'CT-17JUL-Combo-Weekday-01'})The week with the most tripsservice_ids_by_date=ptg.read_busiest_week(inpath)#  {datetime.date(2017, 7, 17): frozenset({'CT-17JUL-Combo-Weekday-01'}),#   datetime.date(2017, 7, 18): frozenset({'CT-17JUL-Combo-Weekday-01'}),#   datetime.date(2017, 7, 19): frozenset({'CT-17JUL-Combo-Weekday-01'}),#   datetime.date(2017, 7, 20): frozenset({'CT-17JUL-Combo-Weekday-01'}),#   datetime.date(2017, 7, 21): frozenset({'CT-17JUL-Combo-Weekday-01'}),#   datetime.date(2017, 7, 22): frozenset({'CT-17JUL-Caltrain-Saturday-03'}),#   datetime.date(2017, 7, 23): frozenset({'CT-17JUL-Caltrain-Sunday-01'})}Dates with active serviceservice_ids_by_date=ptg.read_service_ids_by_date(path)date,service_ids=min(service_ids_by_date.items())#  datetime.date(2017, 7, 15), frozenset({'CT-17JUL-Caltrain-Saturday-03'})date,service_ids=max(service_ids_by_date.items())#  datetime.date(2019, 7, 20), frozenset({'CT-17JUL-Caltrain-Saturday-03'})Dates with identical servicedates_by_service_ids=ptg.read_dates_by_service_ids(inpath)busiest_date,busiest_service=ptg.read_busiest_date(inpath)dates=dates_by_service_ids[busiest_service]min(dates),max(dates)#  datetime.date(2017, 7, 17), datetime.date(2019, 7, 19)Reading a feed_date,service_ids=ptg.read_busiest_date(inpath)view={'trips.txt':{'service_id':service_ids},'stops.txt':{'stop_name':'Gilroy Caltrain'},}feed=ptg.load_feed(path,view)Read shapes and stops as GeoDataFramesservice_ids=ptg.read_busiest_date(inpath)[1]view={'trips.txt':{'service_id':service_ids}}feed=ptg.load_geo_feed(path,view)feed.shapes.head()#       shape_id                                           geometry#  0  cal_gil_sf  LINESTRING (-121.5661454200744 37.003512297983...#  1  cal_sf_gil  LINESTRING (-122.3944115638733 37.776439059278...#  2   cal_sf_sj  LINESTRING (-122.3944115638733 37.776439059278...#  3  cal_sf_tam  LINESTRING (-122.3944115638733 37.776439059278...#  4   cal_sj_sf  LINESTRING (-121.9031703472137 37.330157067882...minlon,minlat,maxlon,maxlat=feed.stops.total_bounds#  -122.412076, 37.003485, -121.566088, 37.77639Extracting a new feedoutpath='gtfs-slim.zip'service_ids=ptg.read_busiest_date(inpath)[1]view={'trips.txt':{'service_id':service_ids}}ptg.extract_feed(inpath,outpath,view)feed=ptg.load_feed(outpath)assertservice_ids==set(feed.trips.service_id)FeaturesSurprisingly fast :)Load only what you need into memoryBuilt-in support for resolving service datesEasily extended to support fields and files outside the official spec
(TODO: document this)Handle nested folders and bad data in zipsPredictable type conversionsThank YouI hope you find this library useful. If you have suggestions for
improving Partridge, please open anissue on
GitHub.History1.1.2 (2022-11-23)Code changes:Remove references to deprecated NumPy types (https://github.com/remix/partridge/pull/69- thanks @BlackSpade741!)Switch fromcChardettocharset-normalizerfor Python 3.10 support (https://github.com/remix/partridge/pull/76- thanks @brockhaywood!)Other changes:Miscellaneous improvements to tests, code formatting, and documentation (https://github.com/remix/partridge/pull/61- thanks @invisiblefunnel!)Relocate usage examples from wiki to README (https://github.com/remix/partridge/pull/70- thanks @landonreed!)README tweaks (https://github.com/remix/partridge/pull/74- thanks @chelsey!)Use GitHub Actions for automated testing (https://github.com/remix/partridge/pull/79- thanks @dget!).Note:we now test against Python versions 3.8, 3.9, 3.10, and 3.11.1.1.1 (2019-09-13)Improve file encoding sniffer, which was misidentifying some Finnish/emoji unicode. Thanks to @dyakovlev!1.1.0 (2019-02-21)Addpartridge.load_geo_feedfor reading stops and shapes into GeoPandas GeoDataFrames.1.0.0 (2018-12-18)This release is a combination of major internal refactorings and some minor interface changes. Overall, you should expect your upgrade from pre-1.0 versions to be relatively painless. A big thank you to @genhernandez and @csb19815 for their valuable design feedback. If you still need Python 2 support, please continue using version 0.11.0.Here is a list of interface changes:The classpartridge.gtfs.feedhas been renamed topartridge.gtfs.Feed.The public interface for instantiating feeds ispartridge.load_feed. This function replaces the previously undocumented functionpartridge.get_filtered_feed.A new function has been added for identifying the busiest week in a feed:partridge.read_busiest_dateThe public functionpartridge.get_representative_feedhas been removed in favor of usingpartridge.read_busiest_datedirectly.The public functionpartridge.writers.extract_feedis now available via the top level module:partridge.extract_feed.Miscellaneous minor changes:Character encoding detection is now done by thecchardetpackage instead ofchardet.cchardetis faster, but may not always return the same result aschardet.Zip files are unpacked into a temporary directory instead of reading directly from the zip. These temporary directories are cleaned up when the feed is garbage collected or when the process exits.The code base is now annotated with type hints and the build runsmypyto verify the types.DataFrames are cached in a dictionary instead of thefunctools.lru_cachedecorator.Thepartridge.extract_feedfunction now writes files concurrently to improve performance.0.11.0 (2018-08-01)Fix major performance issue related to encoding detection. Thank you to @cjer for reporting the issue and advising on a solution.0.10.0 (2018-04-30)Improved handling of non-standard compliant file encodingsOnly require functools32 for Python < 3ptg.parsers.parse_dateno longer accepts dates, only strings0.9.0 (2018-03-24)Improves read time for large feeds by adding LRU caching toptg.parsers.parse_time.0.8.0 (2018-03-14)Gracefully handle completely empty files. This change unifies the behavior of reading from a CSV with a header only (no data rows) and a completely empty (zero bytes) file in the zip.0.7.0 (2018-03-09)Fix handling of nested folders and zip containing nested folders.Addptg.get_filtered_feedfor multi-file filtering.0.6.1 (2018-02-24)Fix bug inptg.read_service_ids_by_date. Reported by @cjer in #27.0.6.0 (2018-02-21)Published package no longer includes unnecessary fixtures to reduce the size.Naively write a feed object to a zip file withptg.write_feed_dangerously.Read the earliest, busiest date and itsservice_id’s from a feed withptg.read_busiest_date.Bug fix: Handlecalendar.txt/calendar_dates.txtentries w/o applicable trips.0.6.0.dev1 (2018-01-23)Add support for reading files from a folder. Thanks again @danielsclint!0.5.0 (2017-12-22)Easily build a representative view of a zip withptg.get_representative_feed. Inspired bypeartree.Extract out GTFS zips by agency_id/route_id withptg.extract_{agencies,routes}.Read arbitrary files from a zip withfeed.get('myfile.txt').Removeservice_ids_by_date,dates_by_service_ids, andtrip_counts_by_datefrom the feed class. Instead useptg.{read_service_ids_by_date,read_dates_by_service_ids,read_trip_counts_by_date}.0.4.0 (2017-12-10)Add support for Python 2.7. Thanks @danielsclint!0.3.0 (2017-10-12)Fix service date resolution for raw_feed. Previously raw_feed considered all days of the week from calendar.txt to be active regardless of 0/1 value.0.2.0 (2017-09-30)Add missing edge from fare_rules.txt to routes.txt in default dependency graph.0.1.0 (2017-09-23)First release on PyPI.