Package: gtfs-proto
GTFS to Protobuf PackagingThis library / command-line tool introduces a protocol buffers-based format
for packaging GTFS feeds. The reasons for this are:Decrease the size of a feed to 8-10% of the original.Allow for even smaller and easy to apply delta files.The recommended file extension for packaged feeds isgtp.Differences with GTFSThe main thing missing from the packaged feed is fare information.
This is planned to be added, refer tothis ticketto track the implementation progress.Packaged feed does not reflect source tables one to one. This is true
for most tables though. Here's what been deliberately skipped from
the official format:stops.txt:tts_stop_name,stop_url,stop_timezone,level_id.routes.txt:route_url,route_sort_order.trips.txt:block_id.stop_times.txt:stop_sequence,shape_dist_travelled.shapes.txt:shape_pt_sequence,shape_dist_travelled.Feed files ignored are all thefare*.txt,timeframes.txt,pathways.txt,levels.txt,translations.txt,feed_info.txt, andattributions.txt.
Fares are to be implemented, and for other tables it is recommended
to use the original feed.Binary FormatNote that the format is to have significant changes, including renumbering
of fields, until version 1.0 is published.Inside the file, first thing is a little-endian two-byte size for the header
block. Then the header serialized message follows.The header contains a list of sizes for each block. Blocks follow in the order
listed in theBlockenum: first identifiers, then strings, then agencies,
and so on.The same enum is used for keys in theIdReferencemessage, that links
generated numeric ids from this packed feed with the original string ids.If the feed is compressed (marked by a flag in the header), each block is
compressed using the Zstandard algorithm. It proved to be both fast and efficient,
decreasing the size by 70%.Location EncodingFloating-point values are stored inefficiently, hence all longitude and latitudes
are multiplied by 100000 (10^5) and rounded. This allows for one-meter precision,
which is good enough on public transit scales.In addition, when coordinates become lists, we store only a difference with the
last coordinate. This applies to both stops (relative to the previous stop) and
shapes: in latter, coordinates are relative to the previous coordinate, or to
the last one in the previous shape.Routes and TripsThe largest file in every GTFS feed isstop_times.txt. Here, it's missing, with
the data spread between routes and trips. The format also adds itineraries:An itinerary is a series of stops for a route with the same headsign and shape.Note that there is no specific block for itineraries, instead they are packaged
inside corresponding routes. But they still have unique identifiers.Route is the same as in GTFS.Trips reference an itinerary for stops, and add departure and arrival times for
each stop (or start and end time when those are specified withfrequencies.txt).So to find a departure time for a given stop, you find itineraries that contain it,
and from those, routes and trips. You get a departure times list from the trip,
and use addition to get the actual time (since we store just differences with previous
times, with 5-second granularity).DeltasDelta files looks the same as the original, but the header size has its last bit
set (size & 0x8000, note the unsigned integer). After that,GtfsDeltaHeaderfollows, which also has version, date, compression fields, and a list of block sizes.How the blocks are different, is explained in theproto file.Installation and UsageInstalling is simple:pip install gtfs-protoPackaging a feedSee a list of commands the tool provides by running it without arguments:gtfs_proto
gtfs_proto pack --helpTo package a feed, call:gtfs_proto pack gtfs.zip --output city.gtpIn a header, a feed stores an URL of a source zip file, and a date on which
that feed was built. You should specify those, although if the date is "today",
you can skip the argument:gtfs_proto pack gtfs.zip --url https://mta.org/gtfs/gtfs.zip --date 2024-03-19 -o city.gtpWhen setting a pipeline to package feeds regularly, do specify the previous feed
file to keep identifiers from altering, and to keep delta file sizes to a minimum:gtfs_proto pack gtfs.zip --prev city_last.gtp -o city.gtpDeltasDelta, a list differences between two feeds, is made with this obvious command:gtfs_proto delta city_last.gtp city.gtp -o city_delta.gtpCurrently it's to be decided whether a delta requires a different file extension.
Technically the format is almost the same, using the same protocol buffers definition.If you lost an even older file and wish to keep your users updated even from very
old feeds, you can merge deltas:gtfs_proto dmerge city_delta_1-2.gtp city_delta_2-3.gtp -o city_delta_1-3.gtpIt's recommended to avoid merging deltas and store old feeds instead to produce
delta files with thedeltacommand.There is no command for applying deltas: it's on end users to read the file and
apply it straight to their inner database.InformationA packaged feed contains a header and an array of blocks, similar but not exactly mirroring
the original GTFS files. You can see the list, sizes and counts by running:gtfs_proto info city.gtpAny block can be dumped into a series of one-line JSON objects by specifying
the block name:gtfs_proto info city.gtp --block stopsCurrently the blocks areids,strings,agency,calendar,shapes,stops,routes,trips,transfers,networks,areas, andfare_links.There are two additional "blocks" that print numbers from the header:versionanddate. Use these to simplify automation. For example, this is
how you make a version-named copy of the lastest feed:cpcity-latest.gtpcity-$(gtfs_protoinfo-bversion).gtpWhen applicable, you can print just the line for a given identifier,
both for the one from the original GTFS feed, and for a numeric generated one:gtfs_proto info city.gtp -p stops --id 45Of course you can view contents of a delta file the same way.Python LibraryReading GTFS protobuf files is pretty straightforward:importgtfs_protoasgtpfeed=GtfsProto(open('city.gtp','rb'))print(f'Feed built on{feed.header.date}')forstopinfeed.stops:print(f'Stop{stop.stop_id}named "{feed.strings[stop.name]}".')TheGtfsProto(andGtfsDelta) object reads the file header and lazily provides
all blocks as lists or dicts. To read all blocks instantly, use theread_now=Trueargument for the constructor.Parsing shapes and calendar services is not easy, so there are some service
functions, namelyparse_shapeandparse_calendar. The latter returns a list
ofCalendarServicewith all the dates and day lists unpacked, and anoperatesmethod to determine whether the line functions on a given date.All built-in commands use this library, so refer to, for example,delta.pyfor an extended usage tutorial.Author and LicenseThe format and the code were written by Ilya Zverev. The code is published under ISC License,
the the format is CC0 or in a public domain, whatever applies in your country.