Package: aio-geojson-query
python-aio-geojson-queryThis library is my attempt at creating a generalized client for theaio-geojson-clientlibrary.Of course, this requires some uncomfortable contorting as the properties in a GeoJson feed are free-form (seerfc7946)This is currently under development, so apologies for the bugs.Installationpip install aio-geojson-queryUsageSee below for examples of how this library can be used. After instantiating a
particular class - feed or feed manager - and supplying the required parameters,
you can callupdateto retrieve the feed data. The return value
will be a tuple of a status code and the actual data in the form of a list of
feed entries specific to the selected feed.Status CodesOK: Update went fine and data was retrieved. The library may still
return empty data, for example because no entries fulfilled the filter
criteria.OK_NO_DATA: Update went fine but no data was retrieved, for example
because the server indicated that there was not update since the last request.ERROR: Something went wrong during the updateParametersParameterDescriptionhome_coordinatesCoordinates (tuple of latitude/longitude)Supported FiltersFilterDescriptionRadiusfilter_radiusRadius in kilometers around the home coordinates in which events from feed are included.Criteriafilter_criteriaArray of filtering conditions.Criteria SyntaxAt this time, criteria are pretty simple and are applied using anoroperator. Therefore, properties matching any function will be a match.Available operators are:==,!=,<,>.The latter two will always compare the property as a float value.MappingsSince this library has no knowledge of the feeds being retrieved, this is used to map property names. Mappings are passed as an additional argument called, of course,mappingsMapping NamesBy default, mappings are as simple as: the first argument is known, in the feed, as the second argument.For instance, let's say you are looking for thedatemapping and, in the feed, that field is calledpublished_date-- you will want to help the library find it by passing this mapping:"date": "published_date"Parametric MappingSometimes, a value can only be extracted from a complex property. For instance,locationmay only be available inside thedescriptionproperty. These mappings are denoted using~~followed by the regular expression that will extract the value:"location": "description~~LOCATION: (?P<{}>[^<]+) <br"Mandatory and default mappingsSome properties are mandatory. For instance, if the library does not find thedateproperty, it will not be able to synchronize the feed properly.If you do not specify mappings for these variables, the library may attempt to guess what their names could be.For instance:PropertyGuessed NamesidEach entry's unique identifierid,guid(@see FeedManager)dateMandatorytime,datedateformatA pseudo mappingHelps the library parse the date property.descriptiondescription,detailsDate ParsingThedateformatpseudo mapping can be:FormatMeaningsecondsThis is an epoch timestamp,millisecondsA timestamp in milliseconds.isoISO-3601 or RFC-3339 compatible format. Allows variations.An arbitrary stringUsed by the library in thestrptimefunction.ExampleI recommend checking outpython-aio-geojson-nsw-rfs-incidentswhich is a library dedicated to retrieving fire incidents information, written by the same author as the library I am creating this middleware for. You may be interested in comparing that library's example code to the one below, which does the same thing (then checks for earthquakes when done):importasynciofromaiohttpimportClientSessionfromaio_geojson_queryimportGeoJsonQueryFeedasyncdefmain()->None:asyncwithClientSession()aswebsession:# NSW Incidents Feed# Home Coordinates: Latitude: -33.0, Longitude: 150.0# Filter radius: 50 km# Filter categories: 'Advice'feed=GeoJsonQueryFeed(websession,"https://www.rfs.nsw.gov.au/feeds/majorIncidents.json",(-33.0,150.0),filter_radius=500,filter_criteria=[['category','==','Advice']],mappings={"dateformat":"iso","date":"pubDate","location":"description~~LOCATION: (?P<{}>[^<]+) <br"})status,entries=awaitfeed.update()print(status)forentryinentries:print("%s[%s]: @%s"%(entry.title,entry.publication_date,entry.location))# Earthquakes, magnitude at least 3, around Los Angelesfeed2=GeoJsonQueryFeed(websession,"https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson",(34.052235,-118.243683),filter_criteria=[['mag','>','3.0']],filter_radius=50,mappings={"dateformat":"milliseconds","date":"updated"})status,entries=awaitfeed2.update()print(status)forentryinentries:print("%s[%s]: @%s"%(entry.title,entry.publication_date,entry.title))asyncio.get_event_loop().run_until_complete(main())Feed ManagerThe Feed Manager helps managing feed updates over time, by notifying the
consumer of the feed about new feed entries, updates and removed entries
compared to the last feed update.If the current feed update is the first one, then all feed entries will be
reported as new. The feed manager will keep track of all feed entries'
external IDs that it has successfully processed.If the current feed update is not the first one, then the feed manager will
produce three sets:Feed entries that were not in the previous feed update but are in the
current feed update will be reported as new.Feed entries that were in the previous feed update and are still in the
current feed update will be reported as to be updated.Feed entries that were in the previous feed update but are not in the
current feed update will be reported to be removed.If the current update fails, then all feed entries processed in the previous
feed update will be reported to be removed.After a successful update from the feed, the feed manager provides two
different dates:last_updatewill be the timestamp of the last update from the feed
irrespective of whether it was successful or not.last_update_successfulwill be the timestamp of the last successful update
from the feed. This date may be useful if the consumer of this library wants
to treat intermittent errors from feed updates differently.last_timestamp(optional, depends on the feed data) will be the latest
timestamp extracted from the feed data.
This requires that the underlying feed data actually contains a suitable
date. This date may be useful if the consumer of this library wants to
process feed entries differently if they haven't actually been updated.SpecifyidWhen in doubt... make sure you specify a mapping forid-- if only one entry is returned by the feed manager when you expect multiple entries, it is likely that the feed entries are not properly identified. If necessary, specify a mapping foridto a property that is unique to each entry. For instance, in the USGS earthquake feed, such an entry iscode.