Package: pyle38
Report Bug·Request FeatureAbout The ProjectThis is an asynchonous Python client for Tile38 that allows for fast and easy
interaction with the worlds fastest in-memory
geodatabaseTile38.Check out my introductory blog post about the project
on myblog.Exampleimportasynciofrompyle38importTile38asyncdefmain():tile38=Tile38(url="redis://localhost:9851",follower_url="redis://localhost:9851")awaittile38.set("fleet","truck").point(52.25,13.37).exec()response=awaittile38.follower().within("fleet").circle(52.25,13.37,1000).asObjects()assertresponse.okprint(response.dict())awaittile38.quit()asyncio.run(main())>{"ok":True,"elapsed":"48.8µs","objects":[{"object":{"type":"Point","coordinates":[13.37,52.25]},"id":"truck"}],"count":1,"cursor":0}Example IPythonIn[1]:%autoawaitasyncioIn[2]:frompyle38importTile38In[3]:tile38=Tile38(url='redis://localhost:9851',follower_url='redis://localhost:9852')In[4]:awaittile38.set("fleet","truck").point(52.25,13.37).exec()Out[4]:JSONResponse(ok=True,elapsed='51.9µs',err=None)In[5]:response=awaittile38.within("fleet")...:.circle(52.25,13.37,1000)...:.asObjects()In[6]:print(response.dict()){"ok":True,"elapsed":"46.3µs","objects":[{"object":{"type":"Point","coordinates":[13.37,52.25]},"id":"truck"}],"count":1,"cursor":0}Featuresfully typed using mypy and pydanticlazy clientoptional build-in leader/follower logiceasy to use and integratenext to no external dependenciesBuilt WithaioredispydanticGetting StartedRequirementsPython==^3.10.0Installationpipinstallpyle38Or usingPoetrypoetryaddpyle38Now start your Tile38 instance(s) either locally using Docker and docker-compose.docker-composeupOr follow the installation instruction ontile38.comto start your install Tile38 and start a server locally.If you already have a Tile38 instance running somewhere read on.Importfrompyle38importTile38tile38=Tile38('redis://localhost:9851')Leader / FollowerWhen it comes to replication, Tile38 follows a leader-follower model. The leader
receives all commands thatSETdata, a follower on the other hand
isread-onlyand can only query data. For more on replication in Tile38
refer to theofficial documentation.This client is not meant to setup a replication, because this should happen in
your infrastructure. But it helps you to specifically execute commands on
leader or follower. This way you can make sure that the leader always has
enough resources to executeSETs and firegeofenceevents onwebhooks.For now you can set one followerurlto set alongside the leaderurl.frompyle38.tile38importTile38tile38=Tile38('redis://localhost:9851','redis://localhost:9851')Once the client is instantiated with a follower, commands can be explicitly
send to the follower, by adding.follower()to your command chaining.awaittile38.follower().get('fleet','truck1').asObject()PaginationTile38 has hidden limits set for the amount of objects that can be returned
in one request. For most queries/searches this limit is set to100.
This client gives you the option to either paginate the results yourself by
add.cursor()and.limit()to your queries, or it abstracts pagination
away from the user by adding.all().Let's say yourfleetinBerlinextends 100 vehicles, thenawaittile38.within('fleet').get('cities','Berlin').asObjects()will only return 100 vehicle objects. Now you can either get the rest
by setting the limit to the amount of vehicles you have in the city and get
them all.awaittile38.within('fleet').limit(10000).get('cities','Berlin').asObjects()Or, you can paginate the results in multiple concurrent requests to fit your requirements.awaittile38.within('fleet').cursor(0).limit(100).get('cities','Berlin').asObjects()awaittile38.within('fleet').cursor(100).limit(100).get('cities','Berlin').asObjects()awaittile38.within('fleet').cursor(200).limit(100).get('cities','Berlin').asObjects();ResponsesFor now, every Tile38 commands response is parsed into a pydantic object
for validation and type safety.response=awaittile38.set('fleet','truck1').point(52.25,13.37).exec()print(response.dict())>{'ok':True,'elapsed':'40.7µs'}response=awaittile38.get('fleet','truck1').asObject()print(response.ok)>Trueprint(response.object)>{'type':'Point','coordinates':[13.37,52.25]}print(response.dict())>{'ok':True,'elapsed':'29.3µs','object':{'type':'Point','coordinates':[13.37,52.25]}}CommandsKeysSETSet the value of an id. If a value is already associated to that key/id,
it'll be overwritten.awaittile38.set('fleet','truck1').fields({"maxSpeed":90,"milage":90000}).point(33.5123,-112.2693).exec()# with z coordinateawaittile38.set('fleet','truck2').point(33.5123,-112.2693,100).exec()awaittile38.set('fleet','truck1:driver').string('John Denton').exec()Optionscommanddescription.fields()Optional additional fields.MUST BEnumerical.ex(value)Set the specified expire time, in seconds..nx()Only set the object if it does not already exist..xx()Only set the object if it already exist.Inputcommanddescription.point(lat, lon)Set a simple point in latitude, longitude.point(lat, lon, z)Set a simple point in latitude, longitude, height.bounds(minlat, minlon, maxlat, maxlon)Set as minimum bounding rectangle.object(feature)Set as feature.hash(geohash)Set as geohash.string(value)Set as string. To retrieve string values you can use.get(),scan()or.search()FSETSet the value for one or more fields of an object.awaittile38.fset('fleet','truck1',{"maxSpeed":90,"milage":90000})Optionscommanddescription.xx()FSETreturns error if fields are set on non-existing ids.xx()options changes this behaviour and instead returns0if id does not exist. If key does not existFSETstill returns errorGETGet the object of an id.awaittile38.get('fleet','truck1').asObject()Get a string object.awaittile38.set('fleet','truck1:driver').string('John').exec()awaittile38.get('fleet','truck1:driver').asStringObject()Optionscommanddescription.withfields()will also return thefieldsthat belong to the object. Field values that are equal to zero are omitted.Outputcommanddescription.asObject()(default) get as object.asBounds()get as minimum bounding rectangle.asHash(precision)get as hash.asPoint()get as point.asStringObject()get a stringDELRemove a specific object by key and id.awaittile38.del('fleet','truck1')PDELRemove objects that match a given pattern.awaittile38.pDel('fleet','truck*')DROPRemove all objects in a given key.awaittile38.drop('fleet')BOUNDSGet the minimum bounding rectangle for all objects in a given keyawaittile38.bounds('fleet')EXPIRESet an expiration/time to live in seconds of an object.awaittile38.expire('fleet','truck1',10)TTLGet the expiration/time to live in seconds of an object.awaittile38.ttl('fleet','truck1',10)PERSISTRemove an existing expiration/time to live of an object.awaittile38.persist('fleet','truck1')KEYSGet all keys matching a glob-style-pattern. Pattern defaults to'*'awaittile38.keys()STATSReturn stats for one or more keys.
The returnedstatsarray contains one or more entries, depending on the number of keys in the request.awaittile38.stats('fleet1','fleet2')Returnscommanddescriptionin_memory_sizeestimated memory size in bytesnum_objectsobjects in the given keynum_pointsnumber of geographical objects in the given keyJSET/JSET/JDELSet a value in a JSON document.
JGET, JSET, and JDEL allow for working with JSON stringsawaittile38.jset('user',901,'name','Tom')awaittile38.jget('user',901)>{'name':'Tom'}awaittile38.jset('user',901,'name.first','Tom')awaittile38.jset('user',901,'name.first','Anderson')awaittile38.jget('user',901)>{'name':{'first':'Tom','last':'Anderson'}}awaittile38.jdel('user',901,'name.last')awaittile38.jget('user',901);>{'name':{'first':'Tom'}}RENAMERenames a collectionkeytonewkey.Optionscommanddescription.nx()Default: false. Changes behavior on how renaming acts ifnewkeyalready existsIf thenewkeyalready exists it will be deleted prior to renaming.awaittile38.rename('fleet','newfleet',false)If thenewkeyalready exists it will do nothing.awaittile38.rename('fleet','newfleet',true)SearchSearches areTile38bread and butter. They are what make Tile38 a ultra-fast, serious and cheap alternative toPostGISfor a lot of use-cases.WITHINWITHINsearches a key for objects that are fully contained within a given bounding area.awaittile38.within('fleet').bounds(33.462,-112.268,33.491,-112.245)>{"ok":true,"objects":[{"id":"1","object":{"type":"Feature","geometry":{"type":"Point","coordinates":[-112.2693,33.5123]},"properties":{}}}],"count":1,"cursor":1,"elapsed":"72.527µs"}awaittile38.within('fleet').nofields().asCount()>{"ok":true,"count":205,"cursor":0,"elapsed":"2.078168µs"}awaittile38.within('fleet').nofields().where("maxspeed",100,120).asCount()>{"ok":true,"count":80,"cursor":0,"elapsed":"2.078168µs"}awaittile38.within('fleet').get('warehouses','Berlin').asCount();>{"ok":true,"count":50,"cursor":0,"elapsed":"2.078168µs"}Optionscommanddescription.cursor(value)used to iterate through your search results. Defaults to0if not set explicitly.limit(value)limit the number of returned objects. Defaults to100if not set explicitly.nofields()if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields..match(pattern)Match can be used to filtered objects considered in the search with a glob pattern..match('truck*')e.g. will only consider ids that start withtruckwithin your key..sparse(value)cautionseems bugged since Tile38 1.22.6. Accepts values between 1 and 8. Can be used to distribute the results of a search evenly across the requested area..where(fieldname, min value, max value)filter output by fieldname and values..where_expr(expr)filter output withfilter-expression..buffer(value)Apply a buffer around area formats to increase the search area by x metersOutputscommanddescription.asObjects()return as array of objects.asBounds()return as array of minimum bounding rectangles:{"id": str,"bounds":{"sw":{"lat": float,"lon": float},"ne":{"lat": float,"lon": float}}}.asCount()returns a count of the objects in the search.asHashes(precision)returns an array of{"id": str,"hash": str}.asIds()returns an array of ids.asPoints()returns objects as points:{"id": str,"point":{"lat": float,"lon": float}. If the searched key is a collection ofPolygonobjects, the returned points are the centroids.Querycommanddescription.get(key, id)Search a given stored item in a collection..circle(lat, lon, radius)Search a given circle of latitude, longitude and radius..bounds(minlat, minlon, maxlat, maxlon)Search a given bounding box..hash(value)Search a givengeohash..quadkey(value)Search a givenquadkey.tile(x, y, z)Search a giventile.object(value)Search a given GeoJSON polygon feature..sector(lat, lon, radius, bearing1, bearing2)Search a given Sector polygon feature.INTERSECTSIntersects searches a key for objects that are fully contained within a given bounding area, but also returns those that intersect the requested area.
When used to search a collection of keys consisting ofPointobjects (e.g. vehicle movement data) it works like a.within()search asPointscannot intersect.
When used to search a collection of keys consisting ofPolygonorLineStringit also returns objects, that only partly overlap the requested area.awaittile38.intersects('warehouses').hash('u33d').asObjects()awaittile38.intersects('fleet').get('warehouses','Berlin').asIds()>{"ok":true,"ids":["truck1"],"count":1,"cursor":0,"elapsed":"2.078168ms"}awaittile38.intersects('warehouses').hash('u33d').where("maxweight",1000,1000).asCounts()>{"ok":true,"count":80,"cursor":0,"elapsed":"2.078168µs"}Optionscommanddescription.clip()Tells Tile38 to clip returned objects at the bounding box of the requested area. Works with.bounds(),.hash(),.tile()and.quadkey().cursor(value)used to iterate through your search results. Defaults to0if not set explicitly.limit(value)limit the number of returned objects. Defaults to100if not set explicitly.nofields()if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields..match(pattern)Match can be used to filtered objects considered in the search with a glob pattern..match('warehouse*')e.g. will only consider ids that start withwarehousewithin your key..sparse(value)cautionseems bugged since Tile38 1.22.6. Accepts values between 1 and 8. Can be used to distribute the results of a search evenly across the requested area..where(fieldname, min value, max value)filter output by fieldname and values..where_expr(expr)filter output withfilter-expression..buffer(value)Apply a buffer around area formats to increase the search area by x metersOutputscommanddescription.asObjects()return as array of objects.asBounds()return as array of minimum bounding rectangles:{"id": str,"bounds":{"sw":{"lat": float,"lon": float},"ne":{"lat": float,"lon": float}}}.asCount()returns a count of the objects in the search.asHashes(precision)returns an array of{"id": str,"hash": str}.asIds()returns an array of ids.asPoints()returns objects as points:{"id": str,"point":{"lat": float,"lon": float}. If the searched key is a collection ofPolygonobjects, the returned points are the centroids.Querycommanddescription.get(key, id)Search a given stored item in a collection..circle(lat, lon, radius)Search a given circle of latitude, longitude and radius..bounds(minlat, minlon, maxlat, maxlon)Search a given bounding box..hash(value)Search a givengeohash..quadkey(value)Search a givenquadkey.tile(x, y, z)Search a giventile.object(value)Search a given GeoJSON polygon feature..sector(lat, lon, radius, bearing1, bearing2)Search a given Sector polygon feature.Nearbyawaittile38.set('fleet','truck1').point(33.5123,-112.2693).fields({"maxspeed":100}).exec()awaittile38.nearby('fleet').where("maxspeed",100,100).point(33.5124,-112.2694).asCount()>{"ok":true,"count":1,"cursor":0,"elapsed":"42.8µs"}awaittile38.nearby('fleet').point(33.5124,-112.2694,10).asCount//becausetruck1isfurtherawaythan10m>{"ok":true,"fields":["maxspeed"],"count":0,"cursor":0,"elapsed":"36µs"}Optionscommanddescription.distance()Returns the distance inmetersto the object from the query.point().cursor(value)used to iterate through your search results. Defaults to0if not set explicitly.limit(value)limit the number of returned objects. Defaults to100if not set explicitly.nofields()if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields..match(pattern)Match can be used to filtered objects considered in the search with a glob pattern..match('warehouse*')e.g. will only consider ids that start withwarehousewithin your key..sparse(value)cautionseems bugged since Tile38 1.22.6. Accepts values between 1 and 8. Can be used to distribute the results of a search evenly across the requested area..where(fieldname, min value, max value)filter output by fieldname and values..where_expr(expr)filter output withfilter-expression.Outputscommanddescription.asObjects()return as array of objects.asBounds()return as array of minimum bounding rectangles:{"id": str,"bounds":{"sw":{"lat": float,"lon": float},"ne":{"lat": float,"lon": float}}}.asCount()returns a count of the objects in the search.asHashes(precision)returns an array of{"id": str,"hash": str}.asIds()returns an array of ids.asPoints()returns objects as points:{"id": str,"point":{"lat": float,"lon": float}. If the searched key is a collection ofPolygonobjects, the returned points are the centroids.Querycommanddescription.point(lat, lon, radius: Optional[int])Search nearby a given of latitude, longitude. If radius is set, searches nearby the given radius.SCANIncrementally iterate through a given collection key.awaittile38.scan('fleet')awaittile38.scan('fleet').where("maxspeed",100,120).asCount()>{"ok":true,"count":1,"cursor":0,"elapsed":"42.8µs"}Optionscommanddescription.asc()Values are returned in ascending order. Default if not set..desc()Values are returned in descending order..cursor(value)used to iterate through your search results. Defaults to0if not set explicitly.limit(value)limit the number of returned objects. Defaults to100if not set explicitly.nofields()if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields..match(pattern)Match can be used to filtered objects considered in the search with a glob pattern..match('warehouse*')e.g. will only consider ids that start withwarehousewithin your key..where(fieldname, min value, max value)filter output by fieldname and values..where_expr(expr)filter output withfilter-expression.Outputscommanddescription.asObjects()return as array of objects.asBounds()return as array of minimum bounding rectangles:{"id": str,"bounds":{"sw":{"lat": float,"lon": float},"ne":{"lat": float,"lon": float}}}.asCount()returns a count of the objects in the search.asHashes(precision)returns an array of{"id": str,"hash": str}.asIds()returns an array of ids.asPoints()returns objects as points:{"id": str,"point":{"lat": float,"lon": float}. If the searched key is a collection ofPolygonobjects, the returned points are the centroids.SEARCHUsed to iterate through a keys string values.awaittile38.set('fleet','truck1:driver').string('John').exec()awaittile38.search('fleet').match('J*').asStringObjects()>{"ok":true,"objects":[{"id":"truck1:driver","object":"John"}],"count":1,"cursor":0,"elapsed":"59.9µs"}Optionscommanddescription.asc()Values are returned in ascending order. Default if not set..desc()Values are returned in descending order..cursor(value)used to iterate through your search results. Defaults to0if not set explicitly.limit(value)limit the number of returned objects. Defaults to100if not set explicitly.nofields()if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields..match(pattern)Match can be used to filtered objects considered in the search with a glob pattern..match('J*')e.g. will only consider string values objects that have a string value starting withJ.where(fieldname, min value, max value)filter output by fieldname and values..where_expr(expr)filter output withfilter-expression.Outputscommanddescription.asStringObjects()return as array of objects.asCount()returns a count of the objects in the search.asIds()returns an array of idsServer and ConnectionCONFIG GET / REWRITE / SETWhile.config_get()fetches the requested configuration,.config_set()can be used to change the configuration.Important, changes made with.set()will only take affect after.config_rewrite()is used.OptionscommanddescriptionrequirepassSet a password and make server password-protected, if not set defaults to""(no password required).leaderauthIf leader is password-protected, followers have to authenticate before they are allowed to follow. Setleaderauthto password of the leader, prior to.follow().protected-modeTile38 only allows authenticated clients or connections fromlocalhost. Defaults to:"yes"maxmemorySet max memory in bytes. Get max memory in bytes/kb/mb/gb.autogcSet auto garbage collection to time in seconds where server performs a garbage collection. Defaults to:0(no garbage collection)keep_aliveTime server keeps client connections alive. Defaults to:300(seconds)awaittile38.config_get('keepalive')>{"ok":true,"properties":{"keepalive":"300"},"elapsed":"54.6µs"}awaittile38.config_set('keepalive',400)>{"ok":true,"elapsed":"36.9µs"}awaittile38.config_rewrite()>{"ok":true,"elapsed":"363µs"}awaittile38.config_get('keepalive')>{"ok":true,"properties":{"keepalive":"400"},"elapsed":"33.8µs"}Advanced optionsAdvanced configuration can not be set with commands, but has to be set
in aconfigfile in your data directory.Optionsabove, as
well as advanced options can be set and are loaded on start-up.commanddescriptionfollow_hostURI of Leader to followfollow_portPORT of Leader to followfollow_pos?follow_idID of Leaderserver_idServer ID of the current instanceread_onlyMake Tile38 read-onlyFLUSHDBDelete all keys and hooks.awaittile38.flushDb()PINGPing your serverawaittile38.ping()HEALTHZReturnsOKif server is the Leader. If server is a Follower,
returnsOKonce the Follower caught up to the Leader.
ForHTTP requestsit
returnsHTTP 200 OKonce caught up, orHTTP 500 Internal Server Errorif not.The command is primarily built to be send via HTTP in orchestration
frameworks such as Kubernetes aslivelinessProbeand/orreadinessProbe.
Since a Follower has to catch up to the state of the Leader before it
can execute queries, it is essential that it does not receive traffic
prior to being caught up.HEALTHZin combination with a readinessProbe ensures a ready state.awaittile38.healthz()// values.yamlreadinessProbe:httpGet:scheme:HTTPpath:/healthzport:9851initialDelaySeconds:60GCInstructs the server to perform a garbage collection.awaittile38.gc()READONLYSets Tile38 into read-only mode. Commands such as.set()and.del()will fail.awaittile38.readonly(True)SERVERGet Tile38 statistics.awaittile38.server()Or get extended statistics:awaittile38.server_extended()INFOGet Tile38 info. Similar toSERVERbut different metrics.awaittile38.info()GeofencingAgeofenceis a virtual boundary
that can detect when an object enters or exits the area. This boundary
can be a radius or anysearch area format,
such as abounding box,GeoJSONobject, etc.
Tile38 can turn any standard search into a geofence monitor by adding the
FENCE keyword to the search.Geofence events can be:inside(object in specified area),outside(object outside specified area),enter(object enters specified area),exit(object exits specified area),crosses(object that was not in specified area, has enter/exit it).Geofence events can be send on upon commands:setwhich sends an event when an object is.set()delwhich sends a last event when the object that resides in the geosearch is deleted via.del()dropwhich sends a message when the entire collection is droppedSETHOOKCreates a webhook that points to a geosearch (NEARBY/WITHIN/INTERSECTS).
Whenever a commands creates/deletes/drops an object that fulfills the geosearch,
an event is send to the specifiedendpoint.# sends event to endpoint, when object in 'fleet'# enters the area of a 500m radius around# latitude 33.5123 and longitude -112.2693awaittile38.sethook('warehouse','http://10.0.20.78/endpoint').nearby('fleet').point(33.5123,-112.2693,500).activate()awaittile38.set('fleet','bus').point(33.5123001,-112.2693001).exec()# results in event =>{"command":"set","group":"5c5203ccf5ec4e4f349fd038","detect":"inside","hook":"warehouse","key":"fleet","time":"2021-03-22T13:06:36.769273-07:00","id":"bus","meta":{},"object":{"type":"Point","coordinates":[-112.2693001,33.5123001]}}Geosearchcommanddescription.nearby(name, endpoint).within(name, endpoint).intersects(name, endpoint)Optionscommanddescription.meta(meta)Optional add additional meta information that are send in the geofence event..ex(value)Optional TTL in seconds.commands(which[])Select on which command a hook should send an event. Defaults to:['set', 'del', 'drop'].detect(what[])Select what events should be detected. Defaults to:['enter', 'exit', 'crosses', 'inside', 'outside']EndpointscommanddescriptionHTTP/HTTPShttp://https://send messages over HTTP/S. For options seelink.gRPCgrpc://send messages overgRPC. For options seelink.Redisredis://send messages toRedis. For options seelinkDisquedisque://send messages toDisque. For options seelink.Kafkakafka://send messages to aKafkatopic. For options seelink.AMQPamqp://send messages toRabbitMQ. For options seelink.MQTTmqtt://send messages to an MQTT broker. For options seelink.SQSsqs://send messages to anAmazon SQSqueue. For options seelink.NATSnats://send messages to aNATStopic. For options seelink.SETCHAN / SUBSCRIBE / PSUBSCRIBESimilar tosethook(), but opens a PUB/SUB channel.# Start a channel that sends event, when object in 'fleet'# enters the area of a 500m radius around# latitude 33.5123 and longitude -112.2693awaittile38.setchan('warehouse','http://10.0.20.78/endpoint').nearby('fleet').point(33.5123,-112.2693,500).activate()Given a proper setup of a pubsub channel, every set.set()results in:awaittile38.set('fleet','bus').point(33.5123001,-112.2693001).exec();# event =>{"command":"set","group":"5c5203ccf5ec4e4f349fd038","detect":"inside","hook":"warehouse","key":"fleet","time":"2021-03-22T13:06:36.769273-07:00","id":"bus","meta":{},"object":{"type":"Point","coordinates":[-112.2693001,33.5123001]}}Geosearchcommanddescription.nearby(name, endpoint).within(name, endpoint).intersects(name, endpoint)Optionscommanddescription.meta(meta)Optional addition meta information that a send in the geofence event..ex(value)Optional TTL in seconds.commands(which[])Select on which command a hook should send an event. Defaults to:['set', 'del', 'drop'].detect(what[])Select what events should be detected. Defaults to:['enter', 'exit', 'crosses', 'inside', 'outside']Addition InformationFor more information, please refer to:Tile38RoadmapSee theopen issuesfor a list
of proposed features (and known issues).ContributingContributions are what make the open source community such an amazing place
to be learn, inspire, and create. Any contributions you make aregreatly appreciated.Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull RequestLicenseMITMaintainerBenjamin Ramser -@iwpndProject Link:https://github.com/iwpnd/pyle38AcknowledgementsJosh Baker - maintainer ofTile38