Package: ps2-census
ps2-censusps2-censusis a low-level client for Daybreak's Planetside 2 Census API written in Python >= 3.8.ps2-censusInstallationFull examplesQuery buildingBasic querySimple joinNesting joinsDeeply nested joinLateraly nested joinsTreeFactoriesCommon enumsEvent StreamUsageNext stepsSimilar projectsDevelopmentEnvironmentTechnical detailsFeatures:Build queries through method chainingJoin collections and nest them through method chainingGet raw (deserialized) responses as Python dictionariesAccess common enums directlySubscribe to event streamsBy default theexampleservice ID is used; however it is not recommended for production.
You should get your own service ID from the webside below and supply it to the client whenever needed.More information about the Census API is available on the official Census documentationhere.Iridar's blogis also recommended to understand
the quirks of this particular API as this knowledge is necessary to useps2-censusproperly.Installationpipinstallps2-censusFull examplesBefore diving in the details of theps2-censusclient, full examples are available in theexamplesfolder.They currently include:all_about_trac_5.py(and the output inall_about_trac_5.json): building and executing a query that
fetches pretty much everything that's to know about the TR TRAC-5 carbinecharacter_death_event_stream.py(and the output incaracter_death_event_stream.ndjson): subscribing to
all character death events on the SolTech server, receive at most 20 events and print them, then gracefuly
disconnectQuery buildingQueries are made on collections with optional joins on other collections as well as various
commands that alter the output.A Census API collection is analog to a relation in a generic relational database system, and a
Census API join is analog to a join between these relations.Basic queryTo build a query, instantiate theQueryclass with aCollection(and your service ID,
though it will be omitted in next examples for conciseness):fromps2_censusimportCollection,Queryquery:Query=Query(Collection.ITEM,service_id=YOUR_SERVICE_ID)Chain methods to alter the query further:query:Query=(Query(Collection.ITEM).lang("en").sort(("item_id",1),("faction_id",-1)).limit(30)[...])Available methods are:filter(field: str, value: Union[str, int], modifier: Optional[SearchModifier]): filter
the query on a field;ps2_census.SearchModifiercontains all the modifiers made available
by the Census API (SearchModifier.CONTAINS,SearchModifier.LESS_OR_EQUAL, ...)show(*args: str): only return the provided fields in resultshide(*args: str): do not return the provided fields in resultssort(*args: Tuple[str, Literal[1, -1]]): sort the results by field, either in increasing or decreasing orderhas(*args: str): only return results which have the specified fieldscase(arg: bool): whetherfilter()s are case sensive or not; default toTruelimit(arg: int): limit the return toat mostargresults; required in tendem withstart()for queries having too large of a result and therefore fail; defaults to1limit_per_db(arg: int): limit the return toat mostarg * databases countresults; useful when
querying theCollection.CHARACTERcollection whose objects are randomly distributed among multiple
databases in order to have more predictable resultsstart(arg: int): start with theargth object within the results of the queryinclude_null(arg: bool): whether to include keys withNonevalues in results; defaults toFalselang(arg: str): only keep the supplied language for internationalized stringsjoin(arg: Join): perform a collection join; see the following documentation for additional detailstree(arg: Tree): rearrange lists of data into trees; see the following documentation for additional detailstiming(arg: bool): show query timing informationexact_match_first(arg: bool): when usingfilter()s withSearchModifiers, put exact matches at the top of the
results list disregardingsort()sdistinct(arg: str): get the distinct values for a certain fieldretry(arg: bool): retry queries at most one time before failing; defaults toTrueExecute the query in one of the 2 ways made available by the Census API:.count()to get the items countquery.count()>{'count':21048}.get()to get the resultsquery.get()>{'item_list':[{...},{...},...],'returned':30}count()andget()calls are when the query is actually sent to the Census API endpoint.
They will raise status exceptions if appropriate.Simple joinIn order to perform joins instantiate theJoinclass with aCollection, add any additional
chained methods to it, and pass it to theQueryobject viaquery.join():fromps2_censusimportCollection,Join,Queryquery:Query=(Query(Collection.ITEM).join(Join(Collection.WEAPON_DATASHEET).outer(0).on("item_id").to("item_id").inject_at("weapon_datasheet")[...]))AvailableJoinmethods are:nest(other: Join): nest another join within this one; see the following documentation for additional detailson(arg: str): specify the field on this collection (theQueryor parentJoin) to join on; if not provided will default to this collection's ID ({this_type}_id)to(arg: str): specify the field on the other collection (theJoin's) to join to; if not provided will default to theonvaluelist(arg: Literal[0, 1]): whether the joined data is a list (and therefore will result in a list of objects) or not;1if it is a list,0if not; default to0show(*args: str): only keep the provided fields in resultshide(*args: str): do not keep the provided fields in resultsinject_at(arg: str): the field name where the joined data will be injected in the parent element (Queryresult item or parentJoinelement)terms(*args: Tuple[str, Union[str, int]]): filter the join result by conditions (eg.terms(("faction_id", 1), ("skill_set_id", 129)))outer(arg: Literam[0, 1]): whether the join will perform an outer join (include non-matches) of an inner join (exclude non-matches);1for outer,0for inner; defaults to1Multiple joins can be performed one after another on the sameQueryand the trees will be merged in
the result:query:Query=(Query(Collection.ITEM).join(Join(Collection.WEAPON_DATASHEET)).join(Join(Collection.ITEM_TO_WEAPON)))Nesting joinsJoins can be nested one into another both deeply and lateraly. Nesting is done by reference: eachJoininstance
contains a list of references to its nestedJoins.Note: be careful about modifying theJoininstances after their creation if you assign them to locals
for convenience, as shown in the following examples.
As nesting is done by reference and the whole structure is "compiled" (into a querystring) on the call
toquery.join(), if any modification is done to an instance between its creation and the resolution,
it will indirectly impact the result of any other parent and/or nestedJoin.
Make use of the below described factories to make clean copies of your instances as necessary.Deeply nested joinDeeply nested join are necessary in order to access data structures deeper in the collections tree.
To deeply nest joins, instantiate theJoinclass multiple times and combine them throughjoin1.nest(join2.nest(join3))wherejoin3is nested withinjoin2andjoin2is nested withinjoin1:fromps2_censusimportCollection,Join,Queryitem_to_weapon_join:Join=(Join(Collection.ITEM_TO_WEAPON).on("item_id").to("item_id").inject_at("item_to_weapon"))weapon_join:Join=(Join(Collection.WEAPON).on("weapon_id").to("weapon_id").inject_at("weapon"))weapon_to_fire_group_join:Join=(Join(Collection.WEAPON_TO_FIRE_GROUP).on("weapon_id").to("weapon_id").inject_at("weapon_to_fire_group"))query:Query=(Query(Collection.ITEM).filter("item_type_id",ItemType.WEAPON).join(item_to_weapon_join.nest(weapon_join.nest(weapon_to_fire_group_join))))Lateraly nested joinsLateraly nested joins are necessary in order to access data structures at the same depth in the collections tree.
To laterally nest joins, instantiate theJoinclass multiple times and combine them throughjoin1.nest(join2).nest(join3)wherejoin2andjoin3are nested withinjoin1:fromps2_censusimportCollection,Join,Queryfire_group_to_fire_mode_join:Join=(Join(Collection.FIRE_GROUP_TO_FIRE_MODE).outer(0).on("fire_group_id").to("fire_group_id").list(1).inject_at("fire_group_to_fire_modes"))fire_mode_join:Join=(Join(Collection.FIRE_MODE_2).outer(0).on("fire_mode_id").to("fire_mode_id").inject_at("fire_mode"))fire_mode_to_projectile_join:Join=(Join(Collection.FIRE_MODE_TO_PROJECTILE).outer(0).on("fire_mode_id").to("fire_mode_id").inject_at("fire_mode_to_projectile"))player_state_group_join:Join=(Join(Collection.PLAYER_STATE_GROUP_2).outer(0).on("player_state_group_id").to("player_state_group_id").list(1).inject_at("player_state_groups"))query:Query=(Query(Collection.WEAPON_TO_FIRE_GROUP).join(fire_group_to_fire_mode_join.nest(fire_mode_join.nest(fire_mode_to_projectile_join).nest(player_state_group_join))))TreeTrees are also built using their own class,Tree, then passed to theQueryobject:fromps2_censusimportCollection,Query,Treequery:Query=(Query(Collection.ITEM).tree(Tree("name.en").prefix("en_name_")[...]))This will return a dictionary of items with their english name prefixed byen_name_as keys
and the objects themselves as values, instead of a flat list of items.Available methods are:list(arg: Literal[0, 1]):0if tree data is not a list,1if it is a list; defaults to0prefix(arg: str): prefix to add to the field valuestart(arg: str): where the tree starts; defaults to the root (root list objects will be formatted as a tree)FactoriesIn order to ease the project-wide definition ofQuery,JoinandTreeobjects,
each of these classes has aget_factory()method.Upon invocation it saves a copy of the current object and returns a callable that, upon each invocation,
returns a fresh copy of that saved state.query:Query=Query(Collection.ABILITY).has("someField").case(False)query_factory:Callable[[],Query]=query.get_factory()query_copy:Query=query_factory()assertquery==query_copy>Truequery=query.timing(True)assertquery!=query_copy>Truequery_copy=query_copy.lang("en")another_copy=query_factory()assertquery==another_copy>Trueassertquery_copy!=another_copy>TrueFactories obtained in this way are useful to defineQuery,JoinorTrees somewhere in your code and store
their factory callables, with the guarantee that you cannot modify the shared definition anywhere.my_module.pyfromps2_censusimportQuery,Collectionmy_query_factory:Callable[[],Query]=(Query(Collection.ABILITY).lang("en").get_factory())main.pyfrommy_moduleimportmy_query_factoryquery:Query=my_query_factory()query.start(100)# does not modify any shared objectCommon enumsCensus API data uses a lot of integer enumerations that are collections themselves.For example thefaction_idkey in items fromCollection.ITEMis an integer that represents
a specific fation, refering toCollection.FACTION: Vanu is1, NC is2etc.In order to reduce the amount of necessary joins, which are arguably the most complex part of Census queries,
some common enumerations are provided inps2_census.enumsas Python enum.IntEnum classes, including:ArmorFacingFacilityTypeFactionFireModeTypeItemTypeItemCategoryMetagameEventStatePlayerStateProjectileFlightTypeResistTypeResourceTypeRewardTypeTargetTypeVehicleWorldThese typically do not change often andps2-censuswill be updated whenever there is such a change.They can be used just for reference, but also in queries for filtering.See the following example for filtering weapon items only usingps2_census.enums.ItemType:fromps2_census.enumsimportItemTypequery=(Query(Collection.ITEM).filter("item_type_id",ItemType.WEAPON))Event Streamps2-censusoffers a client that handles connection to the WebSocket endpoint, subscription
to various streams and reception of the events.Note: because the client uses thewebsocketslibrary,
we need to handle async calls.UsageFirst you need to connect to the WebSocket endpoint; to do this, instantiate theEventStreamclass:fromps2_censusimportEventStreamstream:EventStream=awaitEventStream(service_id=YOUR_SERVICE_ID)Then, subscribe to events:fromps2_censusimportCharacterEvent,WorldEvent,EventStreamWorld,GenericCharacterawaitstream.subscribe(worlds=[EventStreamWorld.SOLTECH,EventStreamWorld.BRIGGS],events=[CharacterEvent.DEATH,WorldEvent.CONTINENT_LOCK],characters=[GenericCharacter.ALL,"1234","5678"],logical_and_characters_with_worlds=True)Where:worldsis a list ofEventStreamWorldobjects. Use[EventStreamWorld.ALL]for all worldseventsis a list ofCharacterEvent,WorldEventorGenericEventobjects. UseGenericEvent.ALLto get all events (character and world)charactersis a list of character IDs as strings or the specialGenericCharacter.ALLto subscribe to all characterslogical_and_characters_with_worldsis True if you want to match all events concerning the charactersANDthe worlds; default is False, so it matches all events concerning the charactersORthe worldsYou can perform multiple subscriptions one after another on the sameEventStreamobject; they are additively merged.Finally, you need to handle received events from your subscription:awaitstream.receive()This simple example put together (you might want to develop it further to do more than simply print events,
handle graceful deconnection, etc):importasynciofromps2_censusimportCharacterEvent,WorldEvent,EventStream,EventStreamWorld,GenericCharacterasyncdefmain():stream:EventStream=awaitEventStream()awaitstream.subscribe(worlds=[EventStreamWorld.SOLTECH,EventStreamWorld.BRIGGS],events=[CharacterEvent.DEATH,WorldEvent.CONTINENT_LOCK],characters=[GenericCharacter.ALL,"1234","5678"],logical_and_characters_with_worlds=True)whileTrue:print(awaitstream.receive())asyncio.run(main())More information about the Planetside2 Census event stream can be found athere.Next stepsImprove theEventStreamclient to make it slightly higher-level (handle disconnections,
filter events, etc)Similar projectsleonhard-s'auraxiumDevelopmentEnvironmentIn order to developps2-census:Setup a virtual environment with python 3.8InstallpoetryInstall dependencies withpoetry installRun tests withpytestUpdate dependencies withpoetry updateTechnical detailsAll interaction with the Census API (except the WebSocket events stream part) is done through
query parameters. Query bodies are always empty and all information is carried in the query URL string.ps2-census' objective is to wrap the building of these (often very long and complex)
querystrings and provide a comfortable, structured programmatic interface.To achieve that, we make use of theQueryclass which:on initialization sets up basic query elements such as endpoint, service ID, collection and namespace,at each "command" method call adds query parameters to itsparametersdictionary attribute,atget()orcount()build the actual URL and execute the query using therequestslibrary.Collection joins are more complicated as they are not built as proper query parameters;
inside the final query, each call toquery.join()creates a new query parameter, eg....&c:join={join_string}&...with thejoin_stringbeing written in a specific format.Hence theJoinclass which:on initialization sets up the collection,at each method call adds items to itsitemsdictionary attribute,at eachnest()call stores a reference to the nestedJoininstance in itsnested_joinslist attribute,at__str__()builds thejoin_stringthat will become a main query's parameter value, in the specific format
and using theitemselements as well as recursively calling__str__()onnested_joinsitems.Therefore, thejoin_stringis not actually computed before a call to aJoininstance's__str__()method,
and it is recursively computed across nested joins.The event streams part is more conventional, asps2-censusjust needs to connect to a WebSocket endpoint, issue
subscription commands in a custom but simple format and then read messages from the stream.