Package: pyngsild
pyngsildOverviewpyngsildis a Python data-centric framework whose goal is to ease and speed up the development ofNGSI-LDagents.By providing a clean and simple structure - with components organized as a NGSI-LD data pipeline - the framework allows the developer to avoid the plumbing and focus on the data.Key FeaturesAgents that rely on the pyngsild framework all share a common structureMany DataSources includedStatisticsMonitoring(for background agents)Error handlingLoggingWell-tested componentsProvide primitives to build NGSI-LD compliant entities(thanks to thengsildclientlibrary)How it worksDataSourcesWhat most differentiates an agent from another is the datasource.Not only the nature of the data differs but also :the data representation : text, json, ...the way data are accessed : read from a file, received through the network, ...pyngsildprovides a level of abstraction in order to expose any datasource in a same way, whether :the agentconsumesa datasource(i.e. reads a file, requests an API)the agentis triggeredby the datasource(acts as a daemon listening to incoming data pushed by the datasource)As datasources have very little in common, the only assumption made by the framework is : apyngsildSource is iterable.For illustrative purposes an element accessed from a Source could be a line from a CSV file, an item from a JSON array, or a row from a Pandas dataframe.Many generic Sources are provided by the framework and it's easy to create new ones.The pipelineA NGSI-LD Agent typically :collects data from a datasourcebuilds "normalized" NGSI-LD entities(according to a domain-specific DataModel)eventually feeds the Context BrokerThe framework allows to create anAgentby providing aSource, aSinkand aprocessorfunction.The Source collects data from the datasource.When the Agent runs, it iterates over the Source to collect Rows.The processor function takes aRowand builds a NGSI-LDEntityfrom it.A Row is an object composed of two attributes : record and providerrecord: Any = the raw incoming dataprovider: str = a label indicating the data providerEventually the Entity is sent to theSinkwhich is in production mode theContext Broker.+-----------------------------------------------------------------------------------+
|                                                                                   |
|                                                                                   |
|      +--------------+                                       +--------------+      |
|      |              |     Row                    Entity     |              |      |
|      |    Source    |-------------> process() ------------->|     Sink     |      |
|      |              |                                       |              |      |
|      +--------------+                                       +--------------+      |
|                                                                                   |
|                                                                                   |
+-----------------------------------------------------------------------------------+
                                        AgentWhere to get itThe source code is currently hosted on GitHub at :https://github.com/Orange-OpenSource/pyngsildBinary installer for the latest released version is available at thePython
package index.pipinstallpyngsildInstallationpyngsildrequires Python 3.10+.One should use a virtual environment. For example with pyenv.mkdirmyfiwareproject&&cdmyfiwareproject
pyenvvirtualenv3.10.2myfiwareproject
pyenvlocalpipinstallpyngsildGetting startedCreate a SourceFor example let's create a Source that collects data about companies bitcoin holdings thanks to the CoinGecko API.importrequestsfrompyngsildimport*fromngsildclientimport*COINGECKO_BTC_CAP_ENDPOINT="https://api.coingecko.com/api/v3/companies/public_treasury/bitcoin"src=SourceApi(lambda:requests.get(COINGECKO_BTC_CAP_ENDPOINT),path="companies",provider="CoinGecko API")Have a lookherefor a sample API result.Provide a processor functionYou have to provide the framework with aprocessorfunction, that will be used to transform a Row into a NGSI-LD compliant entity.defbuild_entity(row:Row)->Entity:record:dict=row.recordmarket,symbol=[x.strip()forxinrecord["symbol"].split(":")]e=Entity("BitcoinCapitalization",f"{market}:{symbol}:{iso8601.utcnow()}")e.obs()e.prop("dataProvider",row.provider)e.prop("companyName",record["name"])e.prop("stockMarket",market)e.prop("stockSymbol",symbol)e.prop("country",record["country"])e.prop("totalHoldings",record["total_holdings"],unitcode="BTC",observedat=Auto)e.prop("totalValue",record["total_current_value_usd"],unitcode="USD",observedat=Auto)returneHave a lookherefor a sample NGSI-LD Entity built with this function.Run the AgentLet's create the Sink, the Agent and make all parts work together.sink=SinkNgsi()# replace by SinkStdout() if you don't have a Context Brokeragent=Agent(src,sink,process=build_entity)agent.run()print(agent.stats)# input=27, processed=27, output=27, filtered=0, error=0, side_entities=0agent.close()We're done !The Context Broker should have created a set of entities(27 at the time of writing).LicenseApache 2.0