Package: wirepas-backend-client
Backend-clientIntroductionInstallationHost dependenciesSetting up a Python virtual environmentInstalling from PyPiInstalling from GithubEntrypointsGateway command line interfaceWPE ViewerWNT ViewerProvisioning serverFrameworkStructureExamplesLogging to fluentdBuilding and running over DockerDockerhubRunning with dockerRunning with composeBuilding the image locallySource documentationContributingLicenseIntroductionThe Backend Client is a tool to serve as a decoder and a test framework
where you can extend the existing interfaces to develop your own test cases.
Interactions with
agatewaymust be compliant with the Wirepas
Backend to Gateway API.In addition to the interaction with the Backend to Gateway Public API,
the Backend Client has also support for Wirepas Positioning Engine and Wirepas
Network Tool. Support for these tools is still work in progress
(seemilestonesandprojects).
However, you can already benefit from existing
entrypoints to help you capture data and debug your
integration (seeentrypoints).Figure 1 illustrates the horizon where Backend Client is located.Figure 1- Backend Client role in the Wirepas ecosystem.InstallationThe Backend Client is written in python and it requires a valid MQTT
broker target to connect to, WNT or WPE credentials.If you need help installing a MQTT broker or auxiliary software please
refer to our tutorials repository.Host dependenciesThe main requirements of Backend Client are:Python 3.7Pip (we recommend the latest available)Linux systemCertain Backend Client's dependencies must be
compiled locally upon installation,
thus require the installation of specific development headers.Under Debian, the extra requirements are met with:sudoapt-getinstalldefault-libmysqlclient-devgccpython3.7-devPlease be advised that the name of such packages might change depending on your
Linux system. Please refer to the Dockerfiles
under thecontainer folderfor more detailed information.Windows native environments are not supported. For help running the tool
in Windows, please use the Linux subsystem, Docker or a virtual machine.Setting up a Python virtual environmentAs a recommendation, it is good practice to install and run the
application on a virtual python environment. Such approach avoids
possible issues with conflicting package versions.We recommend you usepipenv:pip3installpipenvIf you choose to install and run the Backend Client inside a virtual
environment, please activate the environment before moving with the next
steps.For pipenv, a virtual environment is created and activated by:pipenv--threepipenvshellOnce you activate your environment your can safely install the python
package under it.For more information please refer to the tool's webpage.Installing from PyPiThe Backend Client is available fromPyPiand you can install the latest stable version
with:pipinstallwirepas-backend-clientIf you wish to install a particular version please see the release history
from PyPi.Installing from GithubFirst of all, make sure to clone the repository using the https address.Checkout thegit tag that corresponds to the release versionyou want to install and install the package with:pip3install.If you leave your branch checked out to the top of master, it is likely that
you might run into a few issues. If you find them, please be so kind to
raise them to us through anissue or bug report.If you want to develop or patch a bug under your local environment,
you can install the package in development mode through:pip3install-e.When installed in development mode, changes to the source files will be
immediately visible.EntrypointsAfter installation, the Backend Client exposes severalshell entrypointsthat
can be run from any location in your system. This section describes their
usage in more detail.Gateway command line interfaceThe Backend Client exposes an interactive gateway command line which is
meant to interact with a gateway and sink pair connected to a given
MQTT broker.To start an interactive command line to interact with a broker located
in yourlocalhostenvironment type:wm-gw-cliConnecting to a remote server requires you to specify the values as an
argument or by reading from a yaml settings file. To view all the
arguments, please issue the command:wm-gw-cli--helpConnecting to the MQTT brokerFor the gateway client to function properly, it needs to establish a
connection to the same MQTT broker where one or multiple gateways
are connected to.To do so, let's assume the following MQTT credentials are saved in
./examples/settings.yml:# examples/settings.yml# Example on how to set the MQTT credentials towards a local broker# with basic authenticationmqtt_hostname:127.0.0.1mqtt_password:passwordmqtt_username:usernamemqtt_port:1883# defaults to 8883 (secure port)mqtt_force_unsecure:True# defaults to False (secure)To start the gateway against the MQTT broker specified in
./examples/settings.yml, issue the following command:wm-gw-cli--settings./examples/settings.ymlOnce the client launches, you will be greeted with:WelcometotheWirepasGatewayClientcli!Connectingtomosquittouser@127.0.0.1:1883(unsecure:True)Typehelpor?tolistcommandsType!toescapeshellcommandsUseArrowUp/DowntonavigateyourcommandhistoryUseCTRL-Dorbyetoexit09:50.17|wm-gw-cli>CommandsThe gateway client has a set of commands which you can view by typing ?
in the client's shell:wm-gw-cli>?Documentedcommands(typehelp<topic>):========================================EOFplaybackset_loop_iterationsbyeqset_loop_timeoutclear_offline_gatewaysrecordset_reply_greetingeofscratchpad_statusset_sinkgateway_configurationscratchpad_updatesettingsgatewaysscratchpad_uploadshellhelpselectionsinkslistsend_datatoggle_byte_printlsset_app_configtoggle_pretty_printnetworksset_configtrack_devicesnodesset_gatewaytrack_data_packetsFor each available command you can get more help if you type?commandfor example, to view how to send a datagram,
type in the shell:wm-gw-cli>?send_dataWhen typing commands, you can use the up and down arrows to browse your
shell history. There is also support for running host shell commands, by
prefacing the command with!.Viewing connected devicesUpon a MQTT connection, the Backend Client will start populating an
internal structure based on any incoming message. The topic is parsed
and the information about the network devices are constructed.The operation happens as messages are published from the network. It
will take time to build a full map of the network, which will be faster
depending on your network's packets per second.In case you wish to view periodically which devices you have on your network,
type in the following command:wm-gw-cli>?track_devicesTo exit the tracking loop, give a new line feed to the input stream and
it will go back to the main command shell.Device settingsWhen sending a downstream datagram, it is necessary to obey the
Backend to Gateway API (WM-RM-128).
For that reason, it is mandatory to know what is the target
gateway identifier and in most cases the sink id as well.Selecting the target gateway and sinkThe wm-gw-cli provides an easy way to select the target gateway and sink. To
make your choice, please type the following commands:wm-gw-cli>set_gateway0:2485378023427:GatewayState.ONLINE1:2485378023426:GatewayState.OFFLINEPleaseenteryourgatewayselection[0]:0wm-gw-cli>set_sink0:3806491:2485378023427:sink0Pleaseenteryoursinkselection[0]:0When setting the gateway, the wm-gw-cli shows you the following informationobject index : gateway id : state.Similarly, for the sink, the wm-gw-cli
presents the objectindex : network : gateway id: sink id.To make your selection, type the desired object index and press enter.Once your selection is done, your commands will target that gateway id
and sink id. If you wish to view what your current selection is, use the
command:wm-gw-cli>selection:warning:WARNING:warning:The wm-gw-cli prompt will update itself with the current gateway and sink id.
Throughout this document, that information has been edited out of the
prompt.Set app configSetting the app config requires that your gateway and sink are selected.
In case you have not done the selection prior to this command, the wm-gw-cli
will ask for them.When setting the app config, you are setting three values on the target
sink:Diagnostic interval: the interval of network diagnostics (0, 30, 60,
300, 600, 1800)App config: A payload of a given size distributed to all associated
devices on the network;App config sequence: The sequence of the new app config (must be
higher than the current one - otherwise the app config won't take
effect, please refer to WM-RM-100).The app config payload is given either as a set of binary data without
trailing 0x or as a utf8 data. For example, to send the
binary0x10x40x5you should type010405.
If you simply want to have a text string on
your app config, you can for example type "sink by the window".As an example, on the wm-gw-cli you would type:# Remember! no white spaces allowing in the payload!# If you want to toggle byte string to hex string in the# app_config_data, please see toggle_byte_printwm-gw-cli>set_app_config10sink_by_the_window30answer<<gw_id: 2485378023427sink_id: sink0req_id: 12079559850249952277res: GatewayResultCode.GW_RES_OKsink_id: sink0current_ac_range_min:2000current_ac_range_max:8000min_ac:2000max_ac:8000max_mtu:102min_ch:1max_ch:27hw_magic:3stack_profile:1app_config_max_size:80are_keys_set:Falsefirmware_version:[3,4,37,0]node_role:17node_address:4193520network_address:3806491network_channel:22app_config_diag:30app_config_seq:10app_config_data:b'sink_by_the_window\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'channel_map:286331153started:TrueThe answer will carry the gateway result and present you the new app
config.Upload and process a scratchpadThe wm-gw-cli supports uploading a new scratchpad and asking the sink to take
it into use.A full network OTAP is not currently supported with the
wm-gw-cli.To initiate a network OTAP, you will have to send the
corresponding datagrams as mentioned in the remote API documentation.The scratchpad commands are available as:scratchpad_statusscratchpad_updatescratchpad_upload:warning:WARNING:warning:An incorrect scratchpad or an incorrect OTAP can damage your network
permanently. Please use these commands with care.WPE ViewerThe WPE viewer,wm-wpe-viewer, provides a simple way to check
if a WPE service and consume location updates for a given network.The WPE viewer entrypoint requires correct population of the following
settings:#examples/settings.ymlwpe_service_definition:./mywpesettings.jsonwpe_network:1092#optionalThemywpesettings.jsonconsists of a standard WPE client configuration
file. Please refer to WPE's official documentation for more details
on how to obtain and configure such file.The wpe network is the network identifier on which your devices operate.
If you leave the parameter unset, the entrypoint will only perform a
ping to the backend specified under the json file.WNT ViewerThe WNT viewer,wm-wnt-viewer, provides a simple way to check
if a WNT service is reachable and to consume messages streamed by WNT on
its realtime metadata websocket. These message contain network and node
diagnostic information. Please refer to WNT's public API documentation
for further information.The WNT viewer entrypoint requires correct population of the following
settings:#examples/settings.ymlwnt_username:"wntuser"wnt_password:"98asuyd907171ehjmasd"wnt_hostname:"wnthost.com"Provisioning serverThe provisioning server,wm-provisioning-server, provides an example
implementation of the server side of the provisioning protocol. It must be used
with theprovisioning_joining_nodeapplication of the SDK. Please refer to
the provisioning reference manual for further information.The Provisioning server entrypoint requires correct population of the following settings:#examples/settings.ymlmqtt_hostname:mqtt_broker_address_or_ipmqtt_password:passwordmqtt_username:usernamemqtt_port:1883# defaults to 8883 (secure port)mqtt_force_unsecure:True# defaults to False (secure)provisioning_config:"./myprovisioningconfig.yml"Themyprovisioningconfig.ymlconsists of a list of authorized nodes and their
associated provisioning data.FrameworkBesides the useful entrypoints to interact with Wirepas services, the goal of
the Backend Client is to provide you a framework around which you can
build test cases to help you during your integration, research and
development phases.StructureThe framework or package is organized in several submodules:apiapi/influxapi/mqttapi/mysqlapi/wntapi/wpemanagementmeshmesh/interfacesmesh/interfaces/mqttmesh/interfaces/remote_apimesh/interfaces/beacon_apimessagestesttoolscliAPI submoduleTheapi submodulecontains the interfaces to connect
and handle data coming in and out of Wirepas' services and other
3rd party services such as databases
and message brokers.Each API or interface folder is organized with connectors, handlers and
settings. The connectors file contains classes to establish a connection to
the target interface. The handlers contain classes to process streams of input
and output data. The settings file allows for handling parameter
translation from input to class arguments (if needed).Management submoduleThemanagement submodulecontains files regarding
the runtime management of the framework.
In particular the daemon which simplifies the multiprocess aspect
of the Backend Client.Mesh submoduleThemesh submodulecontains files related to how
devices are mapped within the Backend Client.
In addition to those, the submodule contains an interfaces
folder which contains specializations on how to handle specific Wirepas's
APIs, such as theGateway to Backend API.:pencil2:NOTE:pencil2:In our redesign effort we are targeting to improve the device
classes and implement remote API support. Please follow the
framework development for more up to date information.Messages submoduleThemessages submoduleconsists of a collection of
message handler or decoders that handle translation of known APDUS,
based on the endpoint they are associated to.:pencil2:NOTE:pencil2:We are considering moving the message handlers into the overarching Wirepas
Backend APIs project. Please follow the framework development for more up to
date information.Test submoduleThetest submodulecontains classes to handle internal task
scheduling and scripts to address particular mesh use cases.Tools submoduleThetools submodulecontains a collection of files
with classes that handle the acquisition of user
input, setup of the logging interface and general
purpose utility methods and classes.Cli submoduleCurrently still asingle file, this upcoming submodule
will contain a re-factor of the gateway client to allow easier
customization and construction of custom command line interfaces.Logging to fluentdThe Backend Client has integrated logging with fluentd through Python's
logging facility.Routing data to a fluentd host requires that you define the target host
when executing a Backend Client script.To configure the target host, tag and record for the stream, ensure that
you configure the examples/settings.yml file with# tags stream with app.meshfluentd_hostname:"myfluenthost"fluentd_record:meshfluentd_tag:appBuilding and running over DockerDocker allows application to run on a sandbox containing
all the dependencies needed to run and execute them.
If you are not familiar with Docker, please refer to the
official documentation atdocker.com.DockerhubBackend Client builds are available
from dockerhub under theBackend Client registry.The latest tag points to the current stable release,
whereas the edge tag points to the top of master.The latest tag is built automatically at dockerhub whenever this repository
is tagged.The edge tag is built after each single merge into master.To pull the Backend Client image from dockerhub use:dockerpullwirepas/backend-client:latestdockerpullwirepas/backend-client:<tag>Running with dockerAs the container will have no access to your local environment, you will have
to propagate the input parameters by mounting a local file inside the
container,eg,examples/settings.yml.The default image command will launch the gateway client with the settings
present under/home/wirepas/vars/settings.yml(container path).To run it with docker typedockerrun-it\-v$(pwd)/examples/settings.yml:/home/wirepas/backend-client/vars/settings.yml\--net=host\wirepas/backend-client\wm-gw-cli\--settings/home/wirepas/backend-client/vars/settings.yml\--debug_level=critical:warning:WARNING:warning:If you want to run against a MQTT running in your host (localhost),
you must overlay the container over your host's network.
To do so, you must set the docker run parameter--net=host.Running with composeTo run the Backend Client using docker compose, drop or move the settings
file incontainer/examples/settings.ymland start the service with:docker-composecontainer/docker-compose.ymlupBy default this will start theMQTT viewer example.If you wish to run the gateway command client you can do so with:docker-compose-fcontainer/docker-compose.yml\run\backend-clientwm-gw-cli\--settings/home/wirepas/backend-client/vars/settings.ymlIf you prefer alpine based images, please changeslimtoalpine.Building the image locallyTo build the image locally in the root of the repo type:dockerbuild-fcontainer/slim/Dockerfile-tbackend-client.Alternatively you can also build using the docker-compose.yml present in
the root of the directory:docker-compose-fcontainer/docker-compose.ymlbuildSource documentationThe source documentation is found fromread the docs.ContributingWe welcome your contributions!Please read theinstructions on how to do itand please review ourcode of conduct.LicenseCopyright 2019 Wirepas Ltd licensed under Apache License,
Version 2.0 See fileLICENSEfor full license details.