Package: topic-store
Topic StoreTopic Store is a ROS package for storing ROS messages to a database or filesystem.Unlike ROS bags Topic Store adds flexibility by serialising all messages into a data hierarchy that's easily
searchable with database queries and allows for remote storage.You can also use Topic Store as a standalone python package to read and write data without a ROS installation!InstallationROS# From sourcecdcatkin_ws/src
gitclonehttps://github.com/RaymondKirk/topic_storecatkinbuildtopic_store# From aptsudoaptinstallros-melodic-topic-store# you need to add the L-CAS ros sourceInstalling without ROSYou can install topic_store as an independent python2/3 package without a working ROS installation.# From sourcegitclonehttps://github.com/RaymondKirk/topic_storecdtopic_store/src
pipinstall--extra-index-urlhttps://rospypi.github.io/simple/-e.# From PyPipipinstall--extra-index-urlhttps://rospypi.github.io/simple/topic-storeTo install other dependencies i.e.ros_numpyyou can run the followingpipinstall--extra-index-urlhttps://rospypi.github.io/simple/ros_numpyUsageCreate a scenario file by following the documentation inscenariosor by following the example filedefault_config.yaml. The example below will save a history of all messages sent to the ros log topic.# Save to /path/to/your/scenario/file.yaml. This collection behaviour will save your log history.context:"save_ros_logs"storage:"method":"filesystem""location":"default"data:{"ros_msg":"/rosout",}collection:method:"event"watch_topic:"/rosout"Ifstorage.methodis database ensure that your database is accessible at the host/port in the config file found atstorage.config. To launch the default databaseroslaunch topic_store start_database.launch, the default
database creates/uses a Mongo 4.2 server instance in the default${HOME}/.ros/topic_store/databasefolder exposed onlocalhost:65530(defined instorage.configof the scenario file).
Docker is a requirement to use a database backend to avoid conflicts with system mongo. A utility script is provided inscripts/install_docker.shif you do not have it installed.Launch your data collection scenario!roslaunch topic_store run_scenario.launch scenario_file:="/path/to/your/scenario/file.yaml"ExamplesDatabase dataThe below example shows how to load and use files stored in a database.importtopic_storeasts# Read datastorage=ts.load("/path/to/scenario/file/containing/db/connection/info.yaml")foriteminstorage:print("As Python Dict",item.dict)# or item["key"]print("As ROS Msgs",item.msgs)# or item("key")# Write datastorage.insert_one({"important_data":"topic store is great!"})Filesystem dataThe below example shows how to load and use.topic_storefiles, saved from when scenarios are ran with thestorage_method="filesystem"option.importtopic_storeasts# Read datastorage=ts.load("/path/to/file.topic_store")foriteminstorage:print("As Python Dict",item.dict)# or item["key"]print("As ROS Msgs",item.msgs)# or item("key")# Write datastorage.insert_one({"important_data":"topic store is great!"})Launch a databaseWhen launching a data collection scenario wherestorage.method==databaseyou must also launch the database or
ensure it's already running at the uri defined in the host/port parameters of the config file located atstorage.config.To launch a database prior to running a data collection scenario.roslaunchtopic_storestart_database.launchscenario_file:="/path/to/your/scenario/file.yaml"Convert to ROS bagsFilesystem.topic_storefiles and database collections can be converted to ROS bags.# Convert filesystem files to rosbagsrosruntopic_storeconvert.py-iinput.topic_store-ooutput.bag# Convert database collections to rosbags# Pass scenario file containing connection and collection info as the inputrosruntopic_storeconvert.py-iscenario_config.yaml-ooutput.bagConvert between Filesystem and DatabaseFilesystem.topic_storefiles can be migrated to a MongoDB database. Pass the scenario file that contains the database
connection information as the output file.rosruntopic_storeconvert.py-iinput.topic_store-oscenario_config.yamlDatabase collections can be converted to a filesystem.topic_storefile. Pass the scenario file that contains the database
connection information as the input file.rosruntopic_storeconvert.py-iscenario_config.yaml-ooutput.topic_storeExport from Mongodb query into rosbag/filesystemExample call:With typical mongodb URI for SSL and authenticationOn databaseff_rasberryand collection2020_riseholme_framos_camerasSpecify database connection URI (-i) with db nameauthSource=databaseIncluding a query (-q) for a specific documentAdd projection to return only sub-documents (-p)convert.py-i"mongodb://USER:PASS@HOST:PORT/?authSource=ff_rasberry&tls=true&tlsAllowInvalidCertificates=true"-c2020_riseholme_framos_cameras-q'{"_id":"ObjectId(5f115ee6af915351df739757)"}'-p'{"cameras.top.color":1, "robot": 1}'-oout.bagStoring high bandwidth image dataTo store a lot of image data you will be limited to the IO of your machine.Topic Compressionoffers a compression solution to increase capture performance.# Install the ROS packagecdcatkin_ws/src
gitclonehttps://github.com/RaymondKirk/topic_compression
catkinbuildtopic_compression# Compression/Decompression is chosen automatically so just pass the input topic name and optionally the out topic namerosruntopic_compressionrunin:=/camera/colour/image_raw# out=/camera/colour/image_raw/compressedrosruntopic_compressionrunin:=/camera/depth/image_raw# out=/camera/depth/image_raw/compressed