Package: lbsntransform
LBSNTransformA python package that uses thecommon location based social network (LBSN) data structure(ProtoBuf) to import, transform and export Social Media data such as Twitter and Flickr.MotivationThe goal is to provide a common interface to handle Social Media Data,
without the need to individually adapt to the myriad API endpoints available.
As an example, consider the ProtoBuf speclbsn.Post, which can be a Tweet on Twitter,
a Photo shared on Flickr, or a post on Reddit. However, all of these objects share
a common set of attributes, which is reflected in the lbsnstructure.The tool is based on a 4-Facet conceptual framework for LBSN, introduced in a paper
byDunkel et al. (2018).The GDPR directly requests Social Media Network operators to allow
users to transfer accounts and data in-between services.
While there are attempts by Google, Facebook etc. (e.g. see thedata-transfer-project),
this is not currently possible. With the lbsnstructure, a primary motivation is to systematically
characterize LBSN data aspects in a common, cross-network data scheme that enables privacy-by-design
for connected software, data handling and database design.DescriptionThis tool enables data import from a Postgres database, JSON, or CSV and export to CSV,LBSN ProtoBufor thehllandrawversions of the LBSN prepared Postgres Databases.
The tool will map Social Media endpoints (e.g. Twitter tweets) to a commonLBSN Interchange Structureformat in ProtoBuf. LBSNTransform can be used using the command line (CLI) or imported to other Python projects withimport lbsntransform, for on-the-fly conversion.Quick StartThe recommended way to install lbsntransform, for both Linux and Windows,
is through the conda package manager.Create a conda env usingenvironment.ymlFirst, create an environment with the dependencies for lbsntransform using
the [environment.yml][environment.yml] that is provided in the root of the repository.gitclonehttps://github.com/Sieboldianus/lbsntransform.gitcdlbsntransform# not necessary, but recommended:condaconfig--env--setchannel_prioritystrict
condaenvcreate-fenvironment.ymlInstall lbsntransform without dependenciesAfterwards, install lbsntransform using pip, without dependencies.condaactivatelbsntransform
pipinstalllbsntransform--no-deps--upgrade# or locally, from the latest commits on master# pip install . --no-deps --upgradeImport data using a mappingFor each data source, a mapping must be provided that
defines how data is mapped to thelbsnstructure.The default mapping islbsnraw.Additional mappings can be dynamically loaded from a folder.We have provided twoexample mappingsfor theFlickr YFCC100M dataset(CSV)
and Twitter (json).For example, to import the first 1000 records from json data from Twitter to thelbsn raw database, clonefield_mapping_twitter.pyto a local folder./resources/mappings/, startup the Dockerrawdbcontainer,
and use:lbsntransform--origin3\--mappings_path./resources/mappings/\--file_input\--file_type"json"\--mappings_path./resources/mappings/\--dbpassword_output"sample-key"\--dbuser_output"postgres"\--dbserveraddress_output"127.0.0.1:5432"\--dbname_output"rawdb"\--dbformat_output"lbsn"\--transferlimit1000.. with the above input args, the the tool will:read local json from./01_Input/and store lbsn records to thelbsn rawdb.Vice versa, to import data directly to the privacy-aware
version of lbsnstructure, calledhlldb, startup the
Docker container, and use:lbsntransform--origin3\--mappings_path./resources/mappings/\--file_input\--file_type"json"\--mappings_path./resources/mappings/\--dbpassword_output"sample-key"\--dbuser_output"postgres"\--dbserveraddress_output"127.0.0.1:25432"\--dbname_output"hlldb"\--dbformat_output"hll"\--dbpassword_hllworker"sample-key"\--dbuser_hllworker"postgres"\--dbserveraddress_hllworker"127.0.0.1:25432"\--dbname_hllworker"hlldb"\--include_lbsn_objects"origin,post"\--include_lbsn_baseshashtag,place,date,community\--transferlimit1000.. with the above input args, the the tool will:read local json from./01_Input/and store lbsn records to the privacy-awarelbsn hlldbby converting only lbsn objects of typeoriginandpostand updating the HyperLogLog (HLL) target tableshashtag,place,dateandcommunityA full list of possible input and output args is available in thedocumentation.Built Withlbsnstructure- A common language independend and cross-network social-media dataschemeprotobuf- Google's data interchange formatpsycopg2- Python-PostgreSQL Database Adapterppygis3- A PPyGIS port for Pythonshapely- Geometric objects processing in Pythonemoji- Emoji handling in PythonAuthorsAlexander Dunkel- Initial workSee also the list ofcontributors.LicenseThis project is licensed under the GNU GPLv3 or any higher -
see theLICENSE.mdfile for details.