Package: tagpack-tool
GraphSense TagPack Management ToolThis repository provides a command line tool for managingGraphSense TagPacks. It can be used forvalidating TagPacks against the TagPack schemafinding suitable actors for tagsvalidating ActorPacks against the ActorPack schemahandling taxonomies and conceptsingesting TagPacks and related data into a TagStorecalculating the quality of the tags in the TagStorePlease note that the last feature requires (installation of) aPostgresqldatabase.QuickstartInstallationpip install git+https://github.com/graphsense/graphsense-tagpack-tool.gitPrepare a TagStore databaseCheck out the options as describedbelow.Sync TagPack repositoriesCreate a file containing the repositories you want to manage, one repository per line (commenting out lines is possible):git@github.com:graphsense/graphsense-tagpacks.git develop public
# git@github.com:mycompany/graphsense-tagpacks-special.git masterIf you want to import a certain branch add the branch name separated by a white-space as shown above. To indicate that the repository should be imported to seen by everybody then add the keyword after the branch specification. If no branch or public keyword is specified the default branch is used and the tags are treated as private.Then runtagpack-tool sync -r ./tagpack-repos.configto populate the TagStore with Actors and TagPacks.Re-run the command to add newly added tagpack files from the repositories.Add the--forceoption to re-insert TagPacks.Step-by-step overviewValidate a TagPackValidate a single TagPack filetagpack-tool tagpack validate tests/testfiles/simple/ex_addr_tagpack.yamlRecursively validate all TagPacks in (a) given folder(s).tagpack-tool tagpack validate tests/testfiles/TagPacks are validated against thetagpack schema.Confidence settings are validated against a set of acceptableconfidencevalues.Actors for tags and TagPacksActorsare defined in a curated actor tagpack.It is highly encouraged to add suitable actors to TagPacks whenever possible,
and the tagpack-tool offers support for doing so.List suitable actors for a tagFor a specific tag string, actor suggestions can be listed by callingtagpack-tool tagpack suggest_actors <my_tag>and if desired, the number of results can be restricted by adding the--maxparametertagpack-tool tagpack suggest_actors --max 1 <my_tag>Interactive TagPack updateIt is also possible to interactivelyupdatean existing TagPack file with actors:tagpack-tool tagpack add_actors path/to/tagpack.yamlor go through entire directories of TagPack files:tagpack-tool tagpack add_actors path/to/tagpacksFile by file, for each label, the tagpack-tool will suggest suitable actors if any are found:Choose for instadapp_InstaCompoundMapping
    0 instadapp
    1 compound
    ENTER to skip
Your choice: 0The--maxoption is available again to limit the number of candidate suggestions:tagpack-tool tagpack add_actors --max 1 path/to/tagpacksIf any actors have been selected, an updated TagPack is written that contains the users' selected actors:Writing updated Tagpack defi-protocols_instadapp_with_actors.yamlValidate an ActorPackValidate a single ActorPack filetagpack-tool actorpack validate tests/testfiles/actors/ex_actorpack.yamlRecursively validate all TagPacks in (a) given folder(s).tagpack-tool actorpack validate tests/testfiles/actors/Actorpacks are validated against theactorpack schema.Values in the field jurisdictions are validated against a set ofcountry codes.View available taxonomies and conceptsList configured taxonomy keys and URIstagpack-tool taxonomy listFetch and show concepts of a specific remote/local taxonomy (referenced by key: abuse, entity, confidence, country)tagpack-tool taxonomy show entityIngest TagPacks and related data into a TagStorePrerequisites: TagStore - PostgreSQL databaseOption 1: Start a dockerized PostgreSQL databaseDocker, see e.g.https://docs.docker.com/engine/install/Docker Compose:https://docs.docker.com/compose/install/First, copydocker/env.templateto.envand fill the fieldsPOSTGRES_PASSWORDandPOSTGRES_PASSWORD_TAGSTORE.Runcp docker/postgres-conf.sql.template postgres-conf.sqland modify the configuration parameters to your requirements. If no special config is needed an emtpy file is also valid.touch postgres-conf.sqlStart a PostgreSQL instance using Docker Compose:docker-compose up -dThis will automatically create the database schema as defined
insrc/tagpack/db/tagstore_schema.sql.Option 2: Use an existing PostgreSQL databaseCreate the schema and tables in a PostgreSQL instance of your choicepsql -h $POSTGRES_HOST -d $POSTGRES_DB -U $POSTGRES_USER --password -f src/tagpack/db/tagstore_schema.sqlExport .env variablestagpack-tool is able to use the variables configured in the.envfile to avoid specifying the parameter--urleach time it connects to the database. The--urlparameter will override the environment values if needed. To export the environment variables in.envfrom a linux shell (e.g. bash), first use:source .env
export $(grep --regexp ^[A-Z] .env | cut -d= -f1)Or just export each variable using:export POSTGRES_USER=VALUE
export POSTGRES_PASSWORD=VALUE
export POSTGRES_HOST=VALUE
export POSTGRES_DB=VALUEThen call tagpack-tool.Create and display a configuration file that defines which taxonomies to useTo create a default configurationconfig.yamlfile from scratch - i.e. when config.yaml does not exist - use:tagpack-tool configIf a config.yaml already exists, it will not be replaced.Show the contents of the config file:tagpack-tool config -vTo use a specific config file pass the file's location:tagpack-tool --config  path/to/config.yaml configInitialize the tagstore databaseTo initialize the database with all the taxonomies needed for ingesting the tagpacks, use:tagpack-tool tagstore initIngest taxonomies and confidence scoresTo insert individual taxonomies into database, use:tagpack-tool taxonomy insert abuse
tagpack-tool taxonomy insert entity
tagpack-tool taxonomy insert confidence
tagpack-tool taxonomy insert countryTo insert all configured taxonomies at once, simply omit taxonomy nametagpack-tool taxonomy insertIngest TagPacksInsert a single TagPack file or all TagPacks from a given foldertagpack-tool tagpack insert tests/testfiles/simple/ex_addr_tagpack.yaml
tagpack-tool tagpack insert tests/testfiles/simple/multiple_tags_for_address.yaml
tagpack-tool tagpack insert tests/testfiles/By default, TagPacks are declared as non-public in the database.
For public TagPacks, add the--publicflag to your arguments:tagpack-tool tagpack insert --public tests/testfiles/If you try to insert tagpacks that already exist in the database, the ingestion process will be stopped.To forcere-insertion(if tagpack file contents have been modified), add the--forceflag to your arguments:tagpack-tool tagpack insert --force tests/testfiles/To ingestnewtagpacks andskipover already ingested tagpacks, add the--add_newflag to  your arguments:tagpack-tool tagpack insert --add_new tests/testfiles/By default, trying to insert tagpacks from a repository withlocalmodifications willfail.
To force insertion despite local modifications, add the--no_strict_checkcommand-line parametertagpack-tool tagpack insert --no_strict_check tests/testfiles/By default, tagpacks in the TagStore provide a backlink to the original tagpack file in their remote git repository.
To write local file paths instead, add the--no_gitcommand-line parametertagpack-tool tagpack insert --no_git --add_new tests/testfiles/Ingest ActorPacksInsert a single ActorPack file or all ActorPacks from a given folder:tagpack-tool actorpack insert tests/testfiles/simple/ex_addr_actorpack.yaml
tagpack-tool actorpack insert tests/testfiles/You can use the parameters--force,--add_new,--no_strict_checkand--no_gitoptions in the same way as with thetagpackcommand.Align ingested attribution tags with GraphSense cluster IdsThe final step after inserting a tagpack is to fetch the corresponding
Graphsense cluster mapping ids for the crypto addresses in the tagpack.Copysrc/tagpack/conf/ks_map.json.templatetoks_map.jsonand edit the file to
suit your Graphsense setup.Then fetch the cluster mappings from your Graphsense instance and insert them
into the tagstore database:tagpack-tool tagstore insert_cluster_mappings -d $CASSANDRA_HOST -f ks_map.jsonTo update ALL cluster-mappings in your tagstore, add the--updateflag:tagpack-tool tagstore insert_cluster_mappings --update -d $CASSANDRA_HOST -f ks_map.jsonRemove duplicate tagsDifferent tagpacks may contain identical tags - the same label and source for a particular address.
To remove such redundant information, runtagpack-tool tagstore remove_duplicatesIMPORTANT: Keeping data consistency after tagpack insertionAfter all required tagpacks have been ingested, runtagpack-tool tagstore refresh_viewsto update all materialized views.
Depending on the amount of tags contained in the tagstore, this may take a while.Connection Pooling for PostgreSQLFor setups which expect many parallel connections to the tagstore it can be a good option to run all connections over a dedicated connection-pooler (to avoid exhausting the connections). The docker-compose file used to start the postgres instance automatically starts a pg-bounce container as well. The pg-bounce instance runs on port 6432 and can be used as a drop in replacement for the standard pgsql connections over port 5432. To use pg-bounce as connection-pooler configure the additional environment variablesPOSTGRES_USER_TAGSTORE=<user that is used to connect to the tagstore>
POSTGRES_PASSWORD_TAGSTORE=<PASSWORD>for example in the your local .env file. Currently, the pg-bounce setup only allows connections with this specific user configured in POSTGRES_USER_TAGSTORE.Calculate the quality of the tags in the TagStoreTo assess on the quality of address tags we define a quality measure.
For an address tag, it is calculated as theweighted similarity distancebetween all pairs of distinct tags assigned to the same address.An address with a unique tag has a quality equal to 1, while an address with several similar tags has a quality close to 0.To calculate the quality measure for all the tags in the database, run:tagpack-tool quality calculateTo show the quality measures of all the tags in the database, or those of a specific crypto-currency, run:tagpack-tool quality show [--currency [BCH|BTC|ETH|LTC|ZEC]]Show tagstore contents/contributionsTo list all tagpack creators and their contributions to a tagstore's content use:tagpack-tool tagstore show_compositionWorking in development / testing modegit clone https://github.com/graphsense/graphsense-tagpack-tool.git
cd graphsense-tagpack-toolUsing Pip locallyCreate and activate a python environment for required dependencies and activate itVenvpython3 -m venv venv
source venv/bin/activateCondaconda create -n tagpack-tool
conda activate tagpack-toolInstall package and dependencies in local environmentconda install pip
make install-devLinting and FormattingThe code in this repos will be autoformated via black and linted via a pre-commit hook. To manually format and lint the code run:make format && make pre-commitOr linting via toxtox -l lintBuild for Publishingmake buildCreate Html Docsmake docsTestingRun testsmake testOr via toxtoxCheck test coverage (optional)make test
coverage reportUseactto check if test viaGithub actionpass.