Package: fediblockhole
FediBlockHoleA tool for keeping a Mastodon instance blocklist synchronised with remote lists.The broad design goal for FediBlockHole is to support pulling in a list of
blocklists from a set of trusted sources, merge them into a combined blocklist,
and then push that merged list to a set of managed instances.Mastodon admins can choose who they think maintain quality lists and subscribe
to them, helping to distribute the load for maintaining blocklists among a
community of people. Control ultimately rests with the admins themselves so they
can outsource as much, or as little, of the effort to others as they deem
appropriate.Inspired by the way PiHole works for maintaining a set of blocklists of adtech
domains. Builds on the work of@CaribenxMarciaX@scholar.socialand@gingerrroot@kitty.townwho started the
#Fediblock hashtag and did a lot of advocacy around it, often at great personal
cost.FeaturesBlocklist SourcesRead domain block lists from other instances via the Mastodon API.Supports both public lists (no auth required) and 'admin' lists requiring
authentication to an instance.Read domain block lists from arbitrary URLs, including local files.Supports CSV and JSON format blocklistsSupports RapidBlock CSV and JSON format blocklistsBlocklist Export/PushPush a merged blocklist to a set of Mastodon instances.Export per-source, unmerged block lists to local files, in CSV format.Export merged blocklists to local files, in CSV format.Read block lists from multiple remote instancesRead block lists from multiple URLs, including local filesWrite a unified block list to a local CSV filePush unified blocklist updates to multiple remote instancesControl import and export fieldsFlexible ConfigurationProvides (hopefully) sensible defaults to minimise first-time setup.Global and fine-grained configuration options available for those complex situations that crop up sometimes.Allowlists to override blocks in blocklists to ensure you never block instances you want to keep.Blocklist thresholds if you want to only block when an instance shows up in multiple blocklists.InstallingInstallable usingpip.python3 -m pip install fediblockholeInstall from source by cloning the repo,cd fediblockholeand run:python3 -m pip install .Installation adds a commandline tool:fediblock-syncInstance admins who want to use this tool for their instance will need to add an
Application athttps://<instance-domain>/settings/applications/so they can
authorize the tool to create and update domain blocks with an OAuth token.More on authorization by token below.Reading remote instance blocklistsIf a remote instance makes its domain blocks public, you don't need
a token to read them.If a remote instance only shows its domain blocks to local accounts
you'll need to have a token withread:blocksauthorization set up.
If you have an account on that instance, you can get a token by setting up a new
Application athttps://<instance-domain>/settings/applications/.To read admin blocks from a remote instance, you'll need to ask the instance
admin to add a new Application athttps://<instance-domain>/settings/applications/and then tell you the access
token.The application needs theadmin:read:domain_blocksOAuth scope. You can allow
fulladmin:readaccess, but be aware that this authorizes someone to read all
the data in the instance. That's asking a lot of a remote instance admin who
just wants to share domain_blocks with you.Theadmin:read:domain_blocksscope is available as of Mastodon v4.1.0, but for
earlier versions admins will need to use the manual method described below.You can update the scope for your application in the database directly like
this:UPDATE oauth_applications as app
  SET scopes = 'admin:read:domain_blocks'
  FROM oauth_access_tokens as tok
  WHERE app.id = tok.application_id
  AND app.name = '<the_app_name>'
;When that's done, regenerate the token (so it has the new scopes) in the
application screen in the instance GUI. FediBlockHole should then able to use
the app token to read domain blocks via the API, but nothing else.Alternately, you could ask the remote instance admin to set up FediBlockHole and
use it to dump out a CSV blocklist from their instance and then put it somewhere
trusted parties can read it. Then you can define the blocklist as a URL source,
as explained below.Writing instance blocklistsTo write domain blocks into an instance requires both theadmin:readandadmin:write:domain_blocksOAuth scopes.The tool needsadmin:read:domain_blocksscope to read the current list of
domain blocks so we update ones that already exist, rather than trying to add
all new ones and clutter up the instance.admin:readaccess is needed to check if the instance has any accounts that
follow accounts on a domain that is about to getsuspended and automatically
drop the block severity tosilencelevel so people have time to migrate
accounts before a full defederation takes effect. Unfortunately, the statistics
measure used to learn this information requiresadmin:readscope.You can addadmin:readscope in the application admin screen. Please be aware
that this grants full read access to all information in the instance to the
application token, so make sure you keep it a secret. At least remove
world-readable permission to any config file you put it in, e.g.:chmod o-r <configfile>You can also grant fulladmin:writescope to the application, but if you'd
prefer to keep things more tightly secured, limit the scope toadmin:read:domain_blocks.Again, this scope is only available in the application config screen as of
Mastodon v4.1.0. If your instance is on an earlier version, you'll need to use
SQL to set the scopes in the database and then regenerate the token:UPDATE oauth_applications as app
  SET scopes = 'admin:read admin:write:domain_blocks'
  FROM oauth_access_tokens as tok
  WHERE app.id = tok.application_id
  AND app.name = '<the_app_name>'
;When that's done, FediBlockHole should be able to use its token to authorise
adding or updating domain blocks via the API.Using the toolRun the tool like this:fediblock-sync -c <configfile_path>If you put the config file in/etc/default/fediblockhole.conf.tomlyou don't
need to pass in the config file path.For a list of possible configuration options, check the--help.You can also read the heavily commented sample configuration file in the repo atetc/sample.fediblockhole.conf.toml.ConfiguringOnce you have your applications and tokens and scopes set up, create a
configuration file for FediBlockHole to use. You can put it anywhere and use the-c <configfile>commandline parameter to tell FediBlockHole where it is.Or you can use the default location of/etc/default/fediblockhole.conf.toml.As the filename suggests, FediBlockHole uses TOML syntax.There are 4 key sections:blocklist_urls_sources: A list of URLs to read blocklists fromblocklist_instance_sources: A list of Mastodon instances to read blocklists from via APIblocklist_instance_destinations: A list of Mastodon instances to write blocklists to via APIallowlist_url_sources: A list of URLs to read allowlists fromMore detail on configuring the tool is provided below.URL sourcesThe URL sources is a list of URLs to fetch blocklists from.Supported formats are currently:Comma-Separated Values (CSV)JSONMastodon v4.1 flavoured CSVRapidBlock CSVRapidBlock JSONBlocklists must provide adomainfield, and should provide aseverityfield.domainis the domain name of the instance to be blocked/limited.severityis the severity level of the block/limit. Supported values are:noop,silence, andsuspend.Optional fields that the tool understands arepublic_comment,private_comment,reject_media,reject_reports, andobfuscate.CSV formatA CSV format blocklist must contain a header row with at least adomainandseverityfield.Optional fields, as listed about, may also be included.Mastodon v4.1 CSV formatAs of v4.1.0, Mastodon can export domain blocks as a CSV file. However, in their
infinite wisdom, the Mastodon devs decided that field names should begin with a#character in the header, unlike the field names in the JSON output via the
API… or in pretty much any other CSV file anywhere else.Setting the format tomastodon_csvwill strip off the#character when
parsing and FediBlockHole can then use Mastodon v4.1 CSV blocklists like any
other CSV formatted blocklist.JSON formatJSON is also supported. It uses the same format as the JSON returned from the Mastodon API.This is a list of dictionaries, with at minimum adomainfield, and preferably
aseverityfield. The other optional fields are, well, optional.RapidBlock CSV formatThe RapidBlock CSV format has no header and a single field, so it's notstrictlya CSV file as there are no commas separating values. It is basically
just a list of domains to block, separated by '\r\n'.When using this format, the tool assumes theseveritylevel issuspend.RapidBlock JSON formatThe RapidBlock JSON format provides more detailed information about domain
blocks, but is still somewhat limited.It has a singleisBlockedflag indicating if a domain should be blocked or
not. There is no support for the 'silence' block level.There is no support for 'reject_media' or 'reject_reports' or 'obfuscate'.All comments are public, by virtue of the public nature of RapidBlock.Instance sourcesThe tool can also read domain_blocks from instances directly.The configuration is a list of dictionaries of the form:{ domain = '<domain_name>', token = '<BearerToken>', admin = false }Thedomainis the fully-qualified domain name of the API host for an instance
you want to read domain blocks from.Thetokenis an optional OAuth token for the application that's configured in
the instance to allow you to read domain blocks, as discussed above.adminis an optional field that tells the tool to use the more detailed admin
API endpoint for domain_blocks, rather than the more public API endpoint that
doesn't provide as much detail. You will need atokenthat's been configured to
permit access to the admin domain_blocks scope, as detailed above.Instance destinationsThe tool supports pushing a unified blocklist to multiple instances.Configure the list of instances you want to push your blocklist to in theblocklist_instance_detinationslist. Each entry is of the form:{ domain = '<domain_name>', token = '<BearerToken>', import_fields = ['public_comment'], max_severity = 'suspend', max_followed_severity = 'suspend' }The fieldsdomainandtokenare required.The fieldsmax_followed_severityandimport_fieldsare optional.Thedomainis the hostname of the instance you want to push to. Thetokenis
an application token with bothadmin:read:domain_blocksandadmin:write:domain_blocksauthorization.The optionalimport_fieldssetting allows you to restrict which fields are
imported from each instance. If you want to import thereject_reportssettings
from one instance, but no others, you can use theimport_fieldssetting to do
it.Note:Thedomainandseverityfields are always imported.The optionalmax_severitysetting limits the maximum severity you will allow a
remote blocklist to set. This helps you import a list from a remote instance but
only at thesilencelevel, even if that remote instance has a block atsuspendlevel. If not set, defaults tosuspend.The optionalmax_followed_severitysetting sets a per-instance limit on the
severity of a domain_block if there are accounts on the instance that follow
accounts on the domain to be blocked. Ifmax_followed_severityisn't set, it
defaults tosilence.This setting exists to give people time to move off an instance that is about to
be defederated and bring their followers from your instance with them. Without
it, if a newsuspendblock appears in any of the blocklists you subscribe to (or
a block level increases fromsilencetosuspend) and you're using the defaultmaxmergeplan, the tool would immediately suspend the instance, cutting
everyone on the blocked instance off from their existing followers on your
instance, even if they move to a new instance. If you actually want that
outcome, you can setmax_followed_severity = 'suspend'and use themaxmergeplan.Once the follow count drops to 0 on your instance, the tool will automatically
use the highest severity it finds again (if you're using themaxmergeplan).AllowlistsSometimes you might want to completely ignore the blocklist definitions for
certain domains. That's what allowlists are for.Allowlists remove any domain in the list from the merged list of blocks before
the merged list is saved out to a file or pushed to any instance.Allowlists can be in any format supported byblocklist_urls_sourcesbut ignore
all fields that aren'tdomain.You can also allow domains on the commandline by using the-Aor--allowflag and providing the domain name to allow. You can use the flag multiple
times to allow multiple domains.It is probably wise to include your own instance domain in an allowlist so you
don't accidentally defederate from yourself.More advanced configurationFor a list of possible configuration options, check the--helpand read the
sample configuration file inetc/sample.fediblockhole.conf.toml.save_intermediateThis option tells the tool to save the unmerged blocklists it fetches from
remote instances and URLs into separate files. This is handy for debugging, or
just to have a non-unified set of blocklist files.Works with thesavedirsetting to control where to save the files.These are parsed blocklists, not the raw data, and so will be affected byimport_fields.The filename is based on the URL or domain used so you can tell where each list came from.savedirSets where to save intermediate blocklist files. Defaults to/tmp.blocklist_auditfileIf provided, will save an audit file of counts and percentages by domain. Useful for debugging
thresholds. Defaults to None.no_push_instanceDefaults to False.When set, the tool won't actually try to push the unified blocklist to any
configured instances.If you want to see what the tool would try to do, but not actually apply any
updates, use--dryrun.no_fetch_urlSkip the fetching of blocklists from any URLs that are configured.no_fetch_instanceSkip the fetching of blocklists from any remote instances that are configured.override_private_commentDefaults to None.Stamp allnewblocks pushed to a remote server with this comment or code.
Helps to identify blocks you've created on a server via Fediblockhole versus ones that
already existed.mergeplanIf two (or more) blocklists define blocks for the same domain, but they're
different,mergeplantells the tool how to resolve the conflict.maxis the default. It uses thehighestseverity block it finds as the one
that should be used in the unified blocklist.mindoes the opposite. It uses thelowestseverity block it finds as the one
to use in the unified blocklist.A full discussion of severities is beyond the scope of this README, but here is
a quick overview of how it works for this tool.The severities are:noop, level 0: This is essentially an 'unblock' but you can include a
comment.silence, level 1: A silence adds friction to federation with an instance.suspend, level 2: A full defederation with the instance.Withmergeplanset tomax,silencewould take precedence overnoop, andsuspendwould take precedence over both.Withmergeplanset tomin,silencewould take precedence oversuspend,
andnoopwould take precedence over both.You would want to usemaxto ensure that you always block with whichever your
harshest fellow admin thinks should happen.You would want to useminto ensure that your blocks do what your most lenient
fellow admin thinks should happen.import_fieldsimport_fieldscontrols which fields will be imported from remote
instances and URL blocklists, and which fields are pushed to instances from the
unified blocklist.The fieldsdomainandseverityare always included, so only define extra
fields, if you want them.You can't export fields you haven't imported, soexport_fieldsshould be a
subset ofimport_fields, but you can run the tool multiple times. You could,
for example, include lots of fields for an initial import to build up a
comprehensive list for export, combined with the--no-push-instancesoption so
you don't actually apply the full list to anywhere.Then you could use a different set of options when importing so you have all the
detail in a file, but only pushpublic_commentto instances.export_fieldsexport_fieldscontrols which fields will get saved to the unified blocklist
file, if you export one.The fieldsdomainandseverityare always included, so only define extra
fields, if you want them.