Package: masnet
masnetmasnetincludes basic tools for Mastodon network/graph analysis.It can be installed by executingpip install masnet.After installation, following commands can be used:masnet.download: downloads peers information from Mastodon instances.masnet.generate: creates Mastodon graph using the peers information downloaded.masnet.analyze: performs basic graph/network analysis on Mastodon graph.There a few common arguments to each command:-d <directory>: uses the directory instead of the current directory to read and/or write files to. If the directory specified does not exist, it is created. This is important asmasnet.downloadcreates a lot of files andmasnet.generatereads them.-v: enables verbose logging. It should not be normally needed.--debug: enables debug logging. It is only to be used during development.Each command provide verbose logging with-vargument and debug logging with--debugargument.In the software and documentation, I use both Graph and Network interchangeably, but instead of Vertex and Edge, I try to stick to terms Node and Link.The input and output file names cannot be customized, as I think the tools will be run on a clean directory (due to large number of files).Hardware RequirementsMastodon graph has over 10K nodes and more than 20M links. It is not a large graph, but also not small.The tools inmasnethave different requirements:masnet.downloadis network I/O intensive. It is implemented as an async single-thread app using Python'sasyncioand async librariesaiohttp,aiodnsandaiofile.masnet.generateis memory intensive. It consumes more than 4GB of memory at its peak. I recommend having more than 4GB of free memory, hence a computer with 8GB of memory should be OK.masnet.analyzeis compute intensive and uses less memory thanmasnet.generate. As the algorithms are optimized with OpenMP innetworkitpackage, the more cores you have the faster it will run.I am running the tools on an Ubuntu Linux virtual machine with 8 cores and 16GB of memory.masnet.downloadInput Files: none

Output Files:
    - <domain>.peers.json files
    - masnet.download.visits
    - masnet.download.errors
    - masnet.download.skips
    - masnet.download.timesmasnet.downloadconnects to a given start domain (given with-sargument), downloads its peers (withapi/v1/instance/peersAPI call), and then repeats this for each peer until all domains are visited. If start domain is not specified,mastodon.socialis used.After downloading the peers information of a domain, it is saved to<domain>.peers.json.peers.jsonfile is not the same asinstance/peersAPI response, it is augmented with the domain name, thus contains adomainkey with the value of domain name, and apeerskey with the list of domain names of peers.For network connections, a timeout in seconds can be specified with-targument. System default timeout is usually too high, I recommend experimenting with lower values such as 30 (seconds).Because the download process is mostly network I/O, a number of tasks are needed to fetch peers data of different domains. These tasks are implemented as coroutines in Python. The number of tasks can be specified with-nargument (default is 100).Because there are (many) malicious or irrelevant domains in Mastodon network, it is possible to exclude them from traversal as they are seen in peers. Exclusion is specified as regular expression patterns. The name of the file containing such patterns can given with-eargument. If not specified, a default list (masnet/default_exclusion_patterns) is used. This default list excludes all private IP spaces, special domain names and known malicious domains at the time of release. It also filters URLs (names containing/).masnet.downloadis a long running process. The execution ofmasnet.downloadcan be terminated withCtrl-C. Since it is a long running process, it might be a good idea to pipe the output toteeand save the output to a log file.All expected errors are handled gracefully with no stack trace printed to stdout or stderr. If you see any stack trace, that means there is an unexpected error and you can report it as an issue on GitHub.At the end of an execution (also when terminated by Ctrl-C), in addition to (many)<domain>.peers.jsonfiloes, 4 more files are generated:masnet.download.visits: list of (successfully) visited domainsmasnet.download.errors: list of domains where an error is encountered. For each domain, after a space, also the error message is saved.masnet.download.skips: list of skipped domains due exclusion patternsmasnet.download.times: list of download times (in ms) of each peers.json fileAll files other than<domain>.peers.jsonfiles are only for information. Only<domain>.peers.jsonfiles are used bymasnet.generate.An example run, with 30s timeout, saving files to current directory would be:$ masnet.download -t 30This command will print (to stdout) a status line like:q:000000 a:000001 s:138374 N:011867 L:039653324 e:00126507 s:01990292 to:14706 t:00:39:41The meaning of the fields are:q: # of domains in queue (of which peers information will be fetched)a: # of async taskss: # of scheduled domains (already downloaded and will be downloaded)N: # of domains of which peers information is fetched successfullyL: # of links observede: # of domains where an error happened during fetchs: # of skipped domains (due to exclusion)to: number of connections gave timeout errort: elapsed time hh:mm:ssThe status line is printed every second and at the end of the execution.The links observed reported withLis higher than whatmasnet.generatewill report. This is becausemasnet.downloadcounts links also for probably error returning domains. The number of nodesNand number of links observedLare only for information, accurate number of nodes and links will be reported bymasnet.generate.You might see increasing (much more than N) error e and skipped s numbers. At least at the moment, there are many domains in Mastodon network that are either malicious (there are many randomly named subdomains of some domains) or invalid like a private IP. Hence, there are much more domains in peers than the ones actually working.masnet.downloadterminates automatically when there are zero elements observed in queue and there is only one task (which is the main program, that means no download tasks) for 5 seconds.On 2022-12-18,masnet.download -t 30 -n 100andmasnet.download -t 30 -n 1000both completes around 75 minutes with the following outputs:$ masnet.download -t 30 -n 100
....
q:000000 a:000001 s:139500 N:013678 L:047071155 e:00125822 s:02981916 to:4241 t:01:15:26$ masnet.download -t 30 -n 1000
...
q:000000 a:000001 s:139483 N:013688 L:047105958 e:00125795 s:02990283 to:3956 t:01:13:15If you enable verbose mode with-vargument, after the status line, top 5 parent domains encountered during graph traversal will be displayed like below (parent meaning the top domain name part is removed, for a.b.com, it is b.com). This would help to identify malicious or invalid domains.masnet.generateInput Files: <domain>.peers.json files

Output Files: 
    - mastodon.networkit 
    - mastodon.labelsmasnet.generatereads all<domain>.peers.jsonfiles saved bymasnet.downloadand creates a graph and saves it in networkit Binary format tomastodon.networkit.directedandmastodon.networkit.undirectedfiles. The Mastodon peers network is normally directed, but undirected version is also saved by ignoring the direction of peer relationship. In addition to the graphs, the actual labels (domains) are also save intomastodon.labelsfile. Both of these files are read bymasnet.analyze. Both directed and undirected networks have same nodes and node ids.Each<domain>.peers.jsonfile (thus a working domain) will be represented by a node in the graph, and it will have connections to its peers as long as the peer also has its<domain>.peers.jsonfile. Thus if a domain returns error (for the API call) or skipped/exluded, it is also skipped in the generated network, no such node will exist.$ masnet.generate
masnet v0.2
creating the graph from peers...
N:00000000 L:00000000 e:00:00:00 
N:00000562 L:00000000 e:00:00:01 
... repeats lines like this ...
:00011775 e:23631551 t:00:00:38 
graph created.
labels saved.
saving graphs...
0% 46% 76% 99% 100%
0% 26% 49% 70% 91% 100%
graphs saved.
bye.whereNmeans number of nodes,Lmeans number of links andemeans elapsed time.This is a relatively fast operation, it completes under a minute.masnet.analyzeInput Files: 
    - mastodon.networkit.directed or mastodon.network.undirected
    - mastodon.labels

Output Files: depends on the analysismasnet.analyzeperforms basic graph/network analysis on Mastodon graph usingnetworkitlarge-scale network analysis toolkit. As the core ofnetworkitis written in C++ using OpenMP, it is quite a fast implementation.The input graph file is selected using--directedargument. By default, the undirected graph is used (mastodon.networkit.undirected).Runningmasnet.analyzewith--cardargument shows thenetwork cardof Mastodon network specified inmastodon.networkit.$ masnet.analyze --card
masnet v0.3
node labels loaded.
reading undirected graph into networkit...
graph read in 0.6 seconds.
generating the network card...
----------------------------  --------------------------------
Name                          Mastodon peers network
Kind                          undirected, unweighted
Nodes are                     Mastodon instances
Links are                     Peer relationship
----------------------------  --------------------------------
Number of nodes               13688
Number of links               16010592
Degree*                       2339.362 [0, 12881]
Clustering                    0.754
Connected                     2 components [100.0% in largest]
Component size*               6844.0 [1, 13687]
Diameter                      n/a
Largest component's diameter  4
----------------------------  --------------------------------
Data generating process       masnet.download, masnet.generate
----------------------------  --------------------------------
*: avg [min, max]Each analysis is requested with an argument tomasnet.analyze. Please check the help with--helpto see a list. At the moment, the following analysis arguments are supported (there are others mark experimental which I am developing/testing):--card: shows the network cardDemoOnnetwork.bastodon.org, the network card of the daily snapshot of Mastodon peer network can be seen.Release History0.3.6:doc fix0.3.5:masnet.generate fix0.3.4:masnet.analyze --card fix for directed graphs0.3.3:doc fix0.3.2:masnet.analyze --card file support0.3.1:pypi fix0.3:major rewrite of masnet.downloadsome changes on masnet.generate and masnet.analyze0.2.1:terms fixed to Node and Link (rather than Vertex and Edge)masnet.analyze improvementsmasnet.downloads fix for termination of download threads0.2:major rewrite of masnet.downloadinitial release of masnet.generateinitial release of masnet.analyze0.1.1:small fixes0.1:initial release of masnet.download