Package: emerge-viz
EmergeEmerge(oremerge-viz) is an interactive code analysis tool to gather insights about source code structure, metrics, dependencies and complexity of software projects. You can scan the source code of a project, calculate metric results and statistics, generate an interactive web app with graph structures (e.g. a dependency graph or a filesystem graph) and export the results in some file formats. Emerge currently has parsing support for the following languages:C,C++,Groovy,Java,JavaScript,TypeScript,Kotlin,ObjC,Ruby,Swift,Python,Go. The structure, coloring and clustering is calculated and based on the idea of combining aforce-directed graphsimulation andLouvain modularity. emerge is mainly written in Python 3 and is tested on macOS, linux and modern web browsers (i.e. latest Safari, Chrome, Firefox, Edge).Goals of this projectemerge(/ɪˈməːdʒ/)to appear by coming out of something or out from behind somethingto become known, especially as a result of examining something or asking questions about itThe main goal of this project is to create a free/ open source tool, that can easily be used by anyone with interest in software development, architecture, metrics and visualization to gather more insights about those topics. It should facilitate/ support getting a better understanding of a given software project by using anexploratory approach.The following features are currently supported by emergeFile scan support for the following languages:C,C++,Groovy,Java,JavaScript,TypeScript,Kotlin,ObjC,Ruby,Swift,PythonBasic entity scan/extraction (e.g. classes) for the following languages:Groovy,Java,Kotlin,SwiftImplementation of the following software metrics: SLOC, Whitespace Complexity (impl. by A. Tornhill), Number of Methods, Fan-In/Fan-Out, Modularity (Louvain)Experimental implementation of additionalgit-basedmetrics (SLOC, Whitespace Complexity, Change Coupling)Infer meaning by feature/semantic keyword extraction based onterm frequency-inverse document frequencyLogging support with configurable log levelsConfiguration support based on YAML syntax to configure multiple/specific analysesCreate a language/project configuration directly from an included configuration templateExport of scan results/ metrics/ statistics for the following formats/ outputsFile scanDependency graphEntity scanDependency graphInheritance graphComplete graph (composition of dependency and inheritance graph)Includes the extraction ofSwiftUIandComposabledeclarative UI entitiesA Filesystem graph that shows the project filesystem hierarchy as a graphGraphMLJavaScript format suited for aD3 force graph simulationInteractive HTML/ web application for interactive, exploratory analysis and data visualization of your project based on graph structuresHTML app is based onBootstrapForce-directed graph simulation byD3The node colors are based onLouvain modularitywith a bit of post-processing to make the graph coloring more deterministic and stableFast full-screen UI rendering on HTML canvasVisualization of files, entities and given metricsDark mode supportVisual live search (OR'ed with multiple search terms) of entitiesThe option to include a semantic search based onterm frequency-inverse document frequencyThe option to include git-based metainformation e.g. contributor namesSelection and highlighting of individual nodesConcave hullvisualization of single clustersHeatmapvisualization support of potentially harmful nodes based on a SLOC/Fan-Out score[Heatmap] visualization of ofgit-basedmetrics e.g. code churnDisplay of cluster metrics to facilitate comparabilityInteractivity given by translation, zooming, dragging and hovering over nodesTabular console outputTabular file outputJSON file outputHow to install and use emerge as a userBasically there are two ways to install emerge. If you're familiar withpip(a virtual environment by usingpyenv,virtualenvandvirtualenvwrapperis recommended, but not needed) you can simply install the latest version of emerge with the following few steps.0️⃣ ~ (Optional) Setup a virtual environment with pyenvThe recommended way would be to use a virtual env, you can do this by using the following example:pyenv install 3.10.0
pyenv virtualenv 3.10.0 venv-3.10.0
pyenv activate venv-3.10.01️⃣ ~ Install emerge with pipYou can simply install emerge by usingpip.1️⃣.1️⃣ ~ (ubuntu) install prerequisitesOn Ubuntu 20.04+ pleaase make sure that the packagesgraphvizandgraphviz-devare installed, i.e.apt-get install graphviz graphviz-dev1️⃣.2️⃣ ~ Install with pipEither install as new package with:pip install emerge-vizor if it's already installed, just update with:pip install -U emerge-vizand then simply execute it like this:(emerge) user@host ~ % emerge
usage: emerge [-h] [-c YAMLCONFIG] [-v] [-d] [-e] [-a LANGUAGE]

🔎 Welcome to emerge x.y.z (yyyy-mm-dd hh:mm:ss)

options:
  -h, --help            show this help message and exit
  -c YAMLCONFIG, --config YAMLCONFIG
                        set yaml config file
  -v, --verbose         set logging level to INFO
  -d, --debug           set logging level to DEBUG
  -e, --error           set logging level to ERROR
  -a LANGUAGE, --add-config LANGUAGE
                        add a new config from a template, where LANGUAGE is one of [JAVA, SWIFT, C, CPP, GROOVY, JAVASCRIPT,
                        TYPESCRIPT, KOTLIN, OBJC, RUBY, PY, GO]2️⃣ ~ Create and adjust project configurationYou can create a simple project config adhoc from the command line and then simply adjust the necessary source/export paths(emerge) user@host tmp % pwd
/Users/user1/tmp
(emerge) user@host tmp % emerge -a java
✅ created config file from template: /Users/user1/tmp/java-template.yamland then simply adjust the necessary paths (analyses/source_directoryandexport/directory):(emerge) user@host tmp % cat java-template.yaml 
---
project_name: java_project_example
loglevel: info
analyses:
- analysis_name: full java check
  source_directory: /Users/user1/emerge/project/source
  only_permit_languages:
  - java
  only_permit_file_extensions:
  - .java
  file_scan:
  - number_of_methods
  - source_lines_of_code
  - dependency_graph
  - fan_in_out
  - louvain_modularity
  - tfidf
  entity_scan:
  - dependency_graph
  - source_lines_of_code
  - number_of_methods
  - fan_in_out
  - louvain_modularity
  - tfidf
  export:
  - directory: /Users/user1/emerge/project/export
  - graphml
  - json
  - tabular_file
  - tabular_console_overall
  - d3
(emerge) user@host tmp %3️⃣ ~ Start a scanAfter this you can simply start a scan by(emerge) user@host tmp % emerge -c java-template.yaml
2021-12-04 21:18:15   analysis I 👉 starting to analyze java_project_example
2021-12-04 21:18:15   analysis I ⏩ performing analysis 1/1: full java check
2021-12-04 21:18:15   analysis I 👉 starting to create filesystem graph in full java check
2021-12-04 21:18:15   analysis I ⏩ starting scan at directory: ...
...
...
...
2021-12-04 21:18:27   analysis I ✅ all your generated/exported data can be found here: /Users/user1/tmp/java
2021-12-04 21:18:27   analysis I ✅ copy the following path to your browser and start your web app: 👉 file:///Users/user1/tmp/java/html/emerge.html
2021-12-04 21:18:27   analysis I ✅ total runtime of analysis: 00:00:10 + 154 ms4️⃣ ~ Start your web appNow just copy the above mentionedfile://path to any modern web browser and interactively expore your configured codebase 😉How to install and use emerge from source (e.g. for development)You can clone this repository and install it by following this instruction:1️⃣ ~ Clone this repositorygit clone https://github.com/glato/emerge.git2️⃣.1️⃣ ~ (macOS) Install thegraphvizpackage firstbrew install graphvizIf you encounter the following error on an Apple silicon Macpygraphviz/graphviz_wrap.c:2711:10:fatalerror:'graphviz/cgraph.h'filenotfound#include "graphviz/cgraph.h"^~~~~~~~~~~~~~~~~~~1errorgenerated.you need to run the following command once to update the pygraphviz include directories for the new homebrew environmentpipinstall--global-option=build_ext--global-option="-I$(brew--prefixgraphviz)/include/"--global-option="-L$(brew--prefixgraphviz)/lib/"pygraphvizSee the issue in contexthere.2️⃣.2️⃣ ~ (macOS) Create a virtual environmentCheck of you have the latest Python 3 installed on your macOS. I recommend installing/using Python 3 fromHomebrew. Create a Python 3 virtual environment (optionally within the project structure)cd emerge
pip3 install virtualenv
virtualenv -p python3 venv2️⃣ ~ (ubuntu) Create a virtual environmentInstall required packages and create a Python 3 virtual environment (optionally within the project structure)apt-get install python3-venv python3-dev graphviz graphviz-dev
cd emerge
python3 -m venv venv3️⃣ ~ Before using/working with the tool, activate the virtual environmentsource venv/bin/activate4️⃣ ~ (macOS) Install all dependenciesInstall all required dependencies for the project with pippip install -r requirements.txt4️⃣ ~ (ubuntu) Install all dependenciesInstall the wheel package, after that install all required dependencies for the project with pippip install wheel
pip install -r requirements.txt5️⃣ ~ Running unit tests from the command lineExecute the following from the cloned project root:python -m unittest discover -v -s ./emerge -p "test_*.py"otherwise execute the scriptrun_tests.py:python run_tests.pyIf you got in any trouble executing the tests, checkthis woraround.6️⃣ ~ Runningemergeas a standalone tool(emerge) user@host emerge % python emerge.py 
usage: emerge.py [-h] [-c YAMLCONFIG] [-v] [-d] [-e] [-a LANGUAGE]

🔎 Welcome to emerge x.y.z (yyyy-mm-dd hh:mm:ss)

options:
  -h, --help            show this help message and exit
  -c YAMLCONFIG, --config YAMLCONFIG
                        set yaml config file
  -v, --verbose         set logging level to INFO
  -d, --debug           set logging level to DEBUG
  -e, --error           set logging level to ERROR
  -a LANGUAGE, --add-config LANGUAGE
                        add a new config from a template, where LANGUAGE is one of [JAVA, SWIFT, C, CPP, GROOVY, JAVASCRIPT,
                        TYPESCRIPT, KOTLIN, OBJC, RUBY, PY, GO]7️⃣ ~ You're ready to goLet's quickly try to run emerge on its own codebasepython emerge.py -c configs/emerge.yamlThis should produce a similar output:...   analysis I 👉 starting to analyze emerge
...   analysis I ⏩ performing analysis 1/1: self-check
...   analysis I 👉 starting to create filesystem graph in self-check
...   analysis I ⏩ starting scan at directory: .
...   ...
...   analysis I 👉 the following statistics were collected in self-check
+-------------------------------------+-------------------+
|                      statistic name | value             |
+-------------------------------------+-------------------+
|                    scanning_runtime | 00:00:00 + 61 ms  |
|                       scanned_files | 32                |
|                       skipped_files | 176               |
|                        parsing_hits | 313               |
|                      parsing_misses | 141               |
|              extracted_file_results | 32                |
|       file_results_creation_runtime | 00:00:00 + 538 ms |
|    number-of-methods-metric-runtime | 00:00:00 + 4 ms   |
| source-lines-of-code-metric-runtime | 00:00:00 + 11 ms  |
|   louvain-modularity-metric-runtime | 00:00:00 + 161 ms |
|           fan-in-out-metric-runtime | 00:00:00 + 4 ms   |
|                       total_runtime | 00:00:00 + 786 ms |
+-------------------------------------+-------------------+
...   analysis I 👉 the following overall metrics were collected in self-check
+----------------------------------------------+----------------------------+
|                                  metric name | value                      |
+----------------------------------------------+----------------------------+
|                avg-number-of-methods-in-file | 13.0                       |
|                             avg-sloc-in-file | 151.41                     |
|                          total-sloc-in-files | 4845                       |
|         louvain-communities-dependency-graph | 3                          |
|          louvain-modularity-dependency-graph | 0.21                       |
| louvain-biggest-communities-dependency-graph | 0.49, 0.46, 0.05, 0.0, 0.0 |
|                  avg-fan-in-dependency-graph | 5.55                       |
|                 avg-fan-out-dependency-graph | 5.55                       |
|                  max-fan-in-dependency-graph | 29                         |
|             max-fan-in-name-dependency-graph | typing                     |
|                 max-fan-out-dependency-graph | 19                         |
|            max-fan-out-name-dependency-graph | emerge/appear.py           |
+----------------------------------------------+----------------------------+
...   analysis I ✅ all your generated/exported data can be found here: /Users/user1/tmp/python
...   analysis I ✅ copy the following path to your browser and start your web app: 👉 file:///Users/user1/tmp/python/html/emerge.html
...   analysis I ✅ total runtime of analysis: 00:00:00 + 786 ms8️⃣ ~ Start your web appNow just copy the above mentionedfile://path to any modern web browser and interactively expore the emerge codebase 😉8️⃣.1️⃣ ~ Currently emerge offers the following keyboard shortcuts in the interactive web appℹ️ Hovering over a node and pressing ⬆️ +sto select and highlight or deselect a specific nodeℹ️ Resetting the currently active node selection by pressing ⬆️ +rℹ️ Fading all unselected nodes to have a more highlighted visualization of you currenty selected nodes by pressing ⬆️ +fAnd now let's make this more interesting ...Further configuration (using emerge on other projects)If you wand to use emerge on other projects, you can simple copy or customize one of the existing configuration templates from theemerge/configsdirectory.9️⃣ ~ Scan a real projectFor a quick run, it should be enough to adjustsource_directory,directoryinexport.---project_name:c-example-projectloglevel:infoanalyses:-analysis_name:check_c_filessource_directory:/Users/user1/emerge/project/source/github/linux-5.8.5/cryptoonly_permit_languages:-conly_permit_file_extensions:-.c-.hignore_dependencies_containing:-string.hignore_dependencies_matching:-^test_(.*)\.h$file_scan:-number_of_methods-source_lines_of_code-dependency_graph-louvain_modularity-fan_in_out-tfidfexport:-directory:/Users/user1/emerge/project/export-graphml-json-tabular_file-tabular_console_overall-d31️⃣0️⃣ ~ Run emerge with a specific yaml configurationAfter customizing a present config (e.g.config/c-template.yaml) or creating your own, just run emerge again with this new configpython emerge.py -c configs/c-template.yamlAfter the scan, your scan output (including your interactive web app) can be found at the directory that you created and set in the config parameterexport->directory, as seen in the logs above.A full YAML configuration that contains both file and entity scan has the following format:---project_name:java_project_exampleloglevel:infoanalyses:-analysis_name:check_java_files_and_classessource_directory:/Users/user1/emerge/project/sourceonly_permit_languages:-javaonly_permit_file_extensions:-.javaignore_dependencies_containing:-java.utilfile_scan:-number_of_methods-source_lines_of_code-dependency_graph-fan_in_out-louvain_modularity-tfidfentity_scan:-dependency_graph-source_lines_of_code-number_of_methods-fan_in_out-louvain_modularity-tfidfexport:-directory:/Users/user1/emerge/project/export-graphml-json-tabular_file-tabular_console_overall-d3Sometimes it can make sense to exclude platform-usual dependencies or dependencies which do not contribute much to the understanding of a project. A good starting point for e.g. anAndroid projectcould the followingignore_dependencies_containingsection:ignore_dependencies_containing:-android-java-javaxor for aniOS projectthe followingignore_entities_containingsection often makes sense e.g. to not consider SwiftUI previews for the graph output:ignore_entities_containing:-_PreviewsThe yaml configuration is basically defined at the following levels:project levelkeyvalue/ descriptionproject_namea project name for all analyses, scans and exportsloglevelset a loglevel:error(silent, only errors),info(includeserror) gives you basic logs about control flow,debug(includesinfo) will produce a lot of debug logsanalysesan array of analyses that can be configured individually, thus a project can contain one to many analyses.analysis levelkeyvalue/ descriptionanalysis_namea specific analysis namesource_directorythe source directory where the recursive file scan should startgit_directorythe git repo directory, if git metrics should be includedgit_commit_limithow many commits from the last commit should be mined? default:150git_exclude_merge_commitsshould merge commits be excluded from mining all metrics? default:trueignore_files_containingexclude file names from the scan that contain the given substringsignore_directories_containingexclude directory names from the scan that contain the given substringsonly_permit_languagespossible values include: java, kotlin, objc, swift, ruby, groovy, javascript, c - explicitly prevents any other language from scanning besides the one you set hereonly_permit_file_extensionsexplicitly permit the following file extensions you set here, e.g..javaonly_permit_files_matching_absolute_pathonly the following list of absolute file paths is permitted for the file scan, e.g.[/Users/user1/source/file1.java]. The files should followsource_directoryignore_dependencies_containingignore every dependency included in this list of substrings, e.g.java.utilignore_dependencies_matchingignore every dependency matching any of the regular expressions in this list of substrings, e.g.^java\.util\.ignore_entities_containingignore every entity included in this list of substrings, e.g.NotRelevantClassignore_entities_matchingignore every entity matching any of the regular expressions in this list of substrings, e.g.^Testimport_aliasesdefine a list of import aliases, i.e. replace substrings within a full dependency path, e.g."@foo": src/foowill replace any@fooalias bysrc/foooverride_resolve_dependenciesif supported by the language parser, force every dependency in this list to be resolvedoverride_do_not_resolve_dependenciesif supported by the language parser, force every dependency in this list NOT to be resolved (i.e. treated as a global dependency)file_scanperform a file scan, contains the metrics that should be applied on every source fileentity_scanperform an entity scan, contains the metrics that should be applied on every entity (e.g. on every class)exportcontains any export formats that should be created as outputappconfigcontains any configurable app config parametersfile_scan metricskeyvalue/ descriptiondependency_graphcreate a dependency graph structure based on source files, additional metrics will be added to the graph nodessource_lines_of_codeapply a source lines of code metric to every file, create an overall metricnumber_of_methodsapply a number of methods metric to every file, create an overall metricfan_in_outapply a fan in/ fan out graph metric to every file, create an overall metriclouvain_modularityapply a louvain modularity metric to every file, create an overall metrictfidfapply a tfidf metric to every file and extract relevant semantic keywordsws_complexityapply a whitespace complexity metric to every filegit_metricsinclude some git-based metrics and try to apply them to every fileentity_scan metricskeyvalue/ descriptiondependency_graphcreate a dependency graph structure based on extracted entities from files, additional metrics will be added to the graph nodesinheritance_graphcreate an inheritance graph structure based on extracted entities from files, additional metrics will be added to the graph nodescomplete_graphcreate a complete graph structure (union of dependency/ inheritance graph) based on extracted entities from files, additional metrics will be added to the graph nodessource_lines_of_codeapply a source lines of code metric to every entity, create an overall metricnumber_of_methodsapply a number of methods metric to every entity, create an overall metricfan_in_outapply a fan in/ fan out graph metric to every entity, create an overall metriclouvain_modularityapply a louvain modularity metric to every entity, create an overall metrictfidfapply a tfidf metric to every entity and extract relevant semantic keywordsexport configurationkeyvalue/ descriptiondirectorythe output directory for all specified export formatsgraphmlcreate a graphML file that contains the graph structure and metric results mapped to the nodes of the graphtabular_filecreate a tabular formatted text file that contains every metric and statistic resulttabular_consoleprint a tabular formatted output to console that contains every metric and statistic resulttabular_console_overallprint a tabular formatted output to console that contains only overall metric and statistic resultsjsoncreate a JSON file that contains every metric and statistic resultd3create a Bootstrap/D3 web application in the subfolderforce-graph-htmlfor further visual and interactive/ exploratory analysisappconfigkeyvalue/ descriptionradius_fan_outnode radius multiplication factor for the fan-out metric, default:0.1radius_fan_innode radius multiplication factor for the fan-in metric, default:0.1radius_louvainnode radius multiplication factor for the louvain metric, default:0.02radius_slocnode radius multiplication factor for the sloc metric, default:0.005radius_number_of_methodsnode radius multiplication factor for the number of methods metric, default:0.05heatmap_sloc_activeshould the sloc metric be included in the heatmap score calculation? default:trueheatmap_fan_out_activeshould the fan-out metric be included in the heatmap score calculation? default:trueheatmap_sloc_weightweight factor of the sloc metric within the heatmap score calculation, default:1.5heatmap_fan_out_weightweight factor of the fan-out metric within the heatmap score calculation, default:1.7heatmap_score_basemin score threshold for the heatmap color mapping, default:10heatmap_score_limitmax score threshold for the heatmap color mapping, default:300Supported scan types and file extensionsEmerge supports the following file extensions and scan types per language, whereas afile_scansimply calculates metrics and maps nodes within graph structures to scanned files and anentity_scantries to extract more fine-grained entities from files e.g. classes or structs.File extensionLanguage parserFilesEntities.javaJava✅✅.swiftSwift✅✅.c/.hC✅❌.cpp/.hC++✅❌.groovyGroovy✅✅.js/.jsxJavaScript✅❌.ts/.tsxTypeScript✅❌.kKotlin✅✅.m/.hObjective-C✅❌.rbRuby✅❌.pyPython✅❌.goGo✅❌Interpretation of graphsThe interpretation of such graphs can often be very subjective and project dependent. The following examples should help to recognize certain patterns through indicators and hints.⭐️ ModularityThe magic of uncovering modularity lies in applying a community detection algorithm e.g. Louvain optimization to a force-directed graph, so that both distances and coloring influence the result. The following example includes several indicators for a modular codebase.In the first example on the left you can spot multiple coherent colored clusters which show a low coupling by a certain distance (= generated by the force-directed graph).In the second example on the right the same graph is rendered with activated cluster hulls. On this example the hulls show minimal to no overlapping. Such hints can be indicators for a good software architecture e.g. in terms ofmodularity,abstractionand well definedinterfaces.⭐️ Codebases with different characteristicsIn the following example on the left you can see structures with increased modularity e.g. cluster 1 and 2.At the same time you can see another cluster 3 that shows increased overlappings with other clusters. In addition to that an active SLOC metric even shows some huge entities (e.g. classes) which could indicate code smells likegod classes. Such code smells like increased coupling and god classes may indicate increased maintenace efforts and error-proneness.⭐️ Big Ball of Mud"A BIG BALL OF MUD is haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle" (B. Foote, J. Yoder, 1997). This kind of graph often represents a less optimal architecture. To verify this kind ofspaghetti code jungle, one can simply enable hull rendering for all clusters to finally determine: there is only one big cluster after all.⭐️ Abstract irrelevant dependenciesSometimes it can help to better understand the complexity of a software architecture if irrelevant dependencies are ignored.Besides being shocked to see the Big Ball of Mud with irrelevant dependencies like java.lang, java.util or any third party dependencies that does not directly belong to a project ...... you can remove irrelevant dependencies configuratively with the keyignore_dependencies_containing(orignore_dependencies_matchingif you prefer regular expressions). With a comparatively activated fan-out metric, one recognizes more scattering, some distant hub nodes and clearer clusters. All of these are possible clues to the real (= often more understandable) architecture underneath.Further developmentParsing of further entity types for more languages is planned for further development.Contributionsare very welcome ❤️Everyone is invited to contribute to this project, whether the contribution is related with development, testing, bug reporting or any other support. I would appreciate any help. SeeContributingandCreditsfor further details.