Package: minerva-cargo
Codenet Minerva CargoCargo is part of the Minerva project working on refactoring monoliths to microservices. It leveragesData Gravity Insightsfrom the Konveyor.io project and provides recommendations for partitioning code taking into account code relationships, data relationships, and database transaction scope.CARGO: AI-Guided Dependency Analysis for Migrating Monolithic Applications to Microservices ArchitecturePaper:ArXiV PreprintAbstractCARGO (short for Context-sensitive lAbel pRopaGatiOn) is a novel un-/semi-supervised partition refinement technique that uses a comprehensive system dependence graph built using context and flow-sensitive static analysis of a monolithic application to refine and thereby enrich the partitioning quality of the current state-of-the-art algorithms.Figure 1. Overview of CARGOKick-the-tires Instructions (~15 minutes)The instructions will reproduce the key results in Figure 6 (RQ1), Figure 7 (RQ2), and Table 1 (RQ3).Pre-requisitesA Linux/Mac system withDocker.Python>= 3.8, and Pip. Tested with Python 3.9.Step 0: Clone this repositoryWe'll clone this repository and save it's location for the next stepsgitclonehttps://github.com/IBM/codenet-minerva-cargo&&cdcodenet-minerva-cargoexportREPO_ROOT=$PWDStep 1: Set up Data Gravity Insights CLIWe will useData Gravity Insights(aka. DGI) to first build a system dependency graph and persist the graph in a Neo4j.1.1 Install DGIDGI is available as PyPi package, you can also installdgias followspipinstall-Ugit+https://github.com/rahlk/tackle-data-gravity-insightsThis will install the dgi command locally under your home folder in a hidden folder called: ~/.local/bin. If not already, you must add this folder to your PATH with:exportPATH=$HOME/.local/bin:$PATH1.2 Creating a Neo4j Docker containerMake sure that your Docker daemon is running, either by starting up the service (on linux) or by opening the desktop application (on mac).We will need an instance of Neo4j to store the graphs thatdgicreates. We will start one up in a docker container and set an environment variable to letdgiknow where to find it.dockerrun-d--nameneo4j\-p7474:7474\-p7687:7687\-eNEO4J_AUTH="neo4j/konveyor"\-eNEO4J_apoc_export_file_enabled=true\-eNEO4J_apoc_import_file_enabled=true\-eNEO4J_apoc_import_file_use__neo4j__config=true\-eNEO4JLABS_PLUGINS=\["apoc"\]\neo4j:4.4.17exportNEO4J_BOLT_URL="neo4j://neo4j:konveyor@localhost:7687"Installation completeWe can now use thedgicommand to load information about an application into a graph database. We start withdgi --help. This should produce:$ dgi --help

 Usage: dgi [OPTIONS] COMMAND [ARGS]...

 Tackle Data Gravity Insights

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --neo4j-bolt  -n  TEXT  Neo4j Bolt URL                                                                          │
│ --quiet       -q        Be more quiet                                                                           │
│ --validate    -v        Validate but don't populate graph                                                       │
│ --clear       -c        Clear graph before loading                                                              │
│ --help                  Show this message and exit.                                                             │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ c2g            Code2Graph add various program dependencies (i.e., call return, heap, and data) into the graph   │
│ partition      Partition is a command runs the CARGO algorithm to (re-)partition a monolith into microservices  │
│ s2g            Schema2Graph parses SQL schema (*.DDL file) into the graph                                       │
│ tx2g           Transaction2Graph add edges denoting CRUD operations to the graph.                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯Step 2: Setting up a sample applicationFor rest of this walkthrough, we'll work withDayTrader8.Step 3: Build a Program Dependency Graph with DOOP3.1 Prepare the applicationObtain the sample application WAR file. We'll save this inextras/demo/doop-in:wgethttps://github.com/OpenLiberty/sample.daytrader8/releases/download/v1.2/io.openliberty.sample.daytrader8.war--directory-prefix=$REPO_ROOT/extras/demo/doop-in3.2 Getting facts with DOOPWe first need to runDOOP. For ease of use, DOOP has been pre-compiled and hosted as a docker image atquay.io/rkrsn/doop-main. We'll use that for this demo.dockerrun-it--rm\-v$REPO_ROOT/extras/demo/doop-in:/root/doop-data/input\-v$REPO_ROOT/extras/demo/doop-out:/root/doop-data/output/\quay.io/rkrsn/doop-main:latestrundoopNotes:1. If you encounter any error above, please rerun thedocker run ...command2. Running DOOP for the first time may take up to 15 minutes.3.3 Run DGI code2graphIn this step, we'll run DGI code2graph to populate a Neo4j graph database with various static code interaction features pertaining to object/dataflow dependencies.dgi -c c2g -a class -i $REPO_ROOT/extras/demo/doop-outThis will take 4-5 minutes. After successful completion, we should see something like this :❯dgi-cc2g-aclass-i$REPO_ROOT/extras/demo/doop-out[15:57:56]INFOcode2graphgeneratorstarted.INFOVerbosemode:ONINFOBuildingGraph.INFOClasslevelabstraction.WARNINGTheoptionclearisturnedON.Deletingpre-existingnodes.INFOPopulatingheapcarrieddependenciesedges•100%━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•Completed/Total:1192/1192•Elapsed:0:00:02•Remaining:0:00:00[15:57:58]INFOPopulatingdataflowedges•100%━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•Completed/Total:991/991•Elapsed:0:00:01•Remaining:0:00:00[15:58:00]INFOPopulatingcall-returndependenciesedges•100%━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•Completed/Total:2404/2404•Elapsed:0:00:04•Remaining:0:00:00[15:58:04]INFOPopulatingentrypointsINFOcode2graphbuildcomplete3.4 Extracting Database Transactions with Tackle-DiVANote that this step is only for applications with database transactions. We will runTackle-DiVAto extract transactions from our application. DiVA is available as a docker image, so we just need to run DiVA by pointing to thesource code directoryof the application and the desired output directory.Let's first get the source code for DayTrader8:wget-chttps://github.com/OpenLiberty/sample.daytrader8/archive/refs/tags/v1.2.tar.gz-O-|tar-xvz-C$REPO_ROOT/extras/demodockerrun--rm\-v$REPO_ROOT/extras/demo/sample.daytrader8-1.2:/app\-v$REPO_ROOT/extras/demo/txns:/diva-distribution/output\quay.io/konveyor/tackle-divaThis should generate a filetransaction.jsoncontaining all discovered transactions. Finally, we run DGI to load these transaction edges into the program dependency graph.dgi-ctx2g-aclass-i$REPO_ROOT/extras/demo/txns/transaction.jsonAfter successful completion, we should see something like this :❯dgi-ctx2g-aclass-i$REPO_ROOT/extras/demo/txns/transaction.json[16:05:36]INFOVerbosemode:ONWARNINGTheCLIargumentclearisturnedON.Deletingpre-existingnodes.INFOClassTransactionLoader:Populatingtransactions•100%━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•Completed/Total:175/175•Elapsed:0:00:01•Remaining:0:00:00[16:05:38]INFOTransactionspopulatedStep 4: Running CARGOOnce we have created the Neo4j graphs by following the above steps, we can run CARGO as follows:dgi partition --partitions=5