Repository: IN-CORE/pyincore (language: python)

Issue:
convert hua and pd outputs to json
Lisa's notebook does not run pyincore analysis. There is a csv file with retrofits Wangoutput_2020-05-28_sheet1.csv which is merged with IN-CORE_2ev3_SetupJoplin_FourInventories_2019-08-06_HUinventory.csv. There are two lines named Population dislocation which is compared to HUA dislocation. That might be the only code to be run in pyIncore.


Discussion / hints:
Hi Diego - I am responsible for the IN-CORE_2ev3_SetupJoplin_FourInventories_2019-08-06_HUinventory.csv files.

I am not familiar with the specific notebook from Lisa. Is it the same one that is online? https://incore.ncsa.illinois.edu/doc/incore/notebooks/Joplin_testbed/Joplin_testbed.html#6.-Sociological-Analysis
The published Joplin Notebook can be updated with the notebook on my Github Page https://github.com/npr99/IN-CORE_notebooks/blob/main/IN_CORE_CommunityDescription_Joplin.ipynb 

My notebook uses the same building inventory as Lisa's but adds the housing unit inventory and the housing unit allocation method to produce the population data
Is this task for the playbook? if so i think @ncsa-mo has already figured out how to generate the json. Could you confirm?
I just moved this from a task in bitbucket. @ncsa-mo can you please look at the questions above?
This task is for the Playbook. It is assigned to me, we have everything to generate a json outputs.
@ncsa-mo  here is the additional json (Github doesn't allow me to attach json so i turn it to txt, but it should be a json)
[HUA_by_housing_type.txt](https://github.com/IN-CORE/pyincore/files/7211794/HUA_by_housing_type.txt)

There might be one more coming in the future "by_disability", but I haven't got instruction on how to generate this table

Fix patch:
diff --git a/docs/source/modules.rst b/docs/source/modules.rst
index 281ae073d..dc3f3e4b9 100644
--- a/docs/source/modules.rst
+++ b/docs/source/modules.rst
@@ -59,7 +59,7 @@ analyses/example
     :members:
 
 analyses/housingrecoverysequential
-==============================
+==================================
 ..  autoclass:: housingrecoverysequential.housingrecoverysequential.HousingRecoverySequential
     :members:
 
@@ -133,7 +133,7 @@ analyses/populationdislocation
     :members:
 
 analyses/residentialbuildingrecovery
-============================
+====================================
 ..  autoclass:: residentialbuildingrecovery.residentialbuildingrecovery.ResidentialBuildingRecovery
     :members:
 
@@ -259,6 +259,11 @@ utils/analysisutil
 ..  autoclass:: utils.analysisutil.AnalysisUtil
     :members:
 
+utils/cgeoutputprocess
+======================
+..  autoclass:: utils.cgeoutputprocess.CGEOutputProcess
+    :members:
+
 utils/dataprocessutil
 =====================
 ..  autoclass:: utils.dataprocessutil.DataProcessUtil
@@ -279,6 +284,11 @@ utils/geoutil
 ..  autoclass:: utils.geoutil.GeoUtil
     :members:
 
+utils/popdisloutputprocess.py
+=============================
+..  autoclass:: utils.popdisloutputprocess.PopDislOutputProcess
+    :members:
+
 utils/networkutil
 =================
 ..  autoclass:: utils.networkutil.NetworkUtil
diff --git a/pyincore/__init__.py b/pyincore/__init__.py
index 91db49728..d4303cf68 100644
--- a/pyincore/__init__.py
+++ b/pyincore/__init__.py
@@ -20,7 +20,7 @@
 from pyincore.restorationservice import RestorationService
 from pyincore.spaceservice import SpaceService
 from pyincore.utils.analysisutil import AnalysisUtil
-from pyincore.utils.popdisloutputprocessutil import PopdislOutputProcessUtil
+from pyincore.utils.popdisloutputprocess import PopDislOutputProcess
 from pyincore.utils.cgeoutputprocess import CGEOutputProcess
 from pyincore.dataset import Dataset, InventoryDataset, DamageRatioDataset
 from pyincore.models.fragilitycurveset import FragilityCurveSet
diff --git a/pyincore/utils/popdisloutputprocess.py b/pyincore/utils/popdisloutputprocess.py
new file mode 100644
index 000000000..0d92f7ab3
--- /dev/null
+++ b/pyincore/utils/popdisloutputprocess.py
@@ -0,0 +1,523 @@
+# Copyright (c) 2021 University of Illinois and others. All rights reserved.
+#
+# This program and the accompanying materials are made available under the
+# terms of the Mozilla Public License v2.0 which accompanies this distribution,
+# and is available at https://www.mozilla.org/en-US/MPL/2.0/
+
+import json
+import pandas as pd
+import geopandas as gpd
+from shapely import wkt
+
+
+class PopDislOutputProcess:
+    """This class converts csv results outputs of Population dislocation analysis to json format and shapefiles.
+
+    Args:
+        pop_disl_result (obj): IN-CORE dataset for Joplin Population Dislocation (PD) results.
+        pop_disl_result_path (obj): A fallback for the case that Joplin PD object is not provided.
+            For example a user wants to directly pass in csv files, a path to PD results.
+        filter_on (bool): A flag to filter all data, default True counts only Joplin buildings
+
+    """
+    HUPD_CATEGORIES = ["household_characteristics",
+                       "household_dislocated",
+                       "total_households",
+                       "%_household_dislocated",
+                       "population_dislocated",
+                       "total_population",
+                       "%_population_dislocated"
+                       ]
+
+    def __init__(self, pop_disl_result, pop_disl_result_path=None, filter_on=True):
+        if pop_disl_result_path:
+            pd_result = pd.read_csv(pop_disl_result_path, low_memory=False)
+        else:
+            pd_result = pop_disl_result.get_dataframe_from_csv(low_memory=False)
+        pd_result["geometry"] = pd_result["geometry"].apply(wkt.loads)
+
+        pd_result_shp = None
+        # keep only inventory with guid; filter for Joplin since only Joplin inventory has guids
+        if filter_on:
+            pd_result = pd_result[(pd_result["guid"].notnull()) &
+                                  (pd_result["numprec"].notnull())]
+            # only keep guid and dislocated
+            pd_result_shp = pd_result[(pd_result["dislocated"].notnull()) &
+                                      (pd_result["guid"].notnull()) &
+                                      (pd_result["numprec"].notnull())]
+        self.pop_disl_result = pd_result
+        self.pop_disl_result_shp = pd_result_shp
+
+    def get_heatmap_shp(self, filename="pop-disl-numprec.shp"):
+        """ Convert and filter population dislocation output to shapefile that contains only guid and numprec columns
+
+            Args:
+                filename (str): Path and name to save shapefile output file in. E.g "heatmap.shp"
+
+            Returns:
+                str: full path and filename of the shapefile
+
+        """
+        df = self.pop_disl_result_shp
+
+        # save as shapefile
+        gdf = gpd.GeoDataFrame(df, crs='epsg:4326')
+        gdf = gdf[["guid", "numprec", "geometry"]]
+        gdf.to_file(filename)
+
+        return filename
+
+    def pd_by_race(self, filename_json=None):
+        """ Calculate race results from the output files of the Joplin Population Dislocation analysis
+        and convert the results to json format.
+        [
+            {"household_characteristics": "Not Hispanic/White",
+             "household_dislocated": 1521,
+             "total_households": 18507,
+             "%_household_dislocated": 7.3,
+             "population_dislocated",
+             "total_population",
+             "%_population_dislocated"
+            },{"household_characteristics": "Not Hispanic/Black",..,..},{},{},
+            {"No race Ethnicity Data"},{"Total"}
+        ]
+
+        Args:
+            filename_json (str): Path and name to save json output file in. E.g "pd_race_count.json"
+
+        Returns:
+            obj: PD total count by race. A JSON of the hua and population dislocation race results by category.
+
+        """
+        # Race categories
+        # The numbering follows the Community description notebook
+        # 0 - Vacant HU No Race Ethnicity Data, 1 - Not Hispanic/White, 2 - Not Hispanic/Black
+        # 3 - Not Hispanic/Other race, 4 - Hispanic, 5 - No Race or Ethnicity Data
+        race_categories = ["Not Hispanic/White",
+                           "Not Hispanic/Black",
+                           "Not Hispanic/Other Race",
+                           "Hispanic",
+                           "No Race or Ethnicity Data",
+                           "Total"]
+
+        huapd = self.pop_disl_result
+        # Allocated by race and ethnicity
+        huapd["hua_re"] = "0"
+        huapd.loc[(huapd["race"] == 1) & (huapd["hispan"] == 0), "hua_re"] = "1"
+        huapd.loc[(huapd["race"] == 2) & (huapd["hispan"] == 0), "hua_re"] = "2"
+        huapd.loc[(huapd["race"].isin([3, 4, 5, 6, 7])) & (huapd["hispan"] == 0), "hua_re"] = "3"
+        huapd.loc[(huapd["hispan"] == 1), "hua_re"] = "4"
+        huapd.loc[(huapd["gqtype"] >= 1), "hua_re"] = "5"
+        hua_vals = huapd["hua_re"].value_counts()
+        hua_tot = []
+        for i in range(len(race_categories)):
+            try:
+                hua_tot.append(int(hua_vals[str(i)]))
+            except Exception:
+                hua_tot.append(0)
+        hua_tot.append(int(sum(hua_tot[1:])))
+
+        pop_tot = []
+        for i in range(len(race_categories)):
+            pop_tot.append(int(huapd["numprec"].where(huapd["hua_re"] == str(i)).sum()))
+        pop_tot.append(int(sum(pop_tot[1:])))
+
+        # Dislocated by race and ethnicity
+        huapd["hud_re"] = "0"
+        huapd.loc[(huapd["race"] == 1) & (huapd["hispan"] == 0) & huapd["dislocated"], "hud_re"] = "1"
+        huapd.loc[(huapd["race"] == 2) & (huapd["hispan"] == 0) & huapd["dislocated"], "hud_re"] = "2"
+        huapd.loc[(huapd["race"].isin([3, 4, 5, 6, 7])) & (huapd["hispan"] == 0) & huapd["dislocated"], "hud_re"] = "3"
+        huapd.loc[(huapd["hispan"] == 1) & huapd["dislocated"], "hud_re"] = "4"
+        huapd.loc[(huapd["gqtype"] >= 1) & huapd["dislocated"], "hud_re"] = "5"
+        hud_vals = huapd["hud_re"].value_counts()
+        hua_disl = []
+        for i in range(len(race_categories)):
+            try:
+                hua_disl.append(int(hud_vals[str(i)]))
+            except Exception:
+                hua_disl.append(0)
+        hua_disl.append(int(sum(hua_disl[1:])))
+
+        pd_disl = []
+        for i in range(len(race_categories)):
+            pd_disl.append(int(huapd["numprec"].where(huapd["hud_re"] == str(i)).sum()))
+        pd_disl.append(int(sum(pd_disl[1:])))
+
+        pd_by_race_json = []
+        for i in range(len(race_categories)):
+            huapd_race = {}
+            huapd_race[self.HUPD_CATEGORIES[0]] = race_categories[i]
+            huapd_race[self.HUPD_CATEGORIES[1]] = hua_disl[i + 1]
+            huapd_race[self.HUPD_CATEGORIES[2]] = hua_tot[i + 1]
+            if hua_tot[i + 1]:
+                huapd_race[self.HUPD_CATEGORIES[3]] = 100 * (hua_disl[i + 1] / hua_tot[i + 1])
+            else:
+                huapd_race[self.HUPD_CATEGORIES[3]] = None
+            huapd_race[self.HUPD_CATEGORIES[4]] = pd_disl[i + 1]
+            huapd_race[self.HUPD_CATEGORIES[5]] = pop_tot[i + 1]
+            if pop_tot[i + 1]:
+                huapd_race[self.HUPD_CATEGORIES[6]] = 100 * (pd_disl[i + 1] / pop_tot[i + 1])
+            else:
+                huapd_race[self.HUPD_CATEGORIES[6]] = None
+            pd_by_race_json.append(huapd_race)
+        # print(pd_by_race_json)
+
+        if filename_json:
+            with open(filename_json, "w") as outfile:
+                json.dump(pd_by_race_json, outfile)
+        # Serializing json
+        return json.dumps(pd_by_race_json)
+
+    def pd_by_income(self, filename_json=None):
+        """ Calculate income results from the output files of the Joplin Population Dislocation analysis
+        and convert the results to json format.
+        [
+            {"household_characteristics": "HH1 (less than $15,000)",
+             "household_dislocated": 311,
+             "total_households": 3252,
+             "%_household_dislocated": 7.3,
+             "population_dislocated": 311,
+             "total_population": 3252,
+             "%_population_dislocated"
+             },
+             {"HH2 ($15,000 to $35,000)",..,..,..,..},{},{},{},{},
+             {"Unknown",..,..,..,..}
+        ]
+
+        Args:
+            filename_json (str): Path and name to save json output file in. E.g "pd_income_count.json"
+
+        Returns:
+            obj: PD total count by income. A JSON of the hua and population dislocation income results by category.
+
+        """
+        income_categories = ["HH1 (less than $15,000)",
+                             "HH2 ($15,000 to $35,000)",
+                             "HH3 ($35,000 to $70,000)",
+                             "HH4 ($70,000 to $120,000)",
+                             "HH5 (More than $120,000)",
+                             "Unknown",
+                             "Total"]
+
+        huapd = self.pop_disl_result
+        # Allocated by income
+        hua_tot = []
+        for i in range(1, 6):
+            alloc_inc = (huapd["hhinc"] == i).sum()
+            hua_tot.append(int(alloc_inc))
+        hua_tot.append(int(pd.isna(huapd["hhinc"]).sum()))
+        hua_tot.append(int(sum(hua_tot)))
+
+        pop_tot = []
+        for i in range(1, 6):
+            alloc = huapd["numprec"].where(huapd["hhinc"] == i).sum()
+            pop_tot.append(int(alloc))
+        pop_tot.append(int(huapd["numprec"].where(pd.isna(huapd["hhinc"])).sum()))
+        pop_tot.append(int(sum(pop_tot)))
+
+        # Dislocated by income
+        hua_disl = []
+        for i in range(1, 6):
+            disl = huapd.loc[(huapd["hhinc"] == i) & huapd["dislocated"], ["dislocated"]].sum()
+            hua_disl.append(int(disl))
+        disl_unknown = huapd.loc[pd.isna(huapd["hhinc"]) & huapd["dislocated"], ["dislocated"]].sum()
+        hua_disl.append(int(disl_unknown))
+        hua_disl.append(int(sum(hua_disl)))
+
+        pd_disl = []
+        for i in range(1, 6):
+            disl = huapd.loc[(huapd["hhinc"] == i) & huapd["dislocated"], ["numprec"]].sum()
+            pd_disl.append(int(disl))
+        disl_unknown = huapd.loc[pd.isna(huapd["hhinc"]) & huapd["dislocated"], ["numprec"]].sum()
+        pd_disl.append(int(disl_unknown))
+        pd_disl.append(int(sum(pd_disl)))
+
+        pd_by_income_json = []
+        for i in range(len(income_categories)):
+            huapd_income = {}
+            huapd_income[self.HUPD_CATEGORIES[0]] = income_categories[i]
+            huapd_income[self.HUPD_CATEGORIES[1]] = hua_disl[i]
+            huapd_income[self.HUPD_CATEGORIES[2]] = hua_tot[i]
+            if hua_tot[i]:
+                huapd_income[self.HUPD_CATEGORIES[3]] = 100 * (hua_disl[i] / hua_tot[i])
+            else:
+                huapd_income[self.HUPD_CATEGORIES[3]] = None
+            huapd_income[self.HUPD_CATEGORIES[4]] = pd_disl[i]
+            huapd_income[self.HUPD_CATEGORIES[5]] = pop_tot[i]
+            if pop_tot[i]:
+                huapd_income[self.HUPD_CATEGORIES[6]] = 100 * (pd_disl[i] / pop_tot[i])
+            else:
+                huapd_income[self.HUPD_CATEGORIES[6]] = None
+            pd_by_income_json.append(huapd_income)
+        # print(pd_by_income_json)
+
+        if filename_json:
+            with open(filename_json, "w") as outfile:
+                json.dump(pd_by_income_json, outfile)
+        # Serializing json
+        return json.dumps(pd_by_income_json)
+
+    def pd_by_tenure(self, filename_json=None):
+        """ Calculate tenure results from the output files of the Joplin Population Dislocation analysis
+        and convert the results to json format.
+        [
+            {"household_characteristics": "Owner occupied",
+             "household_dislocated": 1018,
+             "total_households": 11344,
+             "%_household_dislocated": 7.3,
+             "population_dislocated": 1018,
+             "total_population": 11344,
+             "%_population_dislocated"
+            },
+            {"household_characteristics": "Renter occupied",..,..,..,..},{},{},{},{},{},
+            {"total",..,..,..,..}
+        ]
+
+        Args:
+            filename_json (str): Path and name to save json output file in. E.g "pd_income_count.json"
+
+        Returns:
+            obj: PD total count by income. A JSON of the hua and population dislocation income results by category.
+
+        """
+        # Tenure categories
+        # The numbering follows the Community description notebook
+        # 0 - Vacant HU No Tenure Data, 1 - Owner occupied, 2 - Renter occupied,
+        # 3 - Nursing facilities, 4 - Other group quarters, 5 - Vacant for rent
+        # 6 - Vacant for sale, 7 - Vacant other
+        tenure_categories = ["Owner occupied",
+                             "Renter occupied",
+                             "Nursing facilities",
+                             "Other group quarters",
+                             "Vacant for rent",
+                             "Vacant for sale",
+                             "Vacant other",
+                             "Total"]
+
+        huapd = self.pop_disl_result
+        # Allocated by tenure
+        huapd["hua_tnr"] = "0"
+        huapd.loc[huapd["ownershp"] == 1.0, "hua_tnr"] = "1"
+        huapd.loc[huapd["ownershp"] == 2.0, "hua_tnr"] = "2"
+        huapd.loc[huapd["gqtype"] == 3, "hua_tnr"] = "3"
+        huapd.loc[huapd["gqtype"].isin([1, 2, 4, 5, 6, 7, 8]), "hua_tnr"] = "4"
+        huapd.loc[huapd["vacancy"].isin([1, 2]), "hua_tnr"] = "5"
+        huapd.loc[huapd["vacancy"].isin([3, 4]), "hua_tnr"] = "6"
+        huapd.loc[huapd["vacancy"].isin([5, 6, 7]), "hua_tnr"] = "7"
+        hua_vals = huapd["hua_tnr"].value_counts()
+        hua_tot = []
+        for i in range(len(tenure_categories)):
+            try:
+                hua_tot.append(int(hua_vals[str(i)]))
+            except Exception:
+                hua_tot.append(0)
+        hua_tot.append(int(sum(hua_tot[1:])))
+
+        pop_tot = []
+        for i in range(len(tenure_categories)):
+            pop_tot.append(int(huapd["numprec"].where(huapd["hua_tnr"] == str(i)).sum()))
+        pop_tot.append(int(sum(pop_tot[1:])))
+
+        # Dislocated by tenure
+        huapd["hud_tnr"] = "0"
+        huapd.loc[(huapd["ownershp"] == 1.0) & huapd["dislocated"], "hud_tnr"] = "1"
+        huapd.loc[(huapd["ownershp"] == 2.0) & huapd["dislocated"], "hud_tnr"] = "2"
+        huapd.loc[(huapd["gqtype"] == 3) & huapd["dislocated"], "hud_tnr"] = "3"
+        huapd.loc[huapd["gqtype"].isin([1, 2, 4, 5, 6, 7, 8]) & huapd["dislocated"], "hud_tnr"] = "4"
+        huapd.loc[huapd["vacancy"].isin([1, 2]) & huapd["dislocated"], "hud_tnr"] = "5"
+        huapd.loc[huapd["vacancy"].isin([3, 4]) & huapd["dislocated"], "hud_tnr"] = "6"
+        huapd.loc[huapd["vacancy"].isin([5, 6, 7]) & huapd["dislocated"], "hud_tnr"] = "7"
+        hud_vals = huapd["hud_tnr"].value_counts()
+        hua_disl = []
+        for i in range(len(tenure_categories)):
+            try:
+                hua_disl.append(int(hud_vals[str(i)]))
+            except Exception:
+                hua_disl.append(0)
+        hua_disl.append(int(sum(hua_disl[1:])))
+
+        pd_disl = []
+        for i in range(len(tenure_categories)):
+            pd_disl.append(int(huapd["numprec"].where(huapd["hud_tnr"] == str(i)).sum()))
+        pd_disl.append(int(sum(pd_disl[1:])))
+
+        pd_by_tenure_json = []
+        for i in range(len(tenure_categories)):
+            huapd_tenure = {}
+            huapd_tenure[self.HUPD_CATEGORIES[0]] = tenure_categories[i]
+            huapd_tenure[self.HUPD_CATEGORIES[1]] = hua_disl[i + 1]
+            huapd_tenure[self.HUPD_CATEGORIES[2]] = hua_tot[i + 1]
+            if hua_tot[i + 1]:
+                huapd_tenure[self.HUPD_CATEGORIES[3]] = 100 * (hua_disl[i + 1] / hua_tot[i + 1])
+            else:
+                huapd_tenure[self.HUPD_CATEGORIES[3]] = None
+            huapd_tenure[self.HUPD_CATEGORIES[4]] = pd_disl[i + 1]
+            huapd_tenure[self.HUPD_CATEGORIES[5]] = pop_tot[i + 1]
+            if pop_tot[i + 1]:
+                huapd_tenure[self.HUPD_CATEGORIES[6]] = 100 * (pd_disl[i + 1] / pop_tot[i + 1])
+            else:
+                huapd_tenure[self.HUPD_CATEGORIES[6]] = None
+            pd_by_tenure_json.append(huapd_tenure)
+        # print(pd_by_tenure_json)
+
+        if filename_json:
+            with open(filename_json, "w") as outfile:
+                json.dump(pd_by_tenure_json, outfile)
+        # Serializing json
+        return json.dumps(pd_by_tenure_json)
+
+    def pd_by_housing(self, filename_json=None):
+        """ Calculate housing results from the output files of the Joplin Population Dislocation analysis
+        using huestimate column (huestimate = 1 is single family, huestimate > 1 means multi family house)
+        and convert the results to json format.
+        [
+            {"household_characteristics": "Single Family",
+             "household_dislocated": 1162,
+             "total_households": 837,
+             "%_household_dislocated": 7.3,
+             "population_dislocated": 1162,
+             "total_population": 837,
+             "%_population_dislocated"
+             },{},{"Total",..,..,..,..}
+        ]
+
+        Args:
+            filename_json (str): Path and name to save json output file in. E.g "pd_housing_count.json"
+
+        Returns:
+            obj: PD total count by housing. A JSON of the hua and population dislocation housing results by category.
+
+        """
+        # Household categories
+        # 0 - Vacant HU No Tenure Data, 1 - Single Family, 2 - Multi Family
+        household_categories = ["Single Family",
+                                "Multi Family",
+                                "Total"]
+
+        huapd = self.pop_disl_result
+        # Allocated by housing
+        huapd["hua_house"] = "0"
+        huapd.loc[(huapd["huestimate"] == 1.0), "hua_house"] = "1"
+        huapd.loc[(huapd["huestimate"] > 1.0), "hua_house"] = "2"
+        hua_vals = huapd["hua_house"].value_counts()
+        hua_tot = []
+        for i in range(len(household_categories)):
+            try:
+                hua_tot.append(int(hua_vals[str(i)]))
+            except Exception:
+                hua_tot.append(0)
+        hua_tot.append(int(sum(hua_tot[1:])))
+
+        pop_tot = []
+        for i in range(len(household_categories)):
+            pop_tot.append(int(huapd["numprec"].where(huapd["hua_house"] == str(i)).sum()))
+        pop_tot.append(int(sum(pop_tot[1:])))
+
+        # Dislocated by household
+        huapd["hud_house"] = "0"
+        huapd.loc[(huapd["huestimate"] == 1.0) & huapd["dislocated"], "hud_house"] = "1"
+        huapd.loc[(huapd["huestimate"] > 1.0) & huapd["dislocated"], "hud_house"] = "2"
+        hud_vals = huapd["hud_house"].value_counts()
+        hua_disl = []
+        for i in range(len(household_categories)):
+            try:
+                hua_disl.append(int(hud_vals[str(i)]))
+            except Exception:
+                hua_disl.append(0)
+        hua_disl.append(int(sum(hua_disl[1:])))
+
+        pd_disl = []
+        for i in range(len(household_categories)):
+            pd_disl.append(int(huapd["numprec"].where(huapd["hud_house"] == str(i)).sum()))
+        pd_disl.append(int(sum(pd_disl[1:])))
+
+        pd_by_housing_json = []
+        for i in range(len(household_categories)):
+            huapd_household = {}
+            huapd_household[self.HUPD_CATEGORIES[0]] = household_categories[i]
+            huapd_household[self.HUPD_CATEGORIES[1]] = hua_disl[i + 1]
+            huapd_household[self.HUPD_CATEGORIES[2]] = hua_tot[i + 1]
+            if hua_tot[i + 1]:
+                huapd_household[self.HUPD_CATEGORIES[3]] = 100 * (hua_disl[i + 1] / hua_tot[i + 1])
+            else:
+                huapd_household[self.HUPD_CATEGORIES[3]] = None
+            huapd_household[self.HUPD_CATEGORIES[4]] = pd_disl[i + 1]
+            huapd_household[self.HUPD_CATEGORIES[5]] = pop_tot[i + 1]
+            if pop_tot[i + 1]:
+                huapd_household[self.HUPD_CATEGORIES[6]] = 100 * (pd_disl[i + 1] / pop_tot[i + 1])
+            else:
+                huapd_household[self.HUPD_CATEGORIES[6]] = None
+            pd_by_housing_json.append(huapd_household)
+        # print(pd_by_housing_json)
+
+        if filename_json:
+            with open(filename_json, "w") as outfile:
+                json.dump(pd_by_housing_json, outfile)
+        return json.dumps(pd_by_housing_json)
+
+    def pd_total(self, filename_json=None):
+        """ Calculate total results from the output files of the Joplin Population Dislocation analysis
+        and convert the results to json format.
+        {   "household_dislocated": {
+                "dislocated": {
+                    "number": 1999,
+                    "percentage": 0.085
+                }, "not_dislocated": {}, "total": {}
+            },"population_dislocated": {"dislocated": {},"not_dislocated": {}, "total": {}}
+        }
+
+        Args:
+            filename_json (str): Path and name to save json output file in. E.g "pd_total_count.json"
+
+        Returns:
+            obj: PD total count. A JSON of the hua and population dislocation total results by category.
+
+        """
+        # Dislocated by race and ethnicity
+        hud = self.pop_disl_result
+        hud_vals = hud["dislocated"].value_counts()
+        hua_disl = [int(hud_vals[False]), int(hud_vals[True])]
+
+        pd_disl = []
+        pd_disl.append(int(hud["numprec"].where(hud["dislocated"] == 0).sum()))
+        pd_disl.append(int(hud["numprec"].where(hud["dislocated"] == 1).sum()))
+
+        hua_tot = sum(hua_disl)
+        pop_tot = sum(pd_disl)
+
+        hua_disl_tot = {}
+        if hua_tot:
+            hua_disl_tot["dislocated"] = {"households": hua_disl[1],
+                                          "%_of_households": 100 * (hua_disl[1]/hua_tot)}
+            hua_disl_tot["not_dislocated"] = {"households": hua_tot - hua_disl[1],
+                                              "%_of_households": 100 * ((hua_tot - hua_disl[1])/hua_tot)}
+            hua_disl_tot["total"] = {"households": hua_tot, "%_of_households": 100}
+        else:
+            hua_disl_tot["dislocated"] = {"households": None,
+                                          "%_of_households": None}
+            hua_disl_tot["not_dislocated"] = {"households": None,
+                                              "%_of_households": None}
+            hua_disl_tot["total"] = {"households": None, "%_of_households": None}
+
+        pop_disl_tot = {}
+        if pop_tot:
+            pop_disl_tot["dislocated"] = {"population": pd_disl[1],
+                                          "%_of_population": 100 * (pd_disl[1]/pop_tot)}
+            pop_disl_tot["not_dislocated"] = {"population": pop_tot - pd_disl[1],
+                                              "%_of_population": 100 * ((pop_tot - pd_disl[1])/pop_tot)}
+            pop_disl_tot["total"] = {"population": pop_tot, "%_of_population": 100}
+        else:
+            pop_disl_tot["dislocated"] = {"population": None,
+                                          "%_of_population": None}
+            pop_disl_tot["not_dislocated"] = {"population": None,
+                                              "%_of_population": None}
+            pop_disl_tot["total"] = {"population": None, "%_of_population": None}
+
+        pd_total_json = {"household_dislocation_in_total": hua_disl_tot,
+                         "population_dislocation_in_total": pop_disl_tot}
+        # print(pd_total_json)
+
+        if filename_json:
+            with open(filename_json, "w") as outfile:
+                json.dump(pd_total_json, outfile)
+        return json.dumps(pd_total_json)
diff --git a/pyincore/utils/popdisloutputprocessutil.py b/pyincore/utils/popdisloutputprocessutil.py
deleted file mode 100644
index ad16ee7ee..000000000
--- a/pyincore/utils/popdisloutputprocessutil.py
+++ /dev/null
@@ -1,44 +0,0 @@
-# Copyright (c) 2021 University of Illinois and others. All rights reserved.
-#
-# This program and the accompanying materials are made available under the
-# terms of the Mozilla Public License v2.0 which accompanies this distribution,
-# and is available at https://www.mozilla.org/en-US/MPL/2.0/
-
-import geopandas as gpd
-import pandas as pd
-from shapely import wkt
-
-
-class PopdislOutputProcessUtil:
-    """This class converts various population dislocation results outputs to json format and shapefiles."""
-
-    @staticmethod
-    def get_heatmap_shp(pop_dislocation_result, pop_dislocation_result_path=None,
-                        filter_on=True, filename="pop-disl-numprec.shp"):
-        """Convert and filter opulation dislocatin output to shapefile that contains only guid and numprec column
-            Args:
-                pop_dislocation_result (obj): IN-CORE output of populationdislocation.
-                pop_dislocation_result_path (string): Direct path of the csv of population dislocation result. A
-                fallback for the case that object of popdisl is not provided.
-                filename (str): Path and name to save json output file in. E.g "heatmap.shp"
-
-            Returns:
-                filename: full path and filename of the shapefile
-        """
-        if pop_dislocation_result_path:
-            df = pd.read_csv(pop_dislocation_result_path, low_memory=False)
-        else:
-            df = pop_dislocation_result.get_dataframe_from_csv()
-
-        df['geometry'] = df['geometry'].apply(wkt.loads)
-
-        # only keep dislocated & guid
-        if filter_on:
-            df = df[(df['dislocated']) & (df['guid'].notnull()) & (df["numprec"].notnull())]
-
-        # save as shapefile
-        gdf = gpd.GeoDataFrame(df, crs='epsg:4326')
-        gdf = gdf[["guid", "numprec", "geometry"]]
-        gdf.to_file(filename)
-
-        return filename


Test patch:
diff --git a/tests/pyincore/analyses/populationdislocation/test_populationdislocation.py b/tests/pyincore/analyses/populationdislocation/test_populationdislocation.py
index 5737c0493..205f0a931 100644
--- a/tests/pyincore/analyses/populationdislocation/test_populationdislocation.py
+++ b/tests/pyincore/analyses/populationdislocation/test_populationdislocation.py
@@ -14,8 +14,7 @@ def run_with_base_class():
     # incore-dev
     building_dmg = "602d96e4b1db9c28aeeebdce"  # dev Joplin
     # building_dmg = "602d975db1db9c28aeeebe35" # 15 guids test - dev Joplin
-    housing_unit_alloc = "5df7c989425e0b00092c5eb4"  # dev Joplin
-    # housing_unit_alloc = "602ea965b1db9c28aeefa5d6" # 23 address ids test - dev Joplin
+    housing_unit_alloc = "61563545483ecb19e4304c2a"  # dev Joplin
     bg_data = "5df7cb0b425e0b00092c9464"  # Joplin 2ev2
     value_loss = "602d508fb1db9c28aeedb2a5"
 
diff --git a/tests/pyincore/utils/test_csvoutputjson.py b/tests/pyincore/utils/test_csvoutputjson.py
new file mode 100644
index 000000000..790c3bedb
--- /dev/null
+++ b/tests/pyincore/utils/test_csvoutputjson.py
@@ -0,0 +1,33 @@
+# This program and the accompanying materials are made available under the
+# terms of the Mozilla Public License v2.0 which accompanies this distribution,
+# and is available at https://www.mozilla.org/en-US/MPL/2.0/
+
+from pyincore.utils.cgeoutputprocess import CGEOutputProcess
+import os
+
+PYINCOREPATH = "path-to-pyincore"
+TESTAPATH = "pyincore/tests/pyincore/analyses/"
+
+def run_convert_cge_json():
+    # run the JoplinCGE analysis first to get results, csv files
+    cge_json = CGEOutputProcess()
+    filepath = os.path.join(PYINCOREPATH, TESTAPATH, "joplincge")
+
+    cge_json.get_cge_household_count(None,
+                                     os.path.join(filepath, "joplin-pop-disl-results.csv"),
+                                     "cge_total_household_count.json")
+    cge_json.get_cge_gross_income(None,
+                                  os.path.join(filepath, "gross-income.csv"),
+                                  "cge_total_household_income.json")
+    cge_json.get_cge_employment(None, None,
+                                os.path.join(filepath, "pre-disaster-factor-demand.csv"),
+                                os.path.join(filepath, "post-disaster-factor-demand.csv"),
+                                "cge_employment.json")
+    cge_json.get_cge_domestic_supply(None,
+                                     os.path.join(filepath, "domestic-supply.csv"),
+                                     "cge_domestic_supply.json")
+    return True
+
+
+if __name__ == '__main__':
+    run_convert_cge_json()
diff --git a/tests/pyincore/utils/test_popdisloutputprocessutil.py b/tests/pyincore/utils/test_popdisloutputprocess.py
similarity index 52%
rename from tests/pyincore/utils/test_popdisloutputprocessutil.py
rename to tests/pyincore/utils/test_popdisloutputprocess.py
index 0a94a5840..16784c30b 100644
--- a/tests/pyincore/utils/test_popdisloutputprocessutil.py
+++ b/tests/pyincore/utils/test_popdisloutputprocess.py
@@ -2,22 +2,48 @@
 # terms of the Mozilla Public License v2.0 which accompanies this distribution,
 # and is available at https://www.mozilla.org/en-US/MPL/2.0/
 
+from pyincore import PopDislOutputProcess
 from pyincore import IncoreClient, DataService, SpaceService
 from pyincore.analyses.populationdislocation import PopulationDislocation
 import pyincore.globals as pyglobals
-from pyincore import PopdislOutputProcessUtil
+import os
 
 
-def run_with_base_class():
+def upload_shapefile_to_services(client):
+    # Upload to incore services and put under commresilience space
+    # It assumes the shapefile is in the utils directory
+    datasvc = DataService(client)
+    dataset_prop = {
+        "title": "Joplin Population Dislocation For Heatmap Plotting",
+        "description": "Contains only dislocated numprec for Joplin playbook plotting usage",
+        "contributors": [],
+        "dataType": "incore:popdislocationShp",
+        "storedUrl": "",
+        "format": "shapefile"
+    }
+    response = datasvc.create_dataset(dataset_prop)
+    dataset_id = response['id']
+    files = ['joplin-pop-disl-numprec.shp',
+             'joplin-pop-disl-numprec.dbf',
+             'joplin-pop-disl-numprec.shx',
+             'joplin-pop-disl-numprec.prj']
+    datasvc.add_files_to_dataset(dataset_id, files)
+
+    # add to space
+    spacesvc = SpaceService(client)
+    spacesvc.add_dataset_to_space("5f99ba8b0ace240b22a82e00", dataset_id=dataset_id)  # commresilience
+    print(dataset_id + " successfully uploaded and move to commresilience space!")
 
-    # run population dislocation first
-    client = IncoreClient(pyglobals.INCORE_API_DEV_URL)
 
+def run_convert_pd_json_chained(client):
     # Joplin population dislocation
+    # incore-dev
     building_dmg = "602d96e4b1db9c28aeeebdce"  # dev Joplin
-    housing_unit_alloc = "5df7c989425e0b00092c5eb4"  # dev Joplin
+    # building_dmg = "602d975db1db9c28aeeebe35" # 15 guids test - dev Joplin
+    housing_unit_alloc = "61563545483ecb19e4304c2a"  # dev Joplin 2ev3
     bg_data = "5df7cb0b425e0b00092c9464"  # Joplin 2ev2
     value_loss = "602d508fb1db9c28aeedb2a5"
+
     result_name = "joplin-pop-disl-results"
     seed = 1111
 
@@ -32,36 +58,41 @@ def run_with_base_class():
     pop_dis.set_parameter("seed", seed)
 
     pop_dis.run_analysis()
+    pd_result = pop_dis.get_output_dataset("result")
 
-    #######################################################
-    # get result
-    pop_dislocation_result = pop_dis.get_output_dataset("result")
-    filename = PopdislOutputProcessUtil.get_heatmap_shp(pop_dislocation_result, filter_on=True,
-                                                        filename="joplin-pop-disl-numprec.shp")
+    return pd_result
 
-    # upload to incore services and put under commresilience space
-    datasvc = DataService(client)
-    dataset_prop = {
-        "title": "Joplin Population Dislocation For Heatmap Plotting",
-        "description": "only contain dislocated numprec for joplin playbook plotting usage",
-        "contributors": [],
-        "dataType": "incore:popdislocationShp",
-        "storedUrl": "",
-        "format": "shapefile"
-    }
-    response = datasvc.create_dataset(dataset_prop)
-    dataset_id = response['id']
-    files = ['joplin-pop-disl-numprec.shp',
-             'joplin-pop-disl-numprec.dbf',
-             'joplin-pop-disl-numprec.shx',
-             'joplin-pop-disl-numprec.prj']
-    datasvc.add_files_to_dataset(dataset_id, files)
 
-    # add to space
-    spacesvc = SpaceService(client)
-    spacesvc.add_dataset_to_space("5f99ba8b0ace240b22a82e00", dataset_id=dataset_id)  # commresilience
-    print(dataset_id + " successfully uploaded and move to commresilience space!")
+if __name__ == '__main__':
+    # test chaining with population dislocation
+    client = IncoreClient(pyglobals.INCORE_API_DEV_URL)
 
+    pd_result = run_convert_pd_json_chained(client)
 
-if __name__ == '__main__':
-    run_with_base_class()
+    pd_process = PopDislOutputProcess(pd_result)
+
+    pd_process.pd_by_race("PD_by_race.json")
+    pd_process.pd_by_income("PD_by_income.json")
+    pd_process.pd_by_tenure("PD_by_tenure.json")
+    pd_process.pd_by_housing("PD_by_housing.json")
+    pd_process.pd_total("PD_by_total.json")
+
+    filename = pd_process.get_heatmap_shp("joplin-pop-disl-numprec.shp")
+    print("Test chaining", filename)
+    # upload_shapefile_to_services(client)
+
+    # test the external file with a path
+    testpath = ""
+    # testpath = "/Users/<user>/<path_to_pyincore>/pyincore/tests/pyincore/utils"
+    if testpath:
+        pd_process = PopDislOutputProcess(None, os.path.join(testpath, "joplin-pop-disl-results.csv"))
+
+        pd_process.pd_by_race("PD_by_race.json")
+        pd_process.pd_by_income("PD_by_income.json")
+        pd_process.pd_by_tenure("PD_by_tenure.json")
+        pd_process.pd_by_housing("PD_by_housing.json")
+        pd_process.pd_total("PD_by_total.json")
+
+        filename = pd_process.get_heatmap_shp("joplin-pop-disl-numprec.shp")
+        print("Test path", filename)
+    print("DONE")
diff --git a/tests/test_format.py b/tests/test_format.py
index 086560616..3553807a2 100644
--- a/tests/test_format.py
+++ b/tests/test_format.py
@@ -55,7 +55,9 @@
     os.path.join(PYINCORE_ROOT_FOLDER, 'pyincore/analyses/joplinempiricalrestoration/joplinempiricalrestoration.py'),
     os.path.join(PYINCORE_ROOT_FOLDER, 'pyincore/analyses/joplinempiricalrestoration/joplinempirrestor_util.py'),
     os.path.join(PYINCORE_ROOT_FOLDER,
-                 'tests/pyincore/analyses/joplinempiricalrestoration/test_joplinempiricalrestoration.py')
+                 'tests/pyincore/analyses/joplinempiricalrestoration/test_joplinempiricalrestoration.py'),
+    os.path.join(PYINCORE_ROOT_FOLDER, 'tests/pyincore/analyses/populationdislocation/test_populationdislocation.py'),
+    os.path.join(PYINCORE_ROOT_FOLDER, 'tests/pyincore/utils/test_popdisloutputprocess.py')
 ]