Repository: usds/justice40-tool (language: python)

Issue:
Low income percentile field is missing from download files 
we have the raw percent field - "Adjusted percent of individuals below 200% Federal Poverty Line" but we don't have the percentile, which is the most important one.


Discussion / hints:

Fix patch:
diff --git a/data/data-pipeline/data_pipeline/content/config/csv.yml b/data/data-pipeline/data_pipeline/content/config/csv.yml
index 93c96a064..120794117 100644
--- a/data/data-pipeline/data_pipeline/content/config/csv.yml
+++ b/data/data-pipeline/data_pipeline/content/config/csv.yml
@@ -68,6 +68,9 @@ fields:
 - score_name: Total population
   label: Total population
   format: float
+- score_name: Percent of individuals below 200% Federal Poverty Line, imputed and adjusted (percentile)
+  label: Adjusted percent of individuals below 200% Federal Poverty Line (percentile)
+  format: float
 - score_name: Percent of individuals below 200% Federal Poverty Line, imputed and adjusted
   label: Adjusted percent of individuals below 200% Federal Poverty Line
   format: float
diff --git a/data/data-pipeline/data_pipeline/content/config/excel.yml b/data/data-pipeline/data_pipeline/content/config/excel.yml
index 34b89deea..0ada16506 100644
--- a/data/data-pipeline/data_pipeline/content/config/excel.yml
+++ b/data/data-pipeline/data_pipeline/content/config/excel.yml
@@ -72,6 +72,9 @@ sheets:
       - score_name: Total population
         label: Total population
         format: float
+      - score_name: Percent of individuals below 200% Federal Poverty Line, imputed and adjusted (percentile)
+        label: Adjusted percent of individuals below 200% Federal Poverty Line (percentile)
+        format: float
       - score_name: Percent of individuals below 200% Federal Poverty Line, imputed and adjusted
         label: Adjusted percent of individuals below 200% Federal Poverty Line
         format: float
diff --git a/data/data-pipeline/data_pipeline/etl/score/constants.py b/data/data-pipeline/data_pipeline/etl/score/constants.py
index 71780516e..c4cb8f6ef 100644
--- a/data/data-pipeline/data_pipeline/etl/score/constants.py
+++ b/data/data-pipeline/data_pipeline/etl/score/constants.py
@@ -382,7 +382,6 @@
     field_names.COUNT_OF_TRIBAL_AREAS_IN_TRACT: "TA_COUNT",
     field_names.PERCENT_OF_TRIBAL_AREA_IN_TRACT: "TA_PERC",
     field_names.PERCENT_OF_TRIBAL_AREA_IN_TRACT_DISPLAY_STRING: "TA_PERC_STR",
-
 }
 
 # columns to round floats to 2 decimals
diff --git a/data/data-pipeline/poetry.lock b/data/data-pipeline/poetry.lock
index 8e2776d8b..b9ea4a230 100644
--- a/data/data-pipeline/poetry.lock
+++ b/data/data-pipeline/poetry.lock
@@ -1537,21 +1537,6 @@ category = "main"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
-[[package]]
-name = "pydantic"
-version = "1.10.2"
-description = "Data validation and settings management using python type hints"
-category = "main"
-optional = false
-python-versions = ">=3.6.1"
-
-[package.dependencies]
-typing-extensions = ">=3.7.4.3"
-
-[package.extras]
-dotenv = ["python-dotenv (>=0.10.4)"]
-email = ["email-validator (>=1.0.3)"]
-
 [[package]]
 name = "pyflakes"
 version = "2.3.1"
@@ -2207,7 +2192,7 @@ testing = ["pytest (>=6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-
 [metadata]
 lock-version = "1.1"
 python-versions = "^3.8"
-content-hash = "f3d61a8c4ca54c580ba8d459aa76a9d956e046c2b8339602497ce53393ba9983"
+content-hash = "5d55ed06ec57aa8b2fe957ff8a5cfd557f310f8fe65377ffd2d5ec3545f2ec82"
 
 [metadata.files]
 ansiwrap = []
@@ -2562,7 +2547,6 @@ pycparser = [
     {file = "pycparser-2.21-py2.py3-none-any.whl", hash = "sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9"},
     {file = "pycparser-2.21.tar.gz", hash = "sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206"},
 ]
-pydantic = []
 pyflakes = []
 pygments = []
 pylint = []
diff --git a/data/data-pipeline/pyproject.toml b/data/data-pipeline/pyproject.toml
index ab7a71103..d83088ac6 100644
--- a/data/data-pipeline/pyproject.toml
+++ b/data/data-pipeline/pyproject.toml
@@ -39,7 +39,6 @@ tqdm = "4.62.0"
 types-requests = "^2.25.0"
 us = "^2.0.2"
 xlsxwriter = "^2.0.0"
-pydantic = "^1.9.0"
 Rtree = "^1.0.0"
 
 [tool.poetry.dev-dependencies]


Test patch:
diff --git a/data/data-pipeline/data_pipeline/etl/score/tests/snapshots/downloadable_data_expected.pkl b/data/data-pipeline/data_pipeline/etl/score/tests/snapshots/downloadable_data_expected.pkl
index 854a6cf66..ced87384f 100644
Binary files a/data/data-pipeline/data_pipeline/etl/score/tests/snapshots/downloadable_data_expected.pkl and b/data/data-pipeline/data_pipeline/etl/score/tests/snapshots/downloadable_data_expected.pkl differ