Repository: mbta/lamp
Pull request #494: FIX: Ingestion processing changes to enable process of backlog 
Author: login: huangh
Created: 2025-04-03T19:24:46Z | Merged: 2025-04-04T15:59:35Z

[📦 Store dev GTFS-RT data in LAMP 2 - incoming -> springboard](https://app.asana.com/0/1205827492903547/1209390211052912/f)

This set of changes following #493 are to get the Dev ingestion job running on backlog data to have the same memory characteristics and performance as the processing on Prod. 

The specific changes made are: 

- add devgreen tripupdates and vehicleposition inputs to be processed by ingestion and published to springboard (the original task...)
- clear out /tmp folder on restart now that the volume persists to eliminate side effects
- **_critically_** fixed bug where light rail gps files were processed at a rate of 5000 per iteration even when the main ingestion loop was grabbing 250k. This resulted in light rail gps files dominating subsequent list calls, reducing the number of files ingested and processed per iteration. 

These changes combined result in max memory usage of ~40%, similar to expectations on PROD, resolving the lingering issue where memory usage consistently approached 80/90%. 

<img width="566" alt="image" src="https://github.com/user-attachments/assets/b5a1470d-dde9-4f04-9f69-1a822cd06889" />

The above graph shows this branch being deployed late 4/2, resulting in drastic reduction of max memory

<img width="1454" alt="image" src="https://github.com/user-attachments/assets/540243ae-d9ce-4846-bb8e-fd1d283ff6a9" />

The number of records processed was preserved

There is variation in exact number of RT S3 Objects moved is due to time of day variation in the amount of data processed in each ingested file. 3AM records are reliably ~half the size, and likely have filtered out non-unique records. 



Diff:
diff --git a/poetry.lock b/poetry.lock
index a4e7433ff..fa32910d7 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -1683,51 +1683,57 @@ tests = ["pytest"]
 
 [[package]]
 name = "pyarrow"
-version = "16.1.0"
+version = "19.0.1"
 description = "Python library for Apache Arrow"
 optional = false
-python-versions = ">=3.8"
+python-versions = ">=3.9"
 files = [
-    {file = "pyarrow-16.1.0-cp310-cp310-macosx_10_15_x86_64.whl", hash = "sha256:17e23b9a65a70cc733d8b738baa6ad3722298fa0c81d88f63ff94bf25eaa77b9"},
-    {file = "pyarrow-16.1.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:4740cc41e2ba5d641071d0ab5e9ef9b5e6e8c7611351a5cb7c1d175eaf43674a"},
-    {file = "pyarrow-16.1.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:98100e0268d04e0eec47b73f20b39c45b4006f3c4233719c3848aa27a03c1aef"},
-    {file = "pyarrow-16.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f68f409e7b283c085f2da014f9ef81e885d90dcd733bd648cfba3ef265961848"},
-    {file = "pyarrow-16.1.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:a8914cd176f448e09746037b0c6b3a9d7688cef451ec5735094055116857580c"},
-    {file = "pyarrow-16.1.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:48be160782c0556156d91adbdd5a4a7e719f8d407cb46ae3bb4eaee09b3111bd"},
-    {file = "pyarrow-16.1.0-cp310-cp310-win_amd64.whl", hash = "sha256:9cf389d444b0f41d9fe1444b70650fea31e9d52cfcb5f818b7888b91b586efff"},
-    {file = "pyarrow-16.1.0-cp311-cp311-macosx_10_15_x86_64.whl", hash = "sha256:d0ebea336b535b37eee9eee31761813086d33ed06de9ab6fc6aaa0bace7b250c"},
-    {file = "pyarrow-16.1.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:2e73cfc4a99e796727919c5541c65bb88b973377501e39b9842ea71401ca6c1c"},
-    {file = "pyarrow-16.1.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bf9251264247ecfe93e5f5a0cd43b8ae834f1e61d1abca22da55b20c788417f6"},
-    {file = "pyarrow-16.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ddf5aace92d520d3d2a20031d8b0ec27b4395cab9f74e07cc95edf42a5cc0147"},
-    {file = "pyarrow-16.1.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:25233642583bf658f629eb230b9bb79d9af4d9f9229890b3c878699c82f7d11e"},
-    {file = "pyarrow-16.1.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:a33a64576fddfbec0a44112eaf844c20853647ca833e9a647bfae0582b2ff94b"},
-    {file = "pyarrow-16.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:185d121b50836379fe012753cf15c4ba9638bda9645183ab36246923875f8d1b"},
-    {file = "pyarrow-16.1.0-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:2e51ca1d6ed7f2e9d5c3c83decf27b0d17bb207a7dea986e8dc3e24f80ff7d6f"},
-    {file = "pyarrow-16.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:06ebccb6f8cb7357de85f60d5da50e83507954af617d7b05f48af1621d331c9a"},
-    {file = "pyarrow-16.1.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b04707f1979815f5e49824ce52d1dceb46e2f12909a48a6a753fe7cafbc44a0c"},
-    {file = "pyarrow-16.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0d32000693deff8dc5df444b032b5985a48592c0697cb6e3071a5d59888714e2"},
-    {file = "pyarrow-16.1.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:8785bb10d5d6fd5e15d718ee1d1f914fe768bf8b4d1e5e9bf253de8a26cb1628"},
-    {file = "pyarrow-16.1.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:e1369af39587b794873b8a307cc6623a3b1194e69399af0efd05bb202195a5a7"},
-    {file = "pyarrow-16.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:febde33305f1498f6df85e8020bca496d0e9ebf2093bab9e0f65e2b4ae2b3444"},
-    {file = "pyarrow-16.1.0-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:b5f5705ab977947a43ac83b52ade3b881eb6e95fcc02d76f501d549a210ba77f"},
-    {file = "pyarrow-16.1.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:0d27bf89dfc2576f6206e9cd6cf7a107c9c06dc13d53bbc25b0bd4556f19cf5f"},
-    {file = "pyarrow-16.1.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0d07de3ee730647a600037bc1d7b7994067ed64d0eba797ac74b2bc77384f4c2"},
-    {file = "pyarrow-16.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fbef391b63f708e103df99fbaa3acf9f671d77a183a07546ba2f2c297b361e83"},
-    {file = "pyarrow-16.1.0-cp38-cp38-manylinux_2_28_aarch64.whl", hash = "sha256:19741c4dbbbc986d38856ee7ddfdd6a00fc3b0fc2d928795b95410d38bb97d15"},
-    {file = "pyarrow-16.1.0-cp38-cp38-manylinux_2_28_x86_64.whl", hash = "sha256:f2c5fb249caa17b94e2b9278b36a05ce03d3180e6da0c4c3b3ce5b2788f30eed"},
-    {file = "pyarrow-16.1.0-cp38-cp38-win_amd64.whl", hash = "sha256:e6b6d3cd35fbb93b70ade1336022cc1147b95ec6af7d36906ca7fe432eb09710"},
-    {file = "pyarrow-16.1.0-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:18da9b76a36a954665ccca8aa6bd9f46c1145f79c0bb8f4f244f5f8e799bca55"},
-    {file = "pyarrow-16.1.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:99f7549779b6e434467d2aa43ab2b7224dd9e41bdde486020bae198978c9e05e"},
-    {file = "pyarrow-16.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f07fdffe4fd5b15f5ec15c8b64584868d063bc22b86b46c9695624ca3505b7b4"},
-    {file = "pyarrow-16.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ddfe389a08ea374972bd4065d5f25d14e36b43ebc22fc75f7b951f24378bf0b5"},
-    {file = "pyarrow-16.1.0-cp39-cp39-manylinux_2_28_aarch64.whl", hash = "sha256:3b20bd67c94b3a2ea0a749d2a5712fc845a69cb5d52e78e6449bbd295611f3aa"},
-    {file = "pyarrow-16.1.0-cp39-cp39-manylinux_2_28_x86_64.whl", hash = "sha256:ba8ac20693c0bb0bf4b238751d4409e62852004a8cf031c73b0e0962b03e45e3"},
-    {file = "pyarrow-16.1.0-cp39-cp39-win_amd64.whl", hash = "sha256:31a1851751433d89a986616015841977e0a188662fcffd1a5677453f1df2de0a"},
-    {file = "pyarrow-16.1.0.tar.gz", hash = "sha256:15fbb22ea96d11f0b5768504a3f961edab25eaf4197c341720c4a387f6c60315"},
+    {file = "pyarrow-19.0.1-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:fc28912a2dc924dddc2087679cc8b7263accc71b9ff025a1362b004711661a69"},
+    {file = "pyarrow-19.0.1-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:fca15aabbe9b8355800d923cc2e82c8ef514af321e18b437c3d782aa884eaeec"},
+    {file = "pyarrow-19.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ad76aef7f5f7e4a757fddcdcf010a8290958f09e3470ea458c80d26f4316ae89"},
+    {file = "pyarrow-19.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d03c9d6f2a3dffbd62671ca070f13fc527bb1867b4ec2b98c7eeed381d4f389a"},
+    {file = "pyarrow-19.0.1-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:65cf9feebab489b19cdfcfe4aa82f62147218558d8d3f0fc1e9dea0ab8e7905a"},
+    {file = "pyarrow-19.0.1-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:41f9706fbe505e0abc10e84bf3a906a1338905cbbcf1177b71486b03e6ea6608"},
+    {file = "pyarrow-19.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:c6cb2335a411b713fdf1e82a752162f72d4a7b5dbc588e32aa18383318b05866"},
+    {file = "pyarrow-19.0.1-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:cc55d71898ea30dc95900297d191377caba257612f384207fe9f8293b5850f90"},
+    {file = "pyarrow-19.0.1-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:7a544ec12de66769612b2d6988c36adc96fb9767ecc8ee0a4d270b10b1c51e00"},
+    {file = "pyarrow-19.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0148bb4fc158bfbc3d6dfe5001d93ebeed253793fff4435167f6ce1dc4bddeae"},
+    {file = "pyarrow-19.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f24faab6ed18f216a37870d8c5623f9c044566d75ec586ef884e13a02a9d62c5"},
+    {file = "pyarrow-19.0.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:4982f8e2b7afd6dae8608d70ba5bd91699077323f812a0448d8b7abdff6cb5d3"},
+    {file = "pyarrow-19.0.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:49a3aecb62c1be1d822f8bf629226d4a96418228a42f5b40835c1f10d42e4db6"},
+    {file = "pyarrow-19.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:008a4009efdb4ea3d2e18f05cd31f9d43c388aad29c636112c2966605ba33466"},
+    {file = "pyarrow-19.0.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:80b2ad2b193e7d19e81008a96e313fbd53157945c7be9ac65f44f8937a55427b"},
+    {file = "pyarrow-19.0.1-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:ee8dec072569f43835932a3b10c55973593abc00936c202707a4ad06af7cb294"},
+    {file = "pyarrow-19.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4d5d1ec7ec5324b98887bdc006f4d2ce534e10e60f7ad995e7875ffa0ff9cb14"},
+    {file = "pyarrow-19.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f3ad4c0eb4e2a9aeb990af6c09e6fa0b195c8c0e7b272ecc8d4d2b6574809d34"},
+    {file = "pyarrow-19.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:d383591f3dcbe545f6cc62daaef9c7cdfe0dff0fb9e1c8121101cabe9098cfa6"},
+    {file = "pyarrow-19.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:b4c4156a625f1e35d6c0b2132635a237708944eb41df5fbe7d50f20d20c17832"},
+    {file = "pyarrow-19.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:5bd1618ae5e5476b7654c7b55a6364ae87686d4724538c24185bbb2952679960"},
+    {file = "pyarrow-19.0.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:e45274b20e524ae5c39d7fc1ca2aa923aab494776d2d4b316b49ec7572ca324c"},
+    {file = "pyarrow-19.0.1-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:d9dedeaf19097a143ed6da37f04f4051aba353c95ef507764d344229b2b740ae"},
+    {file = "pyarrow-19.0.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6ebfb5171bb5f4a52319344ebbbecc731af3f021e49318c74f33d520d31ae0c4"},
+    {file = "pyarrow-19.0.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f2a21d39fbdb948857f67eacb5bbaaf36802de044ec36fbef7a1c8f0dd3a4ab2"},
+    {file = "pyarrow-19.0.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:99bc1bec6d234359743b01e70d4310d0ab240c3d6b0da7e2a93663b0158616f6"},
+    {file = "pyarrow-19.0.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:1b93ef2c93e77c442c979b0d596af45e4665d8b96da598db145b0fec014b9136"},
+    {file = "pyarrow-19.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:d9d46e06846a41ba906ab25302cf0fd522f81aa2a85a71021826f34639ad31ef"},
+    {file = "pyarrow-19.0.1-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:c0fe3dbbf054a00d1f162fda94ce236a899ca01123a798c561ba307ca38af5f0"},
+    {file = "pyarrow-19.0.1-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:96606c3ba57944d128e8a8399da4812f56c7f61de8c647e3470b417f795d0ef9"},
+    {file = "pyarrow-19.0.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8f04d49a6b64cf24719c080b3c2029a3a5b16417fd5fd7c4041f94233af732f3"},
+    {file = "pyarrow-19.0.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5a9137cf7e1640dce4c190551ee69d478f7121b5c6f323553b319cac936395f6"},
+    {file = "pyarrow-19.0.1-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:7c1bca1897c28013db5e4c83944a2ab53231f541b9e0c3f4791206d0c0de389a"},
+    {file = "pyarrow-19.0.1-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:58d9397b2e273ef76264b45531e9d552d8ec8a6688b7390b5be44c02a37aade8"},
+    {file = "pyarrow-19.0.1-cp39-cp39-macosx_12_0_arm64.whl", hash = "sha256:b9766a47a9cb56fefe95cb27f535038b5a195707a08bf61b180e642324963b46"},
+    {file = "pyarrow-19.0.1-cp39-cp39-macosx_12_0_x86_64.whl", hash = "sha256:6c5941c1aac89a6c2f2b16cd64fe76bcdb94b2b1e99ca6459de4e6f07638d755"},
+    {file = "pyarrow-19.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fd44d66093a239358d07c42a91eebf5015aa54fccba959db899f932218ac9cc8"},
+    {file = "pyarrow-19.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:335d170e050bcc7da867a1ed8ffb8b44c57aaa6e0843b156a501298657b1e972"},
+    {file = "pyarrow-19.0.1-cp39-cp39-manylinux_2_28_aarch64.whl", hash = "sha256:1c7556165bd38cf0cd992df2636f8bcdd2d4b26916c6b7e646101aff3c16f76f"},
+    {file = "pyarrow-19.0.1-cp39-cp39-manylinux_2_28_x86_64.whl", hash = "sha256:699799f9c80bebcf1da0983ba86d7f289c5a2a5c04b945e2f2bcf7e874a91911"},
+    {file = "pyarrow-19.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:8464c9fbe6d94a7fe1599e7e8965f350fd233532868232ab2596a71586c5a429"},
+    {file = "pyarrow-19.0.1.tar.gz", hash = "sha256:3bf266b485df66a400f282ac0b6d1b500b9d2ae73314a153dbe97d6d5cc8a99e"},
 ]
 
-[package.dependencies]
-numpy = ">=1.16.6"
+[package.extras]
+test = ["cffi", "hypothesis", "pandas", "pytest", "pytz"]
 
 [[package]]
 name = "pycparser"
@@ -2415,4 +2421,4 @@ files = [
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.10"
-content-hash = "b0c70d1aa81dd382c144fa5f97b71a0323a7b2e8d81f6228f79b928f72bb5d64"
+content-hash = "8e5f7f01af1e1fff50d4cd37dc523dffa966553965e7bb63349ceaf4f2cd10f9"
diff --git a/pyproject.toml b/pyproject.toml
index e3ff29976..b98c34681 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -20,7 +20,7 @@ ad_hoc = 'lamp_py.ad_hoc.pipeline:start'
 [tool.poetry.dependencies]
 python = "^3.10"
 SQLAlchemy = "^2.0.30"
-pyarrow = "^16.1.0"
+pyarrow = "^19.0.1"
 boto3 = "^1.35.2"
 pandas = "^2.2.1"
 numpy = "^1.26.4"
diff --git a/src/lamp_py/ingestion/convert_gtfs_rt.py b/src/lamp_py/ingestion/convert_gtfs_rt.py
index 019bab822..d7710d401 100644
--- a/src/lamp_py/ingestion/convert_gtfs_rt.py
+++ b/src/lamp_py/ingestion/convert_gtfs_rt.py
@@ -101,6 +101,10 @@ def __init__(self, config_type: ConfigType, metadata_queue: Queue[Optional[str]]
             self.detail = RtBusVehicleDetail()
         elif config_type == ConfigType.BUS_TRIP_UPDATES:
             self.detail = RtBusTripDetail()
+        elif config_type == ConfigType.DEV_GREEN_RT_TRIP_UPDATES:
+            self.detail = RtTripDetail()
+        elif config_type == ConfigType.DEV_GREEN_RT_VEHICLE_POSITIONS:
+            self.detail = RtVehicleDetail()
         elif config_type == ConfigType.LIGHT_RAIL:
             raise IgnoreIngestion("Ignore LIGHT_RAIL files")
         else:
@@ -130,6 +134,8 @@ def convert(self) -> None:
                     continue
 
                 self.continuous_pq_update(table)
+                pool = pyarrow.default_memory_pool()
+                pool.release_unused()
                 table_count += 1
                 process_logger.add_metadata(table_count=table_count)
                 # limit number of tables produced on each event loop
@@ -141,6 +147,7 @@ def convert(self) -> None:
         else:
             process_logger.log_complete()
         finally:
+            self.data_parts = {}
             self.move_s3_files()
             self.clean_local_folders()
 
@@ -367,6 +374,8 @@ def make_hash_dataset(self, table: pyarrow.Table, local_path: str) -> pyarrow.da
         :param table: pyarrow Table
         :param local_path: path to local parquet file
         """
+        log = ProcessLogger("make_hash_datset")
+        log.log_start()
         table = hash_gtfs_rt_table(table)
         out_ds = pd.dataset(table)
 
@@ -389,7 +398,7 @@ def make_hash_dataset(self, table: pyarrow.Table, local_path: str) -> pyarrow.da
                     out_ds = pd.dataset(table)
                 else:
                     raise exception
-
+        log.log_complete()
         return out_ds
 
     # pylint: disable=R0914
@@ -413,15 +422,28 @@ def write_local_pq(self, table: pyarrow.Table, local_path: str) -> None:
         with tempfile.TemporaryDirectory() as temp_dir:
             hash_pq_path = os.path.join(temp_dir, "hash.parquet")
             upload_path = os.path.join(temp_dir, "upload.parquet")
-            hash_writer = pq.ParquetWriter(hash_pq_path, schema=out_ds.schema)
-            upload_writer = pq.ParquetWriter(upload_path, schema=no_hash_schema)
+            hash_writer = pq.ParquetWriter(hash_pq_path, schema=out_ds.schema, compression="zstd", compression_level=3)
+            upload_writer = pq.ParquetWriter(
+                upload_path, schema=no_hash_schema, compression="zstd", compression_level=3
+            )
 
             partitions = pc.unique(
                 out_ds.to_table(columns=[self.detail.partition_column]).column(self.detail.partition_column)
             )
             for part in partitions:
-                unique_table = (
-                    pl.DataFrame(
+                logger.add_metadata(table_part=str(part), part_rows=0, part_mbs=0)
+                write_table = out_ds.to_table(
+                    filter=(
+                        (pc.field(self.detail.partition_column) == part) & (pc.field("feed_timestamp") < unique_ts_min)
+                    )
+                )
+                logger.add_metadata(
+                    part_rows=write_table.num_rows, part_mbs=round(write_table.nbytes / (1024 * 1024), 2)
+                )
+                hash_writer.write_table(write_table)
+                upload_writer.write_table(write_table.drop_columns(GTFS_RT_HASH_COL))
+                write_table = (
+                    pl.from_arrow(
                         out_ds.to_table(
                             filter=(
                                 (pc.field(self.detail.partition_column) == part)
@@ -429,21 +451,15 @@ def write_local_pq(self, table: pyarrow.Table, local_path: str) -> None:
                             )
                         )
                     )
-                    .sort(by=["feed_timestamp"])
+                    .sort(by=["feed_timestamp"])  # type: ignore
                     .unique(subset=GTFS_RT_HASH_COL, keep="first")
                     .to_arrow()
                     .cast(out_ds.schema)
                 )
-                ds_table = out_ds.to_table(
-                    filter=(
-                        (pc.field(self.detail.partition_column) == part) & (pc.field("feed_timestamp") < unique_ts_min)
-                    )
+                logger.add_metadata(
+                    part_rows=write_table.num_rows, part_mbs=round(write_table.nbytes / (1024 * 1024), 2)
                 )
-                write_table = pyarrow.concat_tables([unique_table, ds_table]).sort_by(self.detail.table_sort_order)
-
                 hash_writer.write_table(write_table)
-
-                # drop GTFS_RT_HASH_COL column for S3 upload
                 upload_writer.write_table(write_table.drop_columns(GTFS_RT_HASH_COL))
 
             hash_writer.close()
diff --git a/src/lamp_py/ingestion/converter.py b/src/lamp_py/ingestion/converter.py
index 14003ca82..b48318de7 100644
--- a/src/lamp_py/ingestion/converter.py
+++ b/src/lamp_py/ingestion/converter.py
@@ -27,6 +27,8 @@ class ConfigType(Enum):
     BUS_VEHICLE_POSITIONS = auto()
     VEHICLE_COUNT = auto()
     SCHEDULE = auto()
+    DEV_GREEN_RT_TRIP_UPDATES = auto()
+    DEV_GREEN_RT_VEHICLE_POSITIONS = auto()
 
     # this filetype is currently being added from delta into our incoming
     # bucket. we haven't looked into it yet, and its ingestion remains
@@ -74,6 +76,11 @@ def from_filename(cls, filename: str) -> ConfigType:
         if "LightRailRawGPS" in filename:
             return cls.LIGHT_RAIL
 
+        if "https_mbta_gtfs_s3_dev_green.s3.amazonaws.com_rtr_TripUpdates_enhanced" in filename:
+            return cls.DEV_GREEN_RT_TRIP_UPDATES
+        if "https_mbta_gtfs_s3_dev_green.s3.amazonaws.com_rtr_VehiclePositions_enhanced.json" in filename:
+            return cls.DEV_GREEN_RT_VEHICLE_POSITIONS
+
         raise ConfigTypeFromFilenameException(filename)
 
     def is_gtfs(self) -> bool:
@@ -90,6 +97,8 @@ def is_gtfs_rt(self) -> bool:
             self.BUS_VEHICLE_POSITIONS,
             self.VEHICLE_COUNT,
             self.LIGHT_RAIL,
+            self.DEV_GREEN_RT_VEHICLE_POSITIONS,
+            self.DEV_GREEN_RT_TRIP_UPDATES,
         ]
 
 
diff --git a/src/lamp_py/ingestion/ingest_gtfs.py b/src/lamp_py/ingestion/ingest_gtfs.py
index 2eee11f13..8cff8dfa7 100644
--- a/src/lamp_py/ingestion/ingest_gtfs.py
+++ b/src/lamp_py/ingestion/ingest_gtfs.py
@@ -63,7 +63,7 @@ def ingest_gtfs_archive(metadata_queue: Queue[Optional[str]]) -> None:
     logger.log_complete()
 
 
-def ingest_s3_files(metadata_queue: Queue[Optional[str]]) -> None:
+def ingest_s3_files(metadata_queue: Queue[Optional[str]], bucket_filter: str = LAMP) -> None:
     """
     get all of the filepaths currently in the incoming bucket, sort them into
     batches of similar gtfs-rt files, convert each batch into tables, write the
@@ -75,13 +75,12 @@ def ingest_s3_files(metadata_queue: Queue[Optional[str]]) -> None:
     logger.log_start()
 
     try:
-        files = file_list_from_s3(
-            bucket_name=S3_INCOMING,
-            file_prefix=LAMP,
-        )
+        files = file_list_from_s3(bucket_name=S3_INCOMING, file_prefix=bucket_filter, max_list_size=50000)
 
         grouped_files = group_sort_file_list(files)
 
+        for k, v in grouped_files.items():
+            logger.add_metadata(ingest_config_type=k, ingest_number_of_files=len(v))
         # initialize with an error / no impl converter, the rest will be added in as
         # the appear.
         converters: Dict[ConfigType, Converter] = {}
@@ -121,8 +120,11 @@ def ingest_s3_files(metadata_queue: Queue[Optional[str]]) -> None:
     # "spawn" some of the behavior described above only occurs when using
     # "fork". On OSX (and Windows?) to force this behavior, run
     # multiprocessing.set_start_method("fork") when starting the script.
+    process_count = os.cpu_count()
+    if process_count is None:
+        process_count = 4
     if len(converters) > 0:
-        with get_context("spawn").Pool(processes=len(converters)) as pool:
+        with get_context("spawn").Pool(processes=process_count, maxtasksperchild=1) as pool:
             pool.map_async(run_converter, converters.values())
             pool.close()
             pool.join()
@@ -130,7 +132,7 @@ def ingest_s3_files(metadata_queue: Queue[Optional[str]]) -> None:
     logger.log_complete()
 
 
-def ingest_gtfs(metadata_queue: Queue[Optional[str]]) -> None:
+def ingest_gtfs(metadata_queue: Queue[Optional[str]], bucket_filter: str = LAMP) -> None:
     """
     ingest all gtfs file types
 
@@ -138,4 +140,4 @@ def ingest_gtfs(metadata_queue: Queue[Optional[str]]) -> None:
     """
     gtfs_to_parquet()
     ingest_gtfs_archive(metadata_queue)
-    ingest_s3_files(metadata_queue)
+    ingest_s3_files(metadata_queue, bucket_filter=bucket_filter)
diff --git a/src/lamp_py/ingestion/light_rail_gps.py b/src/lamp_py/ingestion/light_rail_gps.py
index d97bb44a9..48e823474 100644
--- a/src/lamp_py/ingestion/light_rail_gps.py
+++ b/src/lamp_py/ingestion/light_rail_gps.py
@@ -174,7 +174,7 @@ def write_parquet(dataframe: pl.DataFrame) -> None:
         logger.log_complete()
 
 
-def ingest_light_rail_gps() -> None:
+def ingest_light_rail_gps(bucket_filter: str = LAMP) -> None:
     """
     retrieve group of LightRailRawGPS gz files from INCOMING bucket
 
@@ -189,7 +189,7 @@ def ingest_light_rail_gps() -> None:
     try:
         s3_files = file_list_from_s3(
             bucket_name=os.environ["INCOMING_BUCKET"],
-            file_prefix=LAMP,
+            file_prefix=bucket_filter,
             max_list_size=5_000,
         )
 
diff --git a/src/lamp_py/ingestion/pipeline.py b/src/lamp_py/ingestion/pipeline.py
index 47a35f9fb..28b3b5cc9 100755
--- a/src/lamp_py/ingestion/pipeline.py
+++ b/src/lamp_py/ingestion/pipeline.py
@@ -4,6 +4,7 @@
 import time
 import logging
 import signal
+import shutil
 
 from lamp_py.aws.ecs import handle_ecs_sigterm, check_for_sigterm
 from lamp_py.aws.kinesis import KinesisReader
@@ -15,11 +16,27 @@
 from lamp_py.ingestion.ingest_gtfs import ingest_gtfs
 from lamp_py.ingestion.glides import ingest_glides_events
 from lamp_py.ingestion.light_rail_gps import ingest_light_rail_gps
+from lamp_py.runtime_utils.remote_files import LAMP
 
 logging.getLogger().setLevel("INFO")
 DESCRIPTION = """Entry Point For GTFS Ingestion Scripts"""
 
 
+def clear_folder(folder: str) -> None:
+    """
+    Delete contents of entire folder.
+    """
+    for filename in os.listdir(folder):
+        file_path = os.path.join(folder, filename)
+        try:
+            if os.path.isfile(file_path) or os.path.islink(file_path):
+                os.unlink(file_path)
+            elif os.path.isdir(file_path):
+                shutil.rmtree(file_path)
+        except Exception as _:
+            pass
+
+
 def main() -> None:
     """
     run the ingestion pipeline
@@ -42,10 +59,10 @@ def main() -> None:
     while True:
         process_logger = ProcessLogger(process_name="main")
         process_logger.log_start()
-
+        bucket_filter = LAMP
         check_for_sigterm(metadata_queue, rds_process)
-        ingest_light_rail_gps()
-        ingest_gtfs(metadata_queue)
+        ingest_light_rail_gps(bucket_filter=bucket_filter)
+        ingest_gtfs(metadata_queue, bucket_filter=bucket_filter)
         ingest_glides_events(glides_reader, metadata_queue)
         check_for_sigterm(metadata_queue, rds_process)
 
@@ -56,6 +73,7 @@ def main() -> None:
 
 def start() -> None:
     """configure and start the ingestion process"""
+    clear_folder("/tmp")
     # setup handling shutdown commands
     signal.signal(signal.SIGTERM, handle_ecs_sigterm)
 
diff --git a/src/lamp_py/ingestion/utils.py b/src/lamp_py/ingestion/utils.py
index 371cf2852..3ab5c8a7c 100644
--- a/src/lamp_py/ingestion/utils.py
+++ b/src/lamp_py/ingestion/utils.py
@@ -13,7 +13,6 @@
 from io import BytesIO
 
 import pyarrow
-import pyarrow.dataset as pd
 import pyarrow.parquet as pq
 import pyarrow.compute as pc
 import polars as pl
@@ -215,6 +214,10 @@ def hash_gtfs_rt_table(table: pyarrow.Table) -> pyarrow.Table:
     """
     add GTFS_RT_HASH_COL column to pyarrow table, if not already present
     """
+    log = ProcessLogger(
+        "hash_gtfs_rt_table", table_rows=table.num_rows, table_mbs=round(table.nbytes / (1024 * 1024), 2)
+    )
+    log.log_start()
     hash_columns = table.column_names
     if GTFS_RT_HASH_COL in hash_columns:
         return table
@@ -225,7 +228,7 @@ def hash_gtfs_rt_table(table: pyarrow.Table) -> pyarrow.Table:
     hash_schema = table.schema.append(pyarrow.field(GTFS_RT_HASH_COL, pyarrow.large_binary()))
 
     table = pl.from_arrow(table)
-
+    log.log_complete()
     return (
         table.with_columns(
             table.select(hash_columns)
@@ -242,20 +245,21 @@ def hash_gtfs_rt_parquet(path: str) -> None:
     """
     add GTFS_RT_HASH_COL to local parquet file, if not already present
     """
-    ds = pd.dataset(path)
-    hash_columns = ds.schema.names
+    ds = pq.ParquetFile(path)
+    ds_schema = ds.schema.to_arrow_schema()
+    hash_columns = ds_schema.names
     if GTFS_RT_HASH_COL in hash_columns:
         return
 
     hash_columns.remove("feed_timestamp")
     hash_columns = sorted(hash_columns)
 
-    hash_schema = ds.schema.append(pyarrow.field(GTFS_RT_HASH_COL, pyarrow.large_binary()))
+    hash_schema = ds_schema.append(pyarrow.field(GTFS_RT_HASH_COL, pyarrow.large_binary()))
 
     with tempfile.TemporaryDirectory() as temp_dir:
         tmp_pq = os.path.join(temp_dir, "temp.parquet")
         with pq.ParquetWriter(tmp_pq, schema=hash_schema) as writer:
-            for batch in ds.to_batches(batch_size=1024 * 1024):
+            for batch in ds.iter_batches(batch_size=512 * 1024):
                 batch = pl.from_arrow(batch)
                 batch = (
                     batch.with_columns(


Reviews:
user: login: rymarczy
body: I'm not quite sure what the light rail gps bug is, or where in this change it got fixed?

Otherwise, looks good.A few questions and a few suggestions that are at your discretion to implement. 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T10:07:00Z
user: login: huangh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-04T15:43:58Z
user: login: huangh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-04T15:44:51Z
user: login: huangh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-04T15:54:41Z
user: login: huangh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-04T15:57:45Z

Review comments:
diff_hunk: @@ -130,6 +134,8 @@ def convert(self) -> None:
                     continue
 
                 self.continuous_pq_update(table)
+                pool = pyarrow.default_memory_pool()
+                pool.release_unused()
path: src/lamp_py/ingestion/convert_gtfs_rt.py
user: login: rymarczy
body: Pretty sure this isn't really doing anything, but doesn't hurt to leave in.

Suggestion: add a comment about it's dubious validity. 
created_at: 2025-04-04T09:49:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -141,6 +147,7 @@ def convert(self) -> None:
         else:
             process_logger.log_complete()
         finally:
+            self.data_parts = {}
path: src/lamp_py/ingestion/convert_gtfs_rt.py
user: login: rymarczy
body: Not sure this is doing anything..
created_at: 2025-04-04T09:49:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,10 +59,10 @@ def main() -> None:
     while True:
         process_logger = ProcessLogger(process_name="main")
         process_logger.log_start()
-
+        bucket_filter = LAMP
         check_for_sigterm(metadata_queue, rds_process)
-        ingest_light_rail_gps()
-        ingest_gtfs(metadata_queue)
+        ingest_light_rail_gps(bucket_filter=bucket_filter)
+        ingest_gtfs(metadata_queue, bucket_filter=bucket_filter)
path: src/lamp_py/ingestion/pipeline.py
user: login: rymarczy
body: I don't quite understand the purpose of adding this value as a function parameter vs just using `LAMP` in the function? 
created_at: 2025-04-04T09:51:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -189,7 +189,7 @@ def ingest_light_rail_gps() -> None:
     try:
         s3_files = file_list_from_s3(
             bucket_name=os.environ["INCOMING_BUCKET"],
-            file_prefix=LAMP,
+            file_prefix=bucket_filter,
path: src/lamp_py/ingestion/light_rail_gps.py
user: login: rymarczy
body: Same question as `pipeline.py` change, don't really understand the purpose of this?
created_at: 2025-04-04T09:53:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -413,37 +422,44 @@ def write_local_pq(self, table: pyarrow.Table, local_path: str) -> None:
         with tempfile.TemporaryDirectory() as temp_dir:
             hash_pq_path = os.path.join(temp_dir, "hash.parquet")
             upload_path = os.path.join(temp_dir, "upload.parquet")
-            hash_writer = pq.ParquetWriter(hash_pq_path, schema=out_ds.schema)
-            upload_writer = pq.ParquetWriter(upload_path, schema=no_hash_schema)
+            hash_writer = pq.ParquetWriter(hash_pq_path, schema=out_ds.schema, compression="zstd", compression_level=3)
+            upload_writer = pq.ParquetWriter(
+                upload_path, schema=no_hash_schema, compression="zstd", compression_level=3
+            )
 
             partitions = pc.unique(
                 out_ds.to_table(columns=[self.detail.partition_column]).column(self.detail.partition_column)
             )
             for part in partitions:
-                unique_table = (
-                    pl.DataFrame(
+                logger.add_metadata(table_part=str(part), part_rows=0, part_mbs=0)
+                write_table = out_ds.to_table(
+                    filter=(
+                        (pc.field(self.detail.partition_column) == part) & (pc.field("feed_timestamp") < unique_ts_min)
+                    )
+                )
+                logger.add_metadata(
+                    part_rows=write_table.num_rows, part_mbs=round(write_table.nbytes / (1024 * 1024), 2)
+                )
+                hash_writer.write_table(write_table)
+                upload_writer.write_table(write_table.drop_columns(GTFS_RT_HASH_COL))
path: src/lamp_py/ingestion/convert_gtfs_rt.py
user: login: rymarczy
body: Suggestion: I would change this back to the previous implementation that was only executing one `write_table` command per partition. I don't think this is providing any memory savings and I wouldn't be surprised if the extra `write_table` wasn't making this 2x slower for each partition. 

I'd also suggest dropping all of these `add_metadata` calls as I was mostly using them for debugging. Unless you're utilizing them for some logging purpose, they add a lot of noise. 
created_at: 2025-04-04T09:58:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -75,13 +75,12 @@ def ingest_s3_files(metadata_queue: Queue[Optional[str]]) -> None:
     logger.log_start()
 
     try:
-        files = file_list_from_s3(
-            bucket_name=S3_INCOMING,
-            file_prefix=LAMP,
-        )
+        files = file_list_from_s3(bucket_name=S3_INCOMING, file_prefix=bucket_filter, max_list_size=50000)
path: src/lamp_py/ingestion/ingest_gtfs.py
user: login: rymarczy
body: Is dropping `max_list_size` from the default 250,000 down to 50,000 for a specific reason? 
created_at: 2025-04-04T10:04:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,6 +134,8 @@ def convert(self) -> None:
                     continue
 
                 self.continuous_pq_update(table)
+                pool = pyarrow.default_memory_pool()
+                pool.release_unused()
path: src/lamp_py/ingestion/convert_gtfs_rt.py
user: login: huangh
body: will address in subsequent ingestion improvement tasks
created_at: 2025-04-04T15:43:58Z
author_association: COLLABORATOR
diff_hunk: @@ -141,6 +147,7 @@ def convert(self) -> None:
         else:
             process_logger.log_complete()
         finally:
+            self.data_parts = {}
path: src/lamp_py/ingestion/convert_gtfs_rt.py
user: login: huangh
body: will address in follow on ingestion improvement/cleanup tasks
created_at: 2025-04-04T15:44:51Z
author_association: COLLABORATOR
diff_hunk: @@ -42,10 +59,10 @@ def main() -> None:
     while True:
         process_logger = ProcessLogger(process_name="main")
         process_logger.log_start()
-
+        bucket_filter = LAMP
         check_for_sigterm(metadata_queue, rds_process)
-        ingest_light_rail_gps()
-        ingest_gtfs(metadata_queue)
+        ingest_light_rail_gps(bucket_filter=bucket_filter)
+        ingest_gtfs(metadata_queue, bucket_filter=bucket_filter)
path: src/lamp_py/ingestion/pipeline.py
user: login: huangh
body: this is left here for now because it will not have any issues processing in current-date prod, but the filter was used to align the ingested date for both ingest_light_rail_gps and ingest_gtfs
created_at: 2025-04-04T15:54:41Z
author_association: COLLABORATOR
diff_hunk: @@ -75,13 +75,12 @@ def ingest_s3_files(metadata_queue: Queue[Optional[str]]) -> None:
     logger.log_start()
 
     try:
-        files = file_list_from_s3(
-            bucket_name=S3_INCOMING,
-            file_prefix=LAMP,
-        )
+        files = file_list_from_s3(bucket_name=S3_INCOMING, file_prefix=bucket_filter, max_list_size=50000)
path: src/lamp_py/ingestion/ingest_gtfs.py
user: login: huangh
body: empirically this was the change that produced and matched the expected memory usage profile on dev vs prod. if it looks the same, then we can revert it back to default 250k
created_at: 2025-04-04T15:57:45Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-03T19:31:35Z
body: ### [LCOV](https://github.com/marketplace/actions/report-lcov) of commit [<code>1d6a313</code>](494/commits/1d6a3132224e48dd7b652493cc31fe85d29bf5ec) during [Continuous Integration (Python) #975](../actions/runs/14251121720)
<pre>Summary coverage rate:
  lines......: 69.5% (2730 of 3926 lines)
  functions..: no data found
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ingestion/convert_gtfs_rt.py                                           |46.0%    235|    -     0|    -      0
  src/lamp_py/ingestion/converter.py                                                 |86.4%     66|    -     0|    -      0
  src/lamp_py/ingestion/light_rail_gps.py                                            |54.4%     79|    -     0|    -      0
  src/lamp_py/ingestion/utils.py                                                     |59.1%    115|    -     0|    -      0</pre>
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-03T20:29:22Z
body: ### [LCOV](https://github.com/marketplace/actions/report-lcov) of commit [<code>2b4684f</code>](494/commits/2b4684f3cd52f91e53e516a5a6720476d8b90f2a) during [Continuous Integration (Python) #976](../actions/runs/14252091965)
<pre>Summary coverage rate:
  lines......: 69.6% (2730 of 3924 lines)
  functions..: no data found
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ingestion/convert_gtfs_rt.py                                           |46.4%    233|    -     0|    -      0
  src/lamp_py/ingestion/converter.py                                                 |86.4%     66|    -     0|    -      0
  src/lamp_py/ingestion/light_rail_gps.py                                            |54.4%     79|    -     0|    -      0
  src/lamp_py/ingestion/utils.py                                                     |59.1%    115|    -     0|    -      0</pre>
author_association: NONE