scaled etl and fixes for memoization and error safety