converted pipeline from csv to sqlite for more robustness/aggregated storage