master flatten and cleaned data structure