update to handle new data formats