now data reformatting and merging seemingly works very well