fix databse charset
add date ranges for sps and idps
process chunks by date
add linking of related entityies