removed the redundancy issue and the script is working fast and is much cleaner and less code. Next step is to run it on the full data set.