adding clickstream, robot control and spamming control
more cleanup and adding proper indexes