And again. Did something and decided to forget about it
1. Raycast is called now only once per 0.5 seconds instead of every frame
2. Added rigidbodies to all the objects -> changed the script to move them