Updated crawler database structure and logic