i don't understand why not just use the item struct, and use a new flag bit  in 'it_flags' to identify it is a crawler. creating a new struct and process it as a item seems not a good idea.