allowed function for crawler