moved some functions between spider and pipeline