integrated spider. added spider status
fixed relative URL join issue when in subdirectory