changes to crawler to make it work without publisher config table