Only check the base URL from the main process when pytest-xdist is in use

@bobsilverberg could you give this a review? It may help with issues like #134 and #137.