swtiched to pmap to allow parallel processing of warcs