Package: thredds_crawler
# thredds_crawler[![Build Status](https://travis-ci.org/ioos/thredds_crawler.svg?branch=master)](https://travis-ci.org/ioos/thredds_crawler)A simple crawler/parser for THREDDS catalogs## Installation```bashpip install thredds_crawler```or```bashconda install -c conda-forge thredds_crawler```## Usage### SelectYou can select datasets based on their THREDDS ID using the 'select' parameter.  Python regex is supported.```pythonfrom thredds_crawler.crawl import Crawlc = Crawl('http://tds.maracoos.org/thredds/MODIS.xml', select=[".*-Agg"])print c.datasets[<LeafDataset id: MODIS-Agg, name: MODIS-Complete Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2009-Agg, name: MODIS-2009 Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2010-Agg, name: MODIS-2010 Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2011-Agg, name: MODIS-2011 Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2012-Agg, name: MODIS-2012 Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2013-Agg, name: MODIS-2013 Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-One-Agg, name: 1-Day-Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-Three-Agg, name: 3-Day-Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-Seven-Agg, name: 7-Day-Aggregation, services: ['OPENDAP', 'ISO']>]```### SkipYou can skip datasets based on their `name` and catalogRefs based on their `xlink:title`.  By default, the crawler uses some common regular expressions to skip lists of thousands upon thousands of individual files that are part of aggregations or FMRCs:*  `.*files.*`*  `.*Individual Files.*`*  `.*File_Access.*`*  `.*Forecast Model Run.*`*  `.*Constant Forecast Offset.*`*  `.*Constant Forecast Date.*`By setting the `skip` parameter to anything other than a superset of the default you run the risk of having some angry system admins after you.You can access the default `skip` list through the Crawl.SKIPS class variable```pythonfrom thredds_crawler.crawl import Crawlprint Crawl.SKIPS['.*files.*','.*Individual Files.*','.*File_Access.*','.*Forecast Model Run.*','.*Constant Forecast Offset.*','.*Constant Forecast Date.*']```If you need to remove or add a new `skip`, it is **strongly** encouraged you use the `SKIPS` class variable as a starting point!```pythonfrom thredds_crawler.crawl import Crawlskips = Crawl.SKIPS + [".*-Day-Aggregation"]c = Crawl('http://tds.maracoos.org/thredds/MODIS.xml',select=[".*-Agg"],skip=skips)print c.datasets[<LeafDataset id: MODIS-Agg, name: MODIS-Complete Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2009-Agg, name: MODIS-2009 Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2010-Agg, name: MODIS-2010 Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2011-Agg, name: MODIS-2011 Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2012-Agg, name: MODIS-2012 Aggregation, services: ['OPENDAP', 'ISO']>,<LeafDataset id: MODIS-2013-Agg, name: MODIS-2013 Aggregation, services: ['OPENDAP', 'ISO']>,]```### WorkersBy default there are `4` worker threads used in the crawling. You can change this by specifying a `workers` parameter.```pythonimport timefrom contextlib import contextmanagerfrom thredds_crawler.crawl import Crawl@contextmanagerdef timeit(name):startTime = time.time()yieldelapsedTime = time.time() - startTimeprint('[{}] finished in {} ms'.format(name, int(elapsedTime * 1000)))for x in range(1, 11):with timeit('{} workers'.format(x)):Crawl("http://tds.maracoos.org/thredds/MODIS.xml", workers=x)[1 workers] finished in 872 ms[2 workers] finished in 397 ms[3 workers] finished in 329 ms[4 workers] finished in 260 ms[5 workers] finished in 264 ms[6 workers] finished in 219 ms[7 workers] finished in 212 ms[8 workers] finished in 185 ms[9 workers] finished in 217 ms[10 workers] finished in 205 ms```### Modified TimeYou can select data by the THREDDS `modified_time` by using a the `before` and `after` parameters. Keep in mind that the modified time is only avaiable for individual files hosted in THREDDS (not aggregations).```pythonimport pytzfrom thredds_crawler.crawl import Crawlbf = datetime(2016, 1, 5, 0, 0)af = datetime(2015, 12, 30, 0, 0, tzinfo=pytz.utc)url = 'http://tds.maracoos.org/thredds/catalog/MODIS-Chesapeake-Salinity/raw/2016/catalog.xml'# afterc = Crawl(url, after=af)assert len(c.datasets) == 3# beforec = Crawl(url, before=bf)assert len(c.datasets) == 3# bothaf = datetime(2016, 1, 20, 0, 0)bf = datetime(2016, 2, 1, 0, 0)c = Crawl(url, before=bf, after=af)assert len(c.datasets) == 11```### AuthenticationYou can pass an auth parameter as needed. It needs to be a [requests compatible auth object](http://docs.python-requests.org/en/latest/user/authentication/).```pythonfrom thredds_crawler.crawl import Crawlauth = ('user', 'password')c = Crawl('http://tds.maracoos.org/thredds/MODIS.xml',select=['.*-Agg'],skip=Crawl.SKIPS,auth=auth)```### DebuggingYou can pass in a `debug=True` parameter to Crawl to log to STDOUT what is actually happening.```pythonfrom thredds_crawler.crawl import Crawlskips = Crawl.SKIPS + [".*-Day-Aggregation"]c = Crawl('http://tds.maracoos.org/thredds/MODIS.xml',select=['.*-Agg'],skip=skips,debug=True)Crawling: http://tds.maracoos.org/thredds/MODIS.xmlSkipping catalogRef based on 'skips'.  Title: MODIS Individual FilesSkipping catalogRef based on 'skips'.  Title: 1-Day Individual FilesSkipping catalogRef based on 'skips'.  Title: 3-Day Individual FilesSkipping catalogRef based on 'skips'.  Title: 8-Day Individual FilesProcessing MODIS-AggProcessing MODIS-2009-AggProcessing MODIS-2010-AggProcessing MODIS-2011-AggProcessing MODIS-2012-AggProcessing MODIS-2013-AggSkipping dataset based on 'skips'.  Name: 1-Day-Aggregation```### LoggingIf you are using `thredds_crawler` from inside of another program, you can access its logsusing the named logger `thredds_crawler` to control the log level.  If you access to the namedlogger, **do not** include `debug=True` when initializing the Crawl object.```pythonimport loggingcrawl_log = logging.getLogger('thredds_crawler')crawl_log.setLevel(logging.WARNING)```## DatasetYou can get some basic information about a LeafDataset, including the services available.```pythonfrom thredds_crawler.crawl import Crawlc = Crawl('http://tds.maracoos.org/thredds/MODIS.xml', select=['.*-Agg'])dataset = c.datasets[0]print dataset.idMODIS-Aggprint dataset.nameMODIS-Complete Aggregationprint dataset.services[{'url': 'http://tds.maracoos.org/thredds/dodsC/MODIS-Agg.nc','name': 'odap','service': 'OPENDAP'},{'url': 'http://tds.maracoos.org/thredds/iso/MODIS-Agg.nc','name': 'iso','service': 'ISO'}]```If you have a list of datasets you can easily return all endpoints of a certain type:```pythonfrom thredds_crawler.crawl import Crawlc = Crawl('http://tds.maracoos.org/thredds/MODIS.xml', select=['.*-Agg'])urls = [s.get("url") for d in c.datasets for s in d.services if s.get("service").lower() == "opendap"]print urls['http://tds.maracoos.org/thredds/dodsC/MODIS-Agg.nc','http://tds.maracoos.org/thredds/dodsC/MODIS-2009-Agg.nc','http://tds.maracoos.org/thredds/dodsC/MODIS-2010-Agg.nc','http://tds.maracoos.org/thredds/dodsC/MODIS-2011-Agg.nc','http://tds.maracoos.org/thredds/dodsC/MODIS-2012-Agg.nc','http://tds.maracoos.org/thredds/dodsC/MODIS-2013-Agg.nc','http://tds.maracoos.org/thredds/dodsC/MODIS-One-Agg.nc','http://tds.maracoos.org/thredds/dodsC/MODIS-Three-Agg.nc','http://tds.maracoos.org/thredds/dodsC/MODIS-Seven-Agg.nc']```You can also obtain the dataset size.  This returns the size on disk if the informaton is available in the TDScatalog.  If it is not available and a DAP endpoint is available, it returns the theoretical size of all of thh variables.This isn't necessarialy the size on disk, because it does not account for `missing_value` and `_FillValue` space.```pythonfrom thredds_crawler.crawl import Crawlc = Crawl('http://thredds.axiomalaska.com/thredds/catalogs/cencoos.html',select=['MB_.*'])sizes = [d.size for d in c.datasets]print sizes[29247.410283999998, 72166.289680000002]```## MetadataThe entire THREDDS catalog metadata record is saved along with the dataset object.  It is an etree Element object ready for you to pull information out of.  See the [THREDDS metadata spec](http://www.unidata.ucar.edu/projects/THREDDS/tech/catalog/v1.0.2/InvCatalogSpec.html#metadata)```pythonfrom thredds_crawler.crawl import Crawlc = Crawl('http://tds.maracoos.org/thredds/MODIS.xml', select=['.*-Agg'])dataset = c.datasets[0]print dataset.metadata.find("{http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0}documentation").textOcean Color data are provided as a service to the broader community, and can beinfluenced by sensor degradation and or algorithm changes. We make efforts to keepthis dataset updated and calibrated. The products in these files are experimental.Aggregations are simple means of available data over the specified time frame. Use atyour own discretion.```## Use CaseBelow is a python script that can be used to harvest THEDDS catalogs and save the ISO metadata filesto a local directory```pythonimport osimport urllibfrom thredds_crawler.crawl import Crawlimport loggingimport logging.handlerslogger = logging.getLogger('thredds_crawler')fh = logging.handlers.RotatingFileHandler('/var/log/iso_harvest/iso_harvest.log', maxBytes=1024*1024*10, backupCount=5)fh.setLevel(logging.DEBUG)ch = logging.StreamHandler()ch.setLevel(logging.DEBUG)formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')fh.setFormatter(formatter)ch.setFormatter(formatter)logger.addHandler(fh)logger.addHandler(ch)logger.setLevel(logging.DEBUG)SAVE_DIR="/srv/http/iso"THREDDS_SERVERS = {"aoos":      "http://thredds.axiomalaska.com/thredds/catalogs/aoos.html","cencoos":   "http://thredds.axiomalaska.com/thredds/catalogs/cencoos.html","maracoos" : "http://tds.maracoos.org/thredds/catalog.html","glos":      "http://tds.glos.us/thredds/catalog.html"}for subfolder, thredds_url in THREDDS_SERVERS.items():logger.info("Crawling %s (%s)" % (subfolder, thredds_url))crawler = Crawl(thredds_url, debug=True)isos = [(d.id, s.get("url")) for d in crawler.datasets for s in d.services if s.get("service").lower() == "iso"]filefolder = os.path.join(SAVE_DIR, subfolder)if not os.path.exists(filefolder):os.makedirs(filefolder)for iso in isos:try:filename = iso[0].replace("/", "_") + ".iso.xml"filepath = os.path.join(filefolder, filename)logger.info("Downloading/Saving %s" % filepath)urllib.urlretrieve(iso[1], filepath)except BaseException:logger.exception("Error!")```