Project: datamanagement
Issue DM-6494: Better error messages from the camera mapper when a template cannot be formatted
The CameraMapper produces a very unhelpful traceback if it cannot format a template string with the provided data ID dict. For example: {code} Traceback (most recent call last): File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_tasks/12.0.rc1-3-gb785bf9/bin/processCcd.py", line 25, in ProcessCcdTask.parseAndRun() File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/cmdLineTask.py", line 450, in parseAndRun parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/argumentParser.py", line 479, in parse_args self._processDataIds(namespace) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/argumentParser.py", line 577, in _processDataIds dataIdContainer.makeDataRefList(namespace) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/argumentParser.py", line 126, in makeDataRefList dataRef=dr)] File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/argumentParser.py", line 935, in dataExists return butler.datasetExists(datasetType = datasetType, dataId = dataRef.dataId) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/butler.py", line 288, in datasetExists locations = self.repository.map(datasetType, dataId) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/repository.py", line 392, in map return self.doParents(Repository.doMap, *args, **kwargs) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/repository.py", line 325, in doParents res = func(parent, *args, **kwargs) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/repository.py", line 405, in doMap loc = self._mapper.map(*args, **kwargs) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/mapper.py", line 169, in map return func(self.validate(dataId), write) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/12.0.rc1+6/python/lsst/daf/butlerUtils/cameraMapper.py", line 284, in mapClosure return mapping.map(mapper, dataId, write) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/12.0.rc1+6/python/lsst/daf/butlerUtils/mapping.py", line 123, in map path = mapper._mapActualToPath(self.template, actualId) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/12.0.rc1+6/python/lsst/daf/butlerUtils/cameraMapper.py", line 732, in _mapActualToPath return template % self._transformId(actualId) TypeError: %d format: a number is required, not NoneType {code} It is unclear what string was being formatted with what data, making the problem difficult to diagnose and correct. I suggest changing line 732 of CameraMapper.py from: {code} return template % self._transformId(actualId) {code} to something like the following: {code} try: transformedId = self._transformId(actualId) return template % transformedId except Exception as e: raise RuntimeError("Failed to format %r with data %r: %s" % (template, transformedId, e)) {code} Here are the last few lines of the same traceback after applying this change: {code} path = mapper._mapActualToPath(self.template, actualId) File "/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/12.0.rc1+6/python/lsst/daf/butlerUtils/cameraMapper.py", line 735, in _mapActualToPath raise RuntimeError("Failed to format %r with data %r: %s" % (template, transformedId, e)) RuntimeError: Failed to format '%(date)s/%(filter)s/decam%(visit)07d.fits.fz[%(hdu)d]' with data {'date': '2013-02-10', 'ccdnum': 10, 'hdu': None, 'visit': 176837, 'filter': 'z'}: %d format: a number is required, not NoneType {code} A bit wordy, but it is much easier to figure out what went wrong. I have stumbled across this problem twice in the last few weeks, so I consider this change fairly important. The first time it was caused by a defective format string in a paf file. This time I'm not yet sure what is causing it, but at least I have something to go on.
Story points: 1