Project: datamanagement
Issue DM-5078: PcaPsf can hit an assertion failure
This is bad for multiple reasons: 1. When multiprocessing, the assertion failure kills a single process, which prevents the final join of the multiple processes, so the job hangs forever. 2. The failure is not logged. 3. Hard assertions like this should only occur when we break the system integrity, which this does not (i.e., it's too big a hammer for the problem). {code} pprice@tiger-sumire:/tigress/pprice/dm-4692 $ eups list -s afw tickets.DM-4692-gd8ad35cd96+1 b1901 setup afwdata 2016_01.0 b1901 b1902 setup astrometry_net 0.50.lsst2+5 b1901 b1902 setup astrometry_net_data sdss-dr9-fink-v5b setup base 2016_01.0 b1901 b1902 setup boost 1.59.lsst5 b1901 b1902 setup cfitsio 3360.lsst4 b1901 b1902 setup coadd_chisquared 2016_01.0+6 b1901 setup coadd_utils 2016_01.0+6 b1901 setup daf_base 2016_01.0 b1901 b1902 setup daf_butlerUtils tickets.DM-4692-g048b33c50e+3 b1901 setup daf_persistence 2016_01.0-1-gf47bb69+1 b1901 b1902 setup display_ds9 2015_10.0+43 b1901 setup doxygen 1.8.5.lsst1 b1901 b1902 setup eigen 3.2.5 b1901 b1902 setup fftw 3.3.4.lsst2 b1901 b1902 setup geom 10.0+50 b1901 b1902 setup gsl 1.16.lsst3 b1901 b1902 setup ip_diffim tickets.DM-4692-g543ea8fde5+3 b1901 setup ip_isr 2016_01.0+6 b1901 setup lsst_build LOCAL:/tigress/pprice/lsstsw/lsst_build setup mariadbclient master-gf2dee38289 b1901 b1902 setup matplotlib 0.0.1+5 b1901 b1902 setup meas_algorithms tickets.DM-4692-g3d073a93d7+1 b1901 setup meas_astrom tickets.DM-4692-gbbf15418e6+1 b1901 setup meas_base LOCAL:/tigress/pprice/dm-4692/meas_base setup meas_deblender 2016_01.0+6 b1901 setup minuit2 5.28.00.lsst2 b1901 b1902 setup ndarray 10.1+58 b1901 b1902 setup numpy 0.0.1+5 b1901 b1902 setup obs_subaru LOCAL:/tigress/pprice/dm-4692/obs_subaru setup obs_test tickets.DM-4692-g1533aee20f+1 b1901 setup pex_config 2016_01.0 b1901 b1902 setup pex_exceptions 2016_01.0 b1901 b1902 setup pex_logging 2016_01.0 b1901 b1902 setup pex_policy 2016_01.0 b1901 b1902 setup pipe_base 2016_01.0+6 b1901 setup pipe_tasks LOCAL:/tigress/pprice/dm-4692/pipe_tasks setup psfex 2016_01.0 b1901 b1902 setup pyfits 3.4.0 b1901 b1902 setup python 0.0.3 b1901 b1902 setup python_d2to1 0.2.12 b1901 b1902 setup pyyaml 3.11.lsst1 b1901 b1902 setup scons 2.3.5 b1901 b1902 setup sconsUtils 2016_01.0 b1901 b1902 setup skymap 2016_01.0+6 b1901 setup skypix 10.0+347 b1901 setup stsci_distutils 0.3.7-1-gb22a065 b1901 b1902 setup swig 3.0.2.lsst1 b1901 b1902 setup utils 2016_01.0 b1901 b1902 setup wcslib 5.13.lsst1 b1901 b1902 setup xpa 2.1.15.lsst3 b1901 b1902 setup pprice@tiger-sumire:/tigress/pprice/dm-4692 $ processCcd.py /tigress/HSC/HSC --rerun price/dm-4692 --rerun price/dm-4692 --id visit=1248 ccd=100 --clobber-config /tigress/pprice/lsstsw/miniconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment. warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.') : Loading config overrride file '/tigress/pprice/dm-4692/obs_subaru/config/processCcd.py' WARNING: Unable to use psfex: No module named extensions.psfex.psfexPsfDeterminer Cannot import lsst.meas.extensions.photometryKron: disabling Kron measurements Cannot enable shapeHSM ( File "src/Utils.cc", line 42, in std::string lsst::utils::getPackageDir(const std::string&) Package meas_extensions_shapeHSM not found {0} lsst::pex::exceptions::NotFoundError: 'Package meas_extensions_shapeHSM not found' ): disabling HSM shape measurements : Loading config overrride file '/tigress/pprice/dm-4692/obs_subaru/config/hsc/processCcd.py' : input=/tigress/HSC/HSC : calib=None : output=/tigress/HSC/HSC/rerun/price/dm-4692 CameraMapper: Loading registry registry from /tigress/HSC/HSC/rerun/price/dm-4692/_parent/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /tigress/HSC/HSC/CALIB/calibRegistry.sqlite3 processCcd: Processing {'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1248, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 100, 'expTime': 270.0} processCcd.isr: Performing ISR on sensor {'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1248, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 100, 'expTime': 270.0} processCcd.isr: Applying linearity corrections to Ccd 100 processCcd.isr.crosstalk: Applying crosstalk correction processCcd.isr: Set 0 BAD pixels to 3147.74 processCcd.isr: Flattened sky level: 3847.800781 +/- 2114.507723 processCcd.isr: Measuring sky levels in 8x16 grids: 3884.324645 processCcd.isr: Sky flatness in 8x16 grids - pp: 15293.248379 rms: 1173.423587 processCcd.isr: Setting rough magnitude zero point: 34.678409 processCcd.charImage: Processing {'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1248, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 100, 'expTime': 270.0} processCcd.charImage.repair: Identified 6044 cosmic rays. processCcd.charImage.detectAndMeasure.detection: Detected 127 positive sources to 5 sigma. processCcd.charImage.detectAndMeasure.detection: Resubtracting the background after object detection processCcd.charImage.detectAndMeasure.measurement: Measuring 127 sources (127 parents, 0 children) processCcd.charImage.measurePsf: Measuring PSF /tigress/pprice/lsstsw/stack/Linux64/meas_algorithms/tickets.DM-4692-g3d073a93d7+1/python/lsst/meas/algorithms/objectSizeStarSelector.py:354: RuntimeWarning: invalid value encountered in less bad = numpy.logical_or(bad, width < self._widthMin) /tigress/pprice/lsstsw/stack/Linux64/meas_algorithms/tickets.DM-4692-g3d073a93d7+1/python/lsst/meas/algorithms/objectSizeStarSelector.py:355: RuntimeWarning: invalid value encountered in greater bad = numpy.logical_or(bad, width > self._widthMax) processCcd.charImage.measurePsf: PSF star selector found 6 candidates meas.algorithms.psfDeterminer WARNING: You only have 3 eigen images (you asked for 4): reducing number of eigen components meas.algorithms.psfDeterminer WARNING: You only have 1 eigen images (you asked for 3): reducing number of eigen components meas.algorithms.psfDeterminer WARNING: You only have 1 eigen images (you asked for 2): reducing number of eigen components python: /tigress/pprice/lsstsw/stack/Linux64/eigen/3.2.5/include/Eigen/src/Core/Redux.h:202: static Eigen::internal::redux_impl::Scalar Eigen::internal::redux_impl::run(const Derived&, const Func&) [with Func = Eigen::internal::scalar_max_op; Derived = Eigen::CwiseUnaryOp, const Eigen::Matrix >; Eigen::internal::redux_impl::Scalar = double]: Assertion `size && "you are using an empty matrix"' failed. Aborted {code} Note: * This occurred while testing DM-4692. The LOCAL pipe_tasks and obs_subaru are on that ticket branch. The LOCAL meas_base is for the fix from DM-5050. * One root cause of the bad PSF modeling may be bad rotations in the application of the calibs ([~lauren] is looking into that; don't know if there's a ticket number), but this should never happen regardless.
Story points: 1