Project: datamanagement
Issue DM-6278: Investigate proper precision for afw::image::Image pixel transforms
The various pixel based transforms in {{afw/src/image/Image.cc}} were converted from using {{boost::lambda}} to C++11 lambda per DM-6091. At many places the previous implementation contained implicit casts (through {{boost::ret}}) of intermediate results to {{PixelT}} (e.g. {{float}}). In particular this affects opperations such as {{result = l + c*r}} where {{l}} is the left hand side image, {{r}} is the right hand side image and {{c}} a {{double}} constant. When calculated at double precision (e.g. without the casts, which are not needed with C++11 lambdas) the result is slightly different and this causes {{tests/testProcessCcd.py}} to fail on {{self.assertAlmostEqual(psfIyy, 2.17386182921239, places=7)}} which is only equal up to the fifth place. In order to not break existing behaviour I added explicit casts to {{PixelT}} for intermediate results. But this approach is questionable as the end result will be less accurate then possible. The aim of this ticket is to decide which approach is best: 1. Calculate at full precision and modify the test case. 2. Cast intermediate results to final precision (as it is done now). 3. Do something else?
Story points: 1