fix awkward sizeof, add more size_t casts

- dt-ify a dcraw-style sizeof in wavelet_denoise_xtrans()
- add (size_t) a few more places

Note that wavelet_denoise() has a similar dcraw-style alloc, but
haven't touched that so as to keep these patches x-trans specific.
Note also that wavelet_denoise() uses calloc instead of malloc to
quiet a clang scan-build warning. Perhaps the (size_t) cast will also
keep clang happy? No need to calloc() the buffer as the necessary
portion is memset to 0.