Properly handle streams with empty value downsamplers.