Non-optimal grayscale conversion algorithm in torchvision?

username: Cepera
cooked: <p>It seems that torchvision Grayscale uses the following formula to convert RGB images to grayscale:<br>
L = R * 0.2989 + G * 0.5870 + B * 0.1140<br>
Some other packages like opencv use pretty similar color conversion values.</p>
<p>These ratios to merge RGB into one channel come from the BT.601 standard. And so far as I understand, this standard was created for television and is based on the two major concepts:</p>
<ol>
<li>Human eye feels some colors better than others. For example, we can see more colors in the green color spectrum than in the red one.</li>
<li>Different color representations in different TV systems. Specified RGB ratios are used in PAL/NTSC standard, but there are TV standards with the other ratios.</li>
</ol>
<p>Torchvision Grayscale is typically used in deep learning, where we take information from camera sensors (i.e. images) and process it to get non-visual results. And (surprise!) none of these is related neither to the human eye, nor to ancient TV standards.</p>
<p>Every time we convert digital camera images from RGB to grayscale, we lose some information. And the more distortion we introduce to conversion algorithms, the more original information we lose.<br>
It’s pretty obvious that applying any arbitrary coefficients while merging color channels leads to irreversible image data loss. And no doubt that standards that are classified as ‘perceptual luminance-preserving conversion’ are pretty arbitrary for most of the Machine Learning tasks.<br>
So why should these RGB ratios still be a thing in Torch? Why not just manage all the colors equally?</p>

username: ptrblck
cooked: <p>I think it makes sense to apply the commonly used transformation methods in <code>torchvision</code> even if their reference implementation is dated. You are however absolutely correct that this transformation might not be ideal and you could easily create your own <code>ToGrayscale</code> transform using your coefficients.</p>

username: KFrank
cooked: <p>Hi Cepera!</p>
<aside class="quote no-group" data-username="Cepera" data-post="1" data-topic="144229" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cepera/48/46698_2.png" class="avatar"> Cepera:</div>
<blockquote>
<p>And no doubt that standards that are classified as ‘perceptual luminance-preserving conversion’ are pretty arbitrary for most of the Machine Learning tasks.<br>
So why should these RGB ratios still be a thing in Torch? Why not just manage all the colors equally?</p>
</blockquote>
</aside>
<p>In addition to what <a class="mention" href="/u/ptrblck">@ptrblck</a> said, I would note that if you are concerned<br>
about the modest difference in loss of information between using a<br>
“perceptual” conversion of RGB to grayscale vs. simply averaging the<br>
three channels together, you should be much more concerned about<br>
the dramatically larger loss of information that occurs when you combine<br>
the three RGB channels into a single grayscale channel in the first place.</p>
<p>If you have RGB data to work with and such information loss is material,<br>
you should use a model that accepts three (RGB) channels for its input.<br>
If your model only accepts a single (grayscale) channel, it would be quite<br>
straightforward to modify the first layer of the model to accept three<br>
channels instead of only one.</p>
<p>You can look at this approach, roughly speaking, as letting your model<br>
learn (via the three-channel weights of the first layer) the specific RGB-<br>
to-grayscale conversion that is optimal for your particular problem (although<br>
the RGB conversion being learnt is potentially more general than just a<br>
simple weighted average).</p>
<p>On a more philosophical note, perhaps we shouldn’t be using RGB<br>
cameras to capture our data – perhaps instead we should use infrared /<br>
visible / ultraviolet cameras, or six-color cameras like some butterflies’<br>
eyes, or four-color cameras like the color vision some people have.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Cepera
cooked: <p>KFrank, my idea of using grayscale in my current project is that the images are from cctv cameras. Half of the time these cameras work in the nightmode which is grayscale only (IR). I need to detect events like people and opened doors on the images and most of these events happen in nightmode, i.e. are grayscaled. I get no efficacy improvements while using 3-color images, but the performance/memory issues only.</p>
<p>So yes, I get some loss of information on converting 3-channel image to grayscale, and I get some ‘loss’ on the fact that infrared-grayscaled do not fully correspond to daylight grayscaled. And I also get the loss on wrong implementation of the Grayscale function in torch.</p>
<p>The problem here is that most ppl in my situation understand first two sources of loss and try to deal with them somehow. BUT the 3rd source of loss from Grayscale function in torchvision isn’t obvious at all. It is not mentioned in the documentation, and there are no discussions about that in the internet, so it cannot be realized and corrected by the programmer.</p>
<p>That is why this situation looks like a bad software practice.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Cepera" data-post="4" data-topic="144229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cepera/48/46698_2.png" class="avatar"> Cepera:</div>
<blockquote>
<p>That is why this situation looks like a bad software practice.</p>
</blockquote>
</aside>
<p>I would claim the opposite, i.e. it’s a good software practice to stick to standards.<br>
I think more users would run into unexpected results when comparing the grayscale output of an RGB image using the <code>torchvision.transforms</code> vs. e.g. the <code>OpenCV</code> or <code>PIL</code> one and would expect to see the same (up to floating point precision) results.<br>
So far your idea sounds interesting but I haven’t seen any verification from you or in the literature of your claim the same scaling for each channel might be advantageous. Could you post literature references for some experiments comparing this transform?</p>

username: Cepera
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="144229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I would claim the opposite, i.e. it’s a good software practice to stick to standards.</p>
</blockquote>
</aside>
<p>I said that it is bad practice to not point in the documentation which one of the many known standards is used. Especially when the original data is distorted by the algorithm that has no relation to the currently used technology.<br>
When there is no detailed documentation for the function like grayscale, I would suppose the most naive way: all channels are dealt equally. That method is called ‘average grayscale method’ and it is often referred to as the default method for grayscale conversion.</p>
<p>It’s interesting that, unlike OpenCV and PIL framework, Torch doesn’t even use the formula defined by the BT.601 standard. It uses just something close to the standardized coefficients.<br>
0.299R + 0.587G + 0.114B	:original formula from BT.601, used by OpenCV and PIL<br>
0.2989R + 0.587G + 0.114B 	:used in torchvision<br>
Here is the link to the ITU - BT.601 standard so you can check: <a href="https://www.itu.int/rec/R-REC-BT.601-7-201103-I/en" class="inline-onebox" rel="noopener nofollow ugc">BT.601&nbsp;:&nbsp;Studio encoding parameters of digital television for standard 4:3 and wide screen 16:9 aspect ratios</a></p>
<p>Let me assume, that the lack of documentation is just based on the fact that when the Grayscale function was coded nobody could explain why exactly BT.601 grayscale conversion standard was chosen, and nobody had read the original standard specifications for correct numbers, - that was just copied from some other package.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="144229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I haven’t seen any verification from you or in the literature of your claim the same scaling for each channel might be advantageous.</p>
</blockquote>
</aside>
<p>For me it’s the opposite: Using the NTSC standard needs to be justified in machine learning frameworks. My first post was about why using NTSC grayscaling here has just no plausible reasons: When copying functions from visual image processing frameworks, we should understand that they often change the original data for the sake of visual representation. Which is exactly the case with the color standard BT.601 as the other standards that are classified as perceptual and human-eye-based.<br>
Wikipedia: Grayscale#Colorimetric_(perceptual_luminance-preserving)_conversion_to_grayscale</p>
<p>Pretty sure there could be practical explanations why BT.601 isn’t good in ML. Here is the paper on face detection where they show equal color distribution on general dataset.<br>
On Conversion from Color to Gray-scale Images for Face Detection</p><aside class="onebox pdf" data-onebox-src="https://projet.liris.cnrs.fr/imagine/pub/proceedings/CVPR-2009-W/data/papers/09/08.pdf">
  <header class="source">

      <a href="https://projet.liris.cnrs.fr/imagine/pub/proceedings/CVPR-2009-W/data/papers/09/08.pdf" target="_blank" rel="noopener nofollow ugc">projet.liris.cnrs.fr</a>
  </header>

  <article class="onebox-body">
    <a href="https://projet.liris.cnrs.fr/imagine/pub/proceedings/CVPR-2009-W/data/papers/09/08.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://projet.liris.cnrs.fr/imagine/pub/proceedings/CVPR-2009-W/data/papers/09/08.pdf" target="_blank" rel="noopener nofollow ugc">08.pdf</a></h3>

  <p class="filesize">291.81 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><em>“In comparison with this, the nonskin-color signal is almost distributed with equal strength in the R, G, B channels. Thus, we have reason to believe that for face detection, it is inappropriate to use the weighting parameters of Eq.3 [BT.601], where the signal in the most important channel (red) is significantly suppressed, while those in the other two channels of minor importance are enhanced too much.”</em></p>
<p>But for me, the general proof of managing all the color channels equally by default is pretty analytical: distorting color ratios of the original image would increase the deviation from the original image as measured by major estimators like mean squared error (MSE).<br>
However, exaggerating some color channels in grayscale could be productive, if the dataset represents these channels more than others. Which doesn’t seem to be the case of widely used datasets.</p>

username: Cepera
cooked: <aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="144229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>If you have RGB data to work with and such information loss is material,<br>
you should use a model that accepts three (RGB) channels for its input.</p>
</blockquote>
</aside>
<p>Btw, there is an interesting paper of guys who improved the performance of the object detection algorithm YOLOv3 by using fine tuned one-channel grayscale satellite images instead of  color ones:</p>
<p><strong>Grayscale Based Algorithm for Remote Sensing with Deep Learning</strong><br>
<em>The results show that the grayscale-based method can sense the target more accurately and effectively than the traditional YOLOv3 approach.</em><br>
<a href="https://arxiv.org/ftp/arxiv/papers/2110/2110.08493.pdf" class="onebox" target="_blank" rel="noopener nofollow ugc">https://arxiv.org/ftp/arxiv/papers/2110/2110.08493.pdf</a></p>

username: KFrank
cooked: <p>Hi Cepera!</p>
<aside class="quote no-group" data-username="Cepera" data-post="7" data-topic="144229" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cepera/48/46698_2.png" class="avatar"> Cepera:</div>
<blockquote>
<p>Btw, there is an interesting paper of guys who improved the performance of the object detection algorithm YOLOv3 by using fine tuned one-channel grayscale satellite images instead of  color ones:</p>
<p><strong>Grayscale Based Algorithm for Remote Sensing with Deep Learning</strong></p>
</blockquote>
</aside>
<p>Just to be clear about the claims in the paper you reference:</p>
<p>They take two separate RGB color-image datasets – call them “dataset<br>
A” and “dataset B” – that are distinctly different in character.  For their<br>
baseline, they combine dataset A and dataset B together into a single<br>
RGB dataset without modification or preprocessing, and train a network<br>
that takes three-channel RGB images as input.</p>
<p>For the “Grayscale Based Algorithm” they convert dataset A from RGB<br>
to single-channel grayscale using an ad hoc (not “learned”) custom<br>
grayscale weighting, call it grayscale-weighting A.  They separately<br>
convert dataset B into grayscale using a second ad hoc weighting,<br>
grayscale-weighting B.  These two grayscale weightings are chosen,<br>
in essence by hand, to “normalize” away much of the difference in<br>
character between dataset A and dataset B.  After this preprocessing<br>
step, they combine the two datasets into a single grayscale dataset.</p>
<p>They then use this combined, “normalized” dataset to train a network<br>
that takes single-channel, grayscale images as input (but that is otherwise<br>
identical to their baseline network).</p>
<p>They claim better results with the grayscale dataset and single-channel<br>
network.</p>
<p>I’ll leave it to others to judge whether this is a fair test of whether grayscale<br>
training offers benefits over RGB training.</p>
<p>Comment 1:  They use mean-average-precision as a performance metric<br>
and claim better (higher) mean-average precision for their grayscale<br>
training.  However they report a worse (higher) loss function for their<br>
grayscale training (but don’t comment on the apparent discrepancy).</p>
<p>Comment 2:  Neither of the custom grayscale weightings uses all the<br>
colors equally, as you seem to advocate in your original post.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Cepera
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="144229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you post literature references for some experiments comparing this transform?</p>
</blockquote>
</aside>
<p>This paper is a bit old (2011), but it has compared the most used grayscale algorithms in terms of image recognition. Sorry for the long quote, but it is exactly about our topic.<br>
They use the term ‘Intensity’ for the equal channel grayscaling method.<br>
Luminance and Luma - are two close standards bt.601 (used in torch) and bt.709</p>
<p><strong>Color-to-Grayscale: Does the Method Matter in Image Recognition?</strong><br>
Our objective was to determine if the method used to convert from color-to-grayscale matters, and we can definitively say that it does influence performance. For all datasets there was a significant gap between the top performing and worst performing methods. Our results indicate that the method used to convert to grayscale should be clearly described in all publications, which is not always the case in image recognition.</p>
<p>For object and face recognition, Gleam is almost always the top performer. For texture recognition, Luminance and Luminance are good choices. Although color descriptors are sometimes extracted in the HSV colorspace, our results suggest replacing Value with Gleam is advisable.</p>
<p>In general, we observed little benefit from using a method based on human brightness perception. The only potential exception was textures. Emulating the way humans perceive certain colors as brighter than others appears to be of limited benefit for grayscale image recognition. However, methods that incorporate a form of gamma correction (e.g., Lightness, Gleam, Luma, Luster, etc.) usually perform better than purely linear methods such as Intensity and Luminance.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0029740">
  <header class="source">
      <img src="https://journals.plos.org/resource/img/favicon.ico" class="site-icon" width="48" height="48">

      <a href="https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0029740" target="_blank" rel="noopener nofollow ugc">journals.plos.org</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://journals.plos.org/plosone/article/figure/image?id=10.1371/journal.pone.0029740.g005&amp;size=inline" class="thumbnail">

<h3><a href="https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0029740" target="_blank" rel="noopener nofollow ugc">Color-to-Grayscale: Does the Method Matter in Image Recognition?</a></h3>

  <p>In image recognition it is often assumed the method used to convert color images to grayscale has little impact on recognition performance. We compare thirteen different grayscale algorithms with four types of image descriptors and demonstrate that...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Cepera
cooked: <aside class="quote no-group" data-username="KFrank" data-post="8" data-topic="144229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>They use mean-average-precision as a performance metric and claim better (higher) mean-average precision for their grayscale training. However they report a worse (higher) loss function for their grayscale training (but don’t comment on the apparent discrepancy)</p>
</blockquote>
</aside>
<p>Don’t forget that they compare the loss for two different datasets (grayscaled and 3-color) that have different data structures.<br>
Stepping aside the comparison methodology, they did what they did: achieved high accuracy with the grayscaling, using some grayscale-specific fine tuning.</p>
<p>Not sure if you are interested, but there is the better quality paper that tries to explain why is it possible to get higher accuracy on grayscale:<br>
Bui, Hieu Minh, et al. “Using grayscale images for object recognition with convolutional-recursive neural network.”</p>
<p>And of course, there are papers that show that for models like ResNet color images are better:<br>
Funt, Brian, and Ligeng Zhu. “Does colour really matter? Evaluation via object classification.”</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Cepera" data-post="6" data-topic="144229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cepera/48/46698_2.png" class="avatar"> Cepera:</div>
<blockquote>
<p>I said that it is bad practice to not point in the documentation which one of the many known standards is used.</p>
</blockquote>
</aside>
<p>That’s a good point and I’m sure a contribution to fix the docs is more than welcome. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Thanks for the references as they are interesting to read!<br>
However, quotes like</p>
<aside class="quote no-group" data-username="Cepera" data-post="9" data-topic="144229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cepera/48/46698_2.png" class="avatar"> Cepera:</div>
<blockquote>
<p>In general, we observed little benefit from using a method based on human brightness perception.</p>
</blockquote>
</aside>
<p>are not particularly convincing to break the compatibility of <code>torchvision.transforms</code> with other image CV libs especially since you can use your custom transformation with a single line of code.</p>

username: Cepera
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="11" data-topic="144229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>are not particularly convincing</p>
</blockquote>
</aside>
<p>Ok. So, having the worst accuracy (12/13 or 13/13) in the most of the datasets (3/4) for the grayscale formula used in torchvision you consider as ‘not convincing’. What would you consider as convincing? Do you have any falsifiable thesis?</p>

username: ptrblck
cooked: <p>Right, as I don’t know how transferable “SIFT <a href="https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0029740#pone.0029740-Lowe1">[14]</a>, SURF <a href="https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0029740#pone.0029740-Bay1">[15]</a>, Geometric Blur <a href="https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0029740#pone.0029740-Berg1">[16]</a>, and Local Binary Patterns (LBP) <a href="https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0029740#pone.0029740-Ojala1">[17]</a>.” to ML/DL learning tasks are.<br>
I’m not trying to disproof the author’s findings, but if the answer to the “best” reduction weights is: “well, it depends on your data, algorithm, task, …” then I still think sticking to the standard approach used other libs would be the right decision to avoid potentially breaking other models.</p>
<p>Again, if your transformation needs other weights, use <code>transforms.Lambda</code> and apply your custom transformation (or write a custom transform class alternatively) .</p>