Bug: Stack smashing in nsSmtpProtocol.cpp.
Status: RESOLVED
Priority: --
Resolution: FIXED
Created: 2020-11-14 22:32:12+00:00
Resolved: 2021-11-23 07:34:24+00:00

This is a serious security bug. It is not a drill.

While I ran xpcshell tests with ASAN version of DEBUG C-C TB
binary, I found a serious problem.
There is a memory overwrite that smashes stack one way or the other.

Luckily the culprit is identified and  it is a single line of code.
But the security implication is serious.

```
Output of grep -i sanitizer: log1274-xpcshell.txt 

 0:08.69 pid:346406 ==346406==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffdb325bb40 at pc 0x7f8c8bad9daa bp 0x7ffdb325b770 sp 0x7ffdb325b768
 0:09.88 pid:346406 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 0:47.54 pid:346883 ==346883==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd70313e40 at pc 0x7f5c8e195daa bp 0x7ffd70313a70 sp 0x7ffd70313a68
 0:48.73 pid:346883 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 1:02.46 pid:347101 ==347101==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffcfb9efa80 at pc 0x7f3463413daa bp 0x7ffcfb9ef6b0 sp 0x7ffcfb9ef6a8
 1:03.62 pid:347101 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 1:04.67 pid:347128 ==347128==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffdec4320a0 at pc 0x7effbe12bdaa bp 0x7ffdec431cd0 sp 0x7ffdec431cc8
 1:05.83 pid:347128 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 1:08.26 pid:347177 ==347177==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd44e37f20 at pc 0x7f1f37688daa bp 0x7ffd44e37b50 sp 0x7ffd44e37b48
 1:09.45 pid:347177 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 1:10.50 pid:347203 ==347203==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffe09142780 at pc 0x7ff5e7023daa bp 0x7ffe091423b0 sp 0x7ffe091423a8
 1:11.70 pid:347203 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 1:12.75 pid:347229 ==347229==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffddf1f1f60 at pc 0x7f7076971daa bp 0x7ffddf1f1b90 sp 0x7ffddf1f1b88
 1:14.05 pid:347229 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 1:33.80 pid:347484 ==347484==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffcf66e0a40 at pc 0x7f2494993daa bp 0x7ffcf66e0670 sp 0x7ffcf66e0668
 1:34.98 pid:347484 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 2:16.33 pid:347965 ==347965==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffdbc966340 at pc 0x7fb7ef7badaa bp 0x7ffdbc965f70 sp 0x7ffdbc965f68
 2:17.50 pid:347965 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 2:32.50 pid:348179 ==348179==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffde9061480 at pc 0x7f2fe1807daa bp 0x7ffde90610b0 sp 0x7ffde90610a8
 2:33.70 pid:348179 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 2:34.84 pid:348206 ==348206==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff5d1317e0 at pc 0x7f339d7e3daa bp 0x7fff5d131410 sp 0x7fff5d131408
 2:36.03 pid:348206 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 2:38.70 pid:348254 ==348254==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffe44b93d20 at pc 0x7ff301338daa bp 0x7ffe44b93950 sp 0x7ffe44b93948
 2:39.87 pid:348254 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 2:41.03 pid:348280 ==348280==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd0075b2c0 at pc 0x7f981c7b4daa bp 0x7ffd0075aef0 sp 0x7ffd0075aee8
 2:42.20 pid:348280 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
 2:43.35 pid:348306 ==348306==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffd3723520 at pc 0x7f553b191daa bp 0x7fffd3723150 sp 0x7fffd3723148
 2:44.65 pid:348306 SUMMARY: AddressSanitizer: stack-buffer-overflow /NEW-SSD/NREF-COMM-CENTRAL/mozilla/nsprpub/pr/src/io/prscanf.c:288:51 in GetInt
```

I am attaching an excerpt from the xpcshelltes log with ASAN version of C-C TB with DEBUG flag.

Problem occurs here.
https://searchfox.org/comm-central/source/mailnews/compose/src/nsSmtpProtocol.cpp#563

```
     char codeClass, codeSubject, codeDetail;
---> if (PR_sscanf(line + chars_read, "%1u.%1u.%1u ", &codeClass, &codeSubject,
                  &codeDetail) == 3)
      m_responseCodeEnhanced = codeClass * 100 + codeSubject * 10 + codeDetail;
```
Note that |codeClass|, |codeSubject|, |codeDetail| are declared as |char| type.
However, the specifier has 'u' and not 'c'.
The intention seems to read a number to the varialbles.

This is the klux of the problem.

As I see it, PR_sscanf() processing of '%1u' is going through the hoops, but
eventually the scan function assumes that value's width is tagged as
(without specification such as 'h','l', 'll', 'L', etc.)
```
  state->sizeSpec = _PR_size_none; 
```
( https://searchfox.org/comm-central/source/mozilla/nsprpub/pr/src/io/prscanf.c#556 )

But then after the value is scanned and computed (for, %1u, that is %u), it is written back to the pointer of the variable that receives the value with the following statement.
This is in the call chain of Convert() -> GetInt().
https://searchfox.org/comm-central/source/mozilla/nsprpub/pr/src/io/prscanf.c#287

```
                unsigned long lval = strtoul(buf, NULL, base);

--->            if (state->sizeSpec == _PR_size_none) {
                    *va_arg(state->ap, PRUintn *) = lval;
```
PRUintn is unsigned int according to the following.:
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSPR/Reference/PRUintn
And it is indeed 4 bytes according to attached ASAN log.

So WE ARE WRITING 4 (FOUR) BYTES INTO CHARACTER VARIABLES ON THE STACK (!)
ASAN aborts the program on the first write, but all the three variables have memory overwrite issues.
Thus the code  stomps on whatever memory value out there on the stack. Grrr...

DoS possibility.

I think this could have been exploited by malicious SMTP server to cause at least a denial of service attack on some unfortunate TB binaries (this depends on how the auto variables are placed on the stack. The layout differs from a compiler to the other, and optimization-level and debug flag may affect the layout and code. So it is hard to say definitely.)

Serious Exploit?

Whether this could have been used to exploit unsuspecting user's computer, I am not sure. But again it is up to the particular compiler and the particular optimization and debug level, and CPU-depdendent... There may have been unfortunate TB binaries.

Still this is a very serious bug.

I don't know why my earlier valgrind run of xpcshell tests did not reveal this bug.  (Maybe, just maybe, the particular tests did not run under valgrind because of the timeout that are caused by x10-20 slowdown due to valgrind. I need to figure out how to extend xpctest timeout for valgrind run. At least, I should be able to reproduce the issue under valgrind with proper setting of timeout for the tests that triggered the issue reported here.)

Getting this fixed is the first priority, but I have a proposal.

I understand ASAN binary of TB is created for nightly these days.

Suggestion/Proposal

On the tryserver, we should run a mochitest and xpcshell test using the ASAN binary at least once a month for C-C, and/or as new nightly appears.
For running xpcshell tests, we should add "--serialize" and "--verbose" flag.
This is how I run my local memory tests of C-C TB.
Otherwise, the error messages may not be visible at all and the error messages get mixed up among different processes and hard to analyze. 

I would also suggest a monthly valgrind run.
But, I have not been able to run C-C TB  mochitest under valgrind at all since we switched to mochitest from |make mozmill| test suite. Tough.
Also, I have no idea why, but valgrind cannot run the big TB binary as ordinary user anymore on my Debian GNU/Linux OS. It seems that Valgrind+TB fails when the stack needs to be extended (i.e., new page needs to be mapped for the stack.).
Valgrind+C-C TB does run under superuser. However, mochitest has its own problem and I never get past the initial window or anything. I set the time out to more than 30 minutes, but eventually, the timeout is triggered before no tests are performed with mochitest.
|make mozmll| ran fine after my tweaking of timeout values...
xpcshelltests do run and I could find a few memory-related errors.