Project: yara (virustotal/yara), language: c++
Vulnerability instance: yara.cve-2017-9438
Sanitizer: address

================= Bug Report (1/1) ==================
## Source: GitHub Issue
## URL: https://github.com/VirusTotal/yara/issues/674
## Description:
Issue: VirusTotal/yara#674
Title: Stack overflow in _yr_re_emit()
State: closed
Created by: fumfel
Created at: 2017-05-29 05:56:00+00:00

Issue Body:
**Stack overflow in _yr_re_emit()**

Tested on Git HEAD: cdbacf53a4ddac2bf1bc2f4bbe93fbe0a06bfff7

[Payload](https://github.com/VirusTotal/yara/files/1034925/yara_so_yr_re_emit.txt)

To reproduce: `yara yara_so_yr_re_emit.yar strings`

ASAN:
```
==29878==ERROR: AddressSanitizer: stack-overflow on address 0x7ffc175c7d7c (pc 0x00000045d4bb bp 0x7ffc175c9190 sp 0x7ffc175c7d70 T0)
    #0 0x45d4ba in _yr_re_emit XYZ/yara/libyara/re.c:754
    #1 0x45ecb9 in _yr_re_emit XYZ/yara/libyara/re.c:931
    ---------------------- !SNIP! ----------------------
    #246 0x45ecb9 in _yr_re_emit XYZ/yara/libyara/re.c:931
    #247 0x45ecb9 in _yr_re_emit XYZ/yara/libyara/re.c:931
    #248 0x45ecb9 in _yr_re_emit XYZ/yara/libyara/re.c:931
    #249 0x45ecb9 in _yr_re_emit XYZ/yara/libyara/re.c:931
    #250 0x45ecb9 in _yr_re_emit XYZ/yara/libyara/re.c:931
    #251 0x45ecb9 in _yr_re_emit XYZ/yara/libyara/re.c:931

SUMMARY: AddressSanitizer: stack-overflow XYZ/yara/libyara/re.c:754 _yr_re_emit
==29878==ABORTING
```




Comments:
Comment by fumfel on 2017-06-01 06:39:07+00:00:
Slightly different issue in 5e2d2798048dbba3aa8428aa563c0ebfbae37027

[Payload (about 78 kb)](https://github.com/VirusTotal/yara/files/1043847/yara_so_yr_re_emit2.txt)

ASAN:

```
==28982==ERROR: AddressSanitizer: stack-overflow on address 0x7fff4a83c8ec (pc 0x00000045c45b bp 0x7fff4a83dd00 sp 0x7fff4a83c8e0 T0)
    #0 0x45c45a in _yr_re_emit XYZ/yara/libyara/re.c:738
    #1 0x45dba9 in _yr_re_emit XYZ/yara/libyara/re.c:915
    ---------------------- !SNIP! ----------------------
    #246 0x45dba9 in _yr_re_emit XYZ/yara/libyara/re.c:915
    #247 0x45dba9 in _yr_re_emit XYZ/yara/libyara/re.c:915
    #248 0x45dba9 in _yr_re_emit XYZ/yara/libyara/re.c:915
    #249 0x45dba9 in _yr_re_emit XYZ/yara/libyara/re.c:915
    #250 0x45dba9 in _yr_re_emit XYZ/yara/libyara/re.c:915
    #251 0x45dba9 in _yr_re_emit XYZ/yara/libyara/re.c:915

SUMMARY: AddressSanitizer: stack-overflow XYZ/yara/libyara/re.c:738 _yr_re_emit
==28982==ABORTING
```




---

Comment by wxsBSD on 2017-06-06 16:04:11+00:00:
I just debugged an issue for a friend who had a bunch of rules break due to the limit of 2000. He has some rules which have very long hex strings in them, and bumping the value up works fine for him but I'm curious if you would consider bumping it up to 3000 (that more than covers the one rule I saw from him). I realize arbitrary limits are, by definition, arbitrary but I'm hoping we can bump this as to not break rules that exist.

---

Comment by plusvic on 2017-06-06 16:42:37+00:00:
Yes sure, as long as it avoids the stack overflow we should go for the largest number.

---

Comment by wxsBSD on 2017-06-06 17:28:13+00:00:
Just to make sure I could reproduce the problem I checked out ```cdbacf5``` and built with ```--enable-address-sanitizer --disable-optimization --enable-debug``` but I'm unable to reproduce it.

```
wxs@wxs-mbp yara % ./yara ~/Desktop/yara_so_yr_re_emit.txt /bin/ls
/Users/wxs/Desktop/yara_so_yr_re_emit.txt(8): error: syntax error, unexpected _IDENTIFIER_, expecting _CONDITION_
wxs@wxs-mbp yara %
```

My plan was once I knew how to trigger it I would keep find the largest value for ```RE_MAX_AST_LIMIT``` which wouldn't crash. Am I just not triggering this correctly?

---

Comment by plusvic on 2017-06-06 18:14:27+00:00:
@wxsBSD actually I had a similar problem and had to create my own rule to trigger the bug. I just used a very long regexp, like /xxxxxxxxx...{a few thousand more}.....xxxxxxxxxxxx/. Maybe @fumfel has a smaller stack size limit and the bug is triggered in his system but not in yours. Check your stack size limit with `ulimit -s`. In my case is 8192KB, @fumfel, what's the maximum stack size in your test system?

---

Comment by wxsBSD on 2017-06-06 18:33:33+00:00:
I forgot about ```ulimit -s```. I think it's probably best to leave it at 3000 for now and see if anyone complains, and if so we can make it runtime dependent or bump it again.

---

Comment by fumfel on 2017-06-07 07:59:54+00:00:
@plusvic 8192KB, but AFAIR ASAN for their purposes have different stack limits - 2/4 MB.

---

Comment by marnao on 2017-06-08 20:16:55+00:00:
We had many rules break with the 2000 limit. I had to go all the way up to 6000. Is this safe? Just potentially uses more resources? Not sure I understand the implications.

---

Comment by plusvic on 2017-06-08 21:14:04+00:00:
@marnao a 6000 limit should be safe too. In a 64 bits system my estimation is that each recursive call to _yr_re_emit() consumes around 180 bytes of stack space. So, setting the limit to 10000 will consume around 1.8MB of stack. With the typical 8MB stack space that's less than 25%.

However I wonder what kind of regular expressions are hitting the 2000 limit, I'm afraid that by raising the limit I could be encouraging massive regular expressions that don't actually make sense.

---

Comment by marnao on 2017-06-08 21:49:13+00:00:
@plusvic thanks for the quick response as always. That's very helpful context.

I think you have a legitimate concern about encouraging poor use cases. Some of our worst offending rules were externally sourced, i.e. we did not write them. I can't speak for these, but the signatures that we wrote that resulted in lengthy hex strings were either:

1) automatically generated signatures on PE artifacts
2) static constants used to identify a particular kind of compression

Hope this helps!

Attachments:
https://github.com/VirusTotal/yara/files/1034925/yara_so_yr_re_emit.txt
https://github.com/VirusTotal/yara/files/1043847/yara_so_yr_re_emit2.txt

Commit References:
cdbacf53a4ddac2bf1bc2f4bbe93fbe0a06bfff7
5e2d2798048dbba3aa8428aa563c0ebfbae37027

Build script:
#!/bin/bash -eu
# Minimized build script with only core build commands
set -eu
./bootstrap.sh
./configure --enable-macho --enable-debug --enable-dex --enable-dotnet --without-crypto
make clean -j$(nproc)
make -j$(nproc) all
make install -j$(nproc)

Dockerfile:
FROM hwiwonlee/secb.base:latest
RUN apt-get update && apt-get install -y automake autoconf make libtool pkg-config bison flex
RUN git clone https://github.com/virustotal/yara yara
RUN git -C yara checkout 5e2d2798048dbba3aa8428aa563c0ebfbae37027
WORKDIR $SRC/yara
COPY build.sh $SRC/

Additional files: