Repository: pwndbg/pwndbg

Issue:
'Could not allocate dynamic translator buffer' Error when execute "start" command
<!--
Before reporting a new issue, make sure that we do not have any duplicates already open.
If there is one it might be good to take part in the discussion there.

Please make sure you have checked that the issue persists on LATEST pwndbg version.

Below is a template for BUG REPORTS.
Don't include it if this is a FEATURE REQUEST.
-->


### Description
'Could not allocate dynamic translator buffer' Error when execute \"start\" command

> pwndbg> start
> Temporary breakpoint 1 at 0x400400
> Temporary breakpoint 1, 0x0000000000400400 in _start ()
> Could not allocate dynamic translator buffer

<!--
Briefly describe the problem you are having in a few paragraphs.
-->

### Steps to reproduce

> gdb blah
> start

<!--
What do we have to do to reproduce the problem?
If this is connected to particular C/asm code, 
please provide the smallest C code that reproduces the issue.
-->

### My setup
OS - Oracle Cloud Instance
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=22.04
> DISTRIB_CODENAME=jammy
> DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
> PRETTY_NAME="Ubuntu 22.04.1 LTS"
> NAME="Ubuntu"
> VERSION_ID="22.04"
> VERSION="22.04.1 LTS (Jammy Jellyfish)"
> VERSION_CODENAME=jammy
> ID=ubuntu
> ID_LIKE=debian

In addition, same issues on Ubuntu 18.04

GDB
**GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1**

Python
**Python 3.10.6**

Commit
**commit 52a479211cf58d1bdc5b42fd50471f565e6a063c (HEAD -> dev, origin/dev, origin/HEAD)**

<!--
Show us your gdb/python/pwndbg/OS/IDA Pro version (depending on your case).

NOTE: We are currently supporting only Ubuntu installations.
It is known that pwndbg is not fully working e.g. on Arch Linux (the heap stuff is not working there).
If you would like to change this situation - help us improving pwndbg and supporting other distros!

This can be displayed in pwndbg through `version` command.

If it is somehow unavailable, use:
* `show version` - for gdb
* `py import sys; print(sys.version)` - for python
* pwndbg version/git commit id
-->



Discussion:
Hey, I've never seen such error and I doubt we can reproduce it.

Is that maybe a QEMU run VM run on Apple M1? I'm asking bcoz on https://github.com/rcore-os/rCore-Tutorial-v3/issues/7#issuecomment-787328981 they say:

> The error Could not allocate dynamic translator buffer comes from the imcompability of Apple M1 and qemu, see the discussions https://github.com/rcore-os/rCore-Tutorial-Book-v3/issues/11.
@disconnect3d
I'm using Oracle Cloud Virtual Instance, as i know this system built via QEMU
You can test it if you create oracle free tier account :P
Btw, on normal gdb (before install pwndbg) 'start' command works great
Can you try: `set emulate off` and then `start`? Does this work?

Querying https://grep.app/search?q=Could%20not%20allocate%20dynamic%20translator%20buffer finds that string in unicorn which is our dependency:

```cpp
static inline void code_gen_alloc(struct uc_struct *uc, size_t tb_size)
{
    TCGContext *tcg_ctx = uc->tcg_ctx;
    tcg_ctx->code_gen_buffer_size = size_code_gen_buffer(tb_size);
    tcg_ctx->code_gen_buffer = alloc_code_gen_buffer(uc);
    tcg_ctx->initial_buffer = tcg_ctx->code_gen_buffer;
    tcg_ctx->initial_buffer_size = tcg_ctx->code_gen_buffer_size;
    if (tcg_ctx->code_gen_buffer == NULL) {
        fprintf(stderr, "Could not allocate dynamic translator buffer\n");
        exit(1);
    }
}
```
(Source: https://github.com/unicorn-engine/unicorn/blob/56f3bdedb42d26bee1532cc01baf5eaf44a9aa23/qemu/accel/tcg/translate-all.c#L961)

 It seems that the library will call exit() on an error path? (what about a graceful error for library users, huh? :/).


@disconnect3d Oh, it works when i execute `set emulate off` before `start`
Cool, so this is indeed comming from Unicorn. Now, let's try to check out why.

If we look into the `code_gen_alloc` code, it prints the `"Could not allocate dynamic translator buffer"` error and exits if the `alloc_code_gen_buffer(uc)` call returns `NULL`. This call seems to be used to allocate the buffer to store the generated code and it has three different implementations: one for windows, one for static buffer and one is "all else". Let's look at the first and last one.

The [first one seems to do sth with RWX memory mappings, but it also aborts on failures](https://github.com/unicorn-engine/unicorn/blob/56f3bdedb42d26bee1532cc01baf5eaf44a9aa23/qemu/accel/tcg/translate-all.c#L833-L870):
```c
    if (qemu_mprotect_rwx(buf, size)) {
        abort();
    }
```


[The last one allocates a RWX memory page like this](https://github.com/unicorn-engine/unicorn/blob/56f3bdedb42d26bee1532cc01baf5eaf44a9aa23/qemu/accel/tcg/translate-all.c#L900-L907):
```c
    int prot = PROT_WRITE | PROT_READ | PROT_EXEC;
    int flags = MAP_PRIVATE | MAP_ANONYMOUS;
    size_t size = tcg_ctx->code_gen_buffer_size;
    void *buf;
#ifdef USE_MAP_JIT
    flags |= MAP_JIT;
#endif
    buf = mmap(NULL, size, prot, flags, -1, 0);
    if (buf == MAP_FAILED) {
        return NULL;
    }
```

Triggering an abort would not show us the error print. So it seems we are hitting the other implementation and that in your case the call to `mmap` with `RWX` protections is failing? It is either that you don't have enough (virtual) memory for the allocation -- which I doubt it would be the case -- or maybe your kernel/VM does not allow you to allocate RWX memory pages.

Can you somehow confirm this? One way to do it would be:
```
strace -e mmap,mmap2 gdb --quiet --batch --ex 'start' --ex 'quit' /bin/ls 2>&1
```

Or even:
```
strace -e mmap,mmap2 gdb --quiet --batch --ex 'start' --ex 'quit' /bin/ls 2>&1 | grep PROT_EXEC | grep PROT_WRITE | grep PROT_READ
```

And I bet there will be a call from Unicorn that fails.

In my case, where it works, it is:
```
$ strace -e mmap,mmap2 gdb --quiet --batch --ex 'start' --ex 'quit' /bin/ls 2>&1 | grep PROT_EXEC | grep PROT_WRITE
mmap(NULL, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1a368ac000
```

If this is the case that this call fails, it can be one of:
1) your kernel has a custom patch that disallows for WRITE+EXECUTE mappings
2) you are using SELinux or other Linux Security Module that disallows for write+execute if not specifically allowed
3) you don't have enough memory for the allocation, but given that it seems to be only 4kB, it's very unlikely
I have created an issue on the Unicorn repository for this case in https://github.com/unicorn-engine/unicorn/issues/1766
@disconnect3d 

> you don't have enough memory for the allocation, but given that it seems to be only 4kB, it's very unlikely

This is probably wrong because Unicorn (QEMU) by default requests 1GB of memory on all platforms. You might observer some other allocation other than the code buffer.

@iiwwnnaa 

> I'm using Oracle Cloud Virtual Instance, as i know this system built via QEMU

AFAIK, the free tier gets 1GB per x86 instance and you could try to allocate some swap spaces as a workaround.
@wtdcode Did you read my entire response? I wrote that the "not enough memory" case is unlikely and what is likely going on here is that the mmap fails to allocate a writable and executable page.

And that's something we likely can't fix anyhow on our side and we likely can't even handle this error anyhow as its aborting the whole GDB. Maybe we can handle/catch this abort somehow but idk if that makes much sense.

Perhaps we should try a W+X mmap ourselves and if it fails, disable emulate and warn the user we did it for this exact reason and that they can enable it explicitly in their config?
> @wtdcode Did you read my entire response? I wrote that the "not enough memory" case is unlikely and what is likely going on here is that the mmap fails to allocate a writable and executable page.

Unfortunately, "not enough memory" is exactly the case and I don't think Oracle Ubuntu enables W^X.

> And that's something we likely can't fix anyhow on our side and we likely can't even handle this error anyhow as its aborting the whole GDB. 

Yes you are right and I'm working on propagating this error to users, which is a bit harder than I thought before because QEMU never considers this case.
Also note that we backport QEMU W^X support for Apple M1 and BSD so W^X is probably not the root cause.
Oh okay, I kinda missed the point with 1GB alloc and that Oracle Cloud VM has 1GB memory. That makes sense now.

Maybe we should then attempt to mmap 1GB RWX and if it fails, disable unicorn with a warning "hey it may abort" :D 
I can perfectly reproduce this btw:

```
ubuntu@instance-20210917-1603 ~>
python3 -c "from unicorn import *; uc=Uc(UC_ARCH_X86, UC_MODE_64);uc.reg_write(0, 0)"
Could not allocate dynamic translator buffer
ubuntu@instance-20210917-1603 ~ [1]> free -h
              total        used        free      shared  buff/cache   available
Mem:          972Mi       376Mi       111Mi       2.0Mi       484Mi       439Mi
Swap:            0B          0B          0B
ubuntu@instance-20210917-1603 ~>
python3 -c "from mmap import *;mmap(-1, 4096, MAP_PRIVATE | MAP_ANON, PROT_WRITE | PROT_READ | PROT_EXEC)"
ubuntu@instance-20210917-1603 ~>
```

> Maybe we should then attempt to mmap 1GB RWX and if it fails, disable unicorn with a warning "hey it may abort" :D

That's cool and enables the max compatibility. I will try to get rid of 1GB requirement for the next release.
@wtdcode Btw what happens if u do allocate 1GB with that mmap? Does it throw an exception?

EDIT: nvm, it should raise `OSError: [Errno 12] Cannot allocate memory`

Resolving patch:
diff --git a/pwndbg/commands/context.py b/pwndbg/commands/context.py
--- a/pwndbg/commands/context.py
+++ b/pwndbg/commands/context.py
@@ -616,12 +616,18 @@ def context_disasm(target=sys.stdout, with_banner=True, width=None):
     if cs is not None and cs.syntax != syntax:
         pwndbg.lib.memoize.reset()
 
-    arch = pwndbg.gdblib.arch.current
-    emulate = bool(pwndbg.gdblib.config.emulate)
+    result = pwndbg.gdblib.nearpc.nearpc(
+        lines=code_lines // 2, emulate=bool(pwndbg.gdblib.config.emulate)
+    )
 
-    info = " / %s / set emulate %s" % (arch, "on" if emulate else "off")
+    # Note: we must fetch emulate value again after disasm since
+    # we check if we can actually use emulation in `can_run_first_emulate`
+    # and this call may disable it
+    info = " / %s / set emulate %s" % (
+        pwndbg.gdblib.arch.current,
+        "on" if bool(pwndbg.gdblib.config.emulate) else "off",
+    )
     banner = [pwndbg.ui.banner("disasm", target=target, width=width, extra=info)]
-    result = pwndbg.gdblib.nearpc.nearpc(lines=code_lines // 2, emulate=emulate)
 
     # If we didn't disassemble backward, try to make sure
     # that the amount of screen space taken is roughly constant.
diff --git a/pwndbg/disasm/__init__.py b/pwndbg/disasm/__init__.py
--- a/pwndbg/disasm/__init__.py
+++ b/pwndbg/disasm/__init__.py
@@ -18,6 +18,7 @@
 import pwndbg.gdblib.symbol
 import pwndbg.ida
 import pwndbg.lib.memoize
+from pwndbg.color import message
 
 try:
     import pwndbg.emu.emulator
@@ -208,6 +209,43 @@ def get(address, instructions=1):
 }
 
 
+def can_run_first_emulate():
+    """
+    Disable the emulate config variable if we don't have enough memory to use it
+    See https://github.com/pwndbg/pwndbg/issues/1534
+    And https://github.com/unicorn-engine/unicorn/pull/1743
+    """
+    global first_time_emulate
+    if not first_time_emulate:
+        return True
+    first_time_emulate = False
+
+    try:
+        from mmap import mmap, MAP_ANON, MAP_PRIVATE, PROT_EXEC, PROT_READ, PROT_WRITE  # isort:skip
+
+        mm = mmap(  # novm
+            -1, 1024 * 1024 * 1024, MAP_PRIVATE | MAP_ANON, PROT_WRITE | PROT_READ | PROT_EXEC
+        )
+        mm.close()
+    except OSError:
+        print(
+            message.error(
+                "Disabling the emulation via Unicorn Engine that is used for computing branches"
+                " as there isn't enough memory (1GB) to use it (since mmap(1G, RWX) failed). See also:\n"
+                "* https://github.com/pwndbg/pwndbg/issues/1534\n"
+                "* https://github.com/unicorn-engine/unicorn/pull/1743\n"
+                "Either free your memory or explicitly set `set emulate off` in your Pwndbg config"
+            )
+        )
+        gdb.execute("set emulate off", to_string=True)
+        return False
+
+    return True
+
+
+first_time_emulate = True
+
+
 def near(address, instructions=1, emulate=False, show_prev_insns=True):
     """
     Disasms instructions near given `address`. Passing `emulate` makes use of
@@ -246,7 +284,7 @@ def near(address, instructions=1, emulate=False, show_prev_insns=True):
     emu = None
 
     # If we hit the current instruction, we can do emulation going forward from there.
-    if address == pc and emulate:
+    if address == pc and emulate and (not first_time_emulate or can_run_first_emulate()):
         emu = pwndbg.emu.emulator.Emulator()
         # skip current line
         emu.single_step()


Test patch: