Repository: pwndbg/pwndbg

Issue:
Get typecheck errors down to zero
With https://github.com/python/typeshed/pull/9411, https://github.com/pwndbg/pwndbg/pull/1469, https://github.com/pwndbg/pwndbg/pull/1471, https://github.com/pwndbg/pwndbg/pull/1472, 
 and https://github.com/pwndbg/pwndbg/pull/1473, most of the code type checks properly and we're down to 13 errors at the moment. This issue is to track the remaining work that needs to be done to get this down to zero. To typecheck the code, run `mypy pwndbg` in the root of the repo.

### [Fixed] ~~`argparse.ArgumentError`~~

```
pwndbg/commands/hexdump.py: note: In function "address_or_module_name":
pwndbg/commands/hexdump.py:37: error: Missing positional argument "message" in call to "ArgumentError"  [call-arg]
                raise argparse.ArgumentError("Could not find pages for module %s" % module_name)
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pwndbg/commands/hexdump.py:37: error: Argument 1 to "ArgumentError" has incompatible type "str"; expected
"Optional[Action]"  [arg-type]
                raise argparse.ArgumentError("Could not find pages for module %s" % module_name)
                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
`argparse.ArgumentError` should take the message as the second option, not the first. The first argument should be an `Action`. Looking at some examples online, it seems like the correct way to do this is not to have a custom type parsing function, but to implement a class that extends `argparse.Action` and then if the argument is invalid, `self` can be put as the first argument of the error. I think this is probably the best solution.

### `pages.index(page_addr)`

```
pwndbg/gdblib/elf.py: note: In function "map_inner":
pwndbg/gdblib/elf.py:375: error: Argument 1 to "index" of "list" has incompatible type "int"; expected "Page"
[arg-type]
                    page = pages[pages.index(page_addr)]
                                             ^~~~~~~~~
```

While typing this up, I figured out how this code works and what the issue is. The implementation for `Page.__eq__` is:
```
def __eq__(self, other) -> bool:
    return self.vaddr == getattr(other, "vaddr", other)
```
So it's possible to use an `int` with `List[Page].index()` because the `getattr` will fail and the int will be compared directly to `self.vaddr`. ~The fix for this is probably to use a `Union`~. (EDIT: doesn't work)

### [Fixed] ~~gdb.Value.fetch_lazy() does not return a value~~

```
pwndbg/heap/ptmalloc.py: note: In member "thread_cache" of class "DebugSymsHeap":
pwndbg/heap/ptmalloc.py:1150: error: "fetch_lazy" of "Value" does not return a value  [func-returns-value]
                    _ = self._thread_cache["entries"].fetch_lazy()
                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

### Ignored Errors
This is currently what's being ignored, we should try to fix as much of this as possible:
```toml
[tool.mypy]
disable_error_code = [
    "misc",
    # No type stubs for capstone, unicorn, pwnlib, and elftools
    "import",
    # Lots of dynamic attribute access
    "attr-defined",
    # Issues with indexing Module
    "index",
    # https://github.com/python/mypy/issues/6232
    "assignment",
    # Issues with @property
    "operator",
]

[[tool.mypy.overrides]]
module = [
   # Capstone constants
   "pwndbg.disasm.*",
   "pwndbg.commands.nearpc",
   # Module fields
   "pwndbg.gdblib.typeinfo",
   "pwndbg.gdblib.elf",
   "pwndbg.auxv",
]
disable_error_code = ["name-defined"]
```

- [x] misc: various errors, needs more investigation
- [x] import: nothing we can do about this as we depend on libraries without type stubs
- [ ] attr-defined: this will probably be most difficult one to fix. We use "dynamic" attributes all over the place, and use attributes assigned directly to modules, so there's no proper type checking on these.
- [x] index: The main issue is that we sometimes treat a module like a dictionary and attempt to index some attribute inside of it. Not sure how to deal with this.
- [ ] assignment: I'd prefer to wait until https://github.com/python/mypy/issues/6232 is implemented to do anything about this, but that might take a while.
- [x] operator: The main issue is when we access some functions annotated with `@property`, the type is assumed to be a `Callable` instead of whatever that function returns. This issue only appears if `@property` is combined with other decorators, but I'm not sure why
- [ ] name-defined: Part of these issues are related to Capstone constants we've imported with `from capstone import *`. This could be fixed by https://github.com/pwndbg/pwndbg/pull/1262, but that hasn't been merged because we don't have proper testing in this area and I don't want to break anything. The other issue causing this errors is the fact that we're often assigning attributes directly to `sys.module[__name__]`, and so the typechecker thinks these fields are undefined


Discussion:

Resolving patch:
diff --git a/pwndbg/lib/memoize.py b/pwndbg/lib/memoize.py
--- a/pwndbg/lib/memoize.py
+++ b/pwndbg/lib/memoize.py
@@ -6,6 +6,7 @@
 
 import functools
 import sys
+import typing as t
 from collections.abc import Hashable
 from typing import Any
 from typing import Callable
@@ -14,179 +15,180 @@
 
 debug = False
 
+# https://stackoverflow.com/a/75013308/803801
+if t.TYPE_CHECKING:
+    F = t.TypeVar("F")
+    reset_on_stop: Callable[[F], F]
+    reset_on_prompt: Callable[[F], F]
+    reset_on_exit: Callable[[F], F]
+    reset_on_objfile: Callable[[F], F]
+    reset_on_start: Callable[[F], F]
+    reset_on_cont: Callable[[F], F]
+    reset_on_thread: Callable[[F], F]
+else:
+
+    class memoize:
+        """
+        Base memoization class. Do not use directly. Instead use one of classes defined below.
+        """
+
+        caching = True
+
+        def __init__(self, func: Callable) -> None:
+            self.func = func
+            self.cache: Dict[Any, Any] = {}
+            self.caches.append(self)  # must be provided by base class
+            functools.update_wrapper(self, func)
+
+        def __call__(self, *args: Any, **kwargs: Any) -> Any:
+            how = None
+
+            if not isinstance(args, Hashable):
+                print("Cannot memoize %r!", file=sys.stderr)
+                how = "Not memoizeable!"
+                value = self.func(*args)
+
+            if self.caching and args in self.cache:
+                how = "Cached"
+                value = self.cache[args]
+
+            else:
+                how = "Executed"
+                value = self.func(*args, **kwargs)
+                self.cache[args] = value
 
-class memoize:
-    """
-    Base memoization class. Do not use directly. Instead use one of classes defined below.
-    """
+                if isinstance(value, list):
+                    print("Should not cache mutable types! %r" % self.func.__name__)
 
-    caching = True
+            if debug:
+                print("%s: %s(%r)" % (how, self, args))
+                print(".... %r" % (value,))
+            return value
 
-    def __init__(self, func: Callable) -> None:
-        self.func = func
-        self.cache: Dict[Any, Any] = {}
-        self.caches.append(self)  # must be provided by base class
-        functools.update_wrapper(self, func)
+        def __repr__(self) -> str:
+            funcname = self.func.__module__ + "." + self.func.__name__
+            return "<%s-memoized function %s>" % (self.kind, funcname)
 
-    def __call__(self, *args: Any, **kwargs: Any) -> Any:
-        how = None
+        def __get__(self, obj, objtype: type) -> Callable:
+            return functools.partial(self.__call__, obj)
 
-        if not isinstance(args, Hashable):
-            print("Cannot memoize %r!", file=sys.stderr)
-            how = "Not memoizeable!"
-            value = self.func(*args)
+        def clear(self) -> None:
+            if debug:
+                print("Clearing %s %r" % (self, self.cache))
+            self.cache.clear()
 
-        if self.caching and args in self.cache:
-            how = "Cached"
-            value = self.cache[args]
+    class forever(memoize):
+        """
+        Memoizes forever - for a pwndbg session or until `_reset` is called explicitly.
+        """
+
+        caches = []  # type: List[forever]
+        kind = "forever"
 
-        else:
-            how = "Executed"
-            value = self.func(*args, **kwargs)
-            self.cache[args] = value
+        @staticmethod
+        def _reset() -> None:
+            for obj in forever.caches:
+                obj.cache.clear()
+
+    class reset_on_stop(memoize):
+        caches = []  # type: List[reset_on_stop]
+        kind = "stop"
+
+        @staticmethod
+        def __reset_on_stop() -> None:
+            for obj in reset_on_stop.caches:
+                obj.cache.clear()
+
+        _reset = __reset_on_stop
+
+    class reset_on_prompt(memoize):
+        caches = []  # type: List[reset_on_prompt]
+        kind = "prompt"
+
+        @staticmethod
+        def __reset_on_prompt() -> None:
+            for obj in reset_on_prompt.caches:
+                obj.cache.clear()
+
+        _reset = __reset_on_prompt
+
+    class reset_on_exit(memoize):
+        caches = []  # type: List[reset_on_exit]
+        kind = "exit"
+
+        @staticmethod
+        def __reset_on_exit() -> None:
+            for obj in reset_on_exit.caches:
+                obj.clear()
+
+        _reset = __reset_on_exit
+
+    class reset_on_objfile(memoize):
+        caches = []  # type: List[reset_on_objfile]
+        kind = "objfile"
 
-            if isinstance(value, list):
-                print("Should not cache mutable types! %r" % self.func.__name__)
+        @staticmethod
+        def __reset_on_objfile() -> None:
+            for obj in reset_on_objfile.caches:
+                obj.clear()
 
-        if debug:
-            print("%s: %s(%r)" % (how, self, args))
-            print(".... %r" % (value,))
-        return value
+        _reset = __reset_on_objfile
 
-    def __repr__(self) -> str:
-        funcname = self.func.__module__ + "." + self.func.__name__
-        return "<%s-memoized function %s>" % (self.kind, funcname)
+    class reset_on_start(memoize):
+        caches = []  # type: List[reset_on_start]
+        kind = "start"
 
-    def __get__(self, obj, objtype: type) -> Callable:
-        return functools.partial(self.__call__, obj)
+        @staticmethod
+        def __reset_on_start() -> None:
+            for obj in reset_on_start.caches:
+                obj.clear()
 
-    def clear(self) -> None:
-        if debug:
-            print("Clearing %s %r" % (self, self.cache))
-        self.cache.clear()
+        _reset = __reset_on_start
+
+    class reset_on_cont(memoize):
+        caches = []  # type: List[reset_on_cont]
+        kind = "cont"
+
+        @staticmethod
+        def __reset_on_cont() -> None:
+            for obj in reset_on_cont.caches:
+                obj.clear()
 
+        _reset = __reset_on_cont
 
-class forever(memoize):
-    """
-    Memoizes forever - for a pwndbg session or until `_reset` is called explicitly.
-    """
+    class reset_on_thread(memoize):
+        caches = []  # type: List[reset_on_thread]
+        kind = "thread"
 
-    caches = []  # type: List[forever]
-    kind = "forever"
+        @staticmethod
+        def __reset_on_thread() -> None:
+            for obj in reset_on_thread.caches:
+                obj.clear()
 
-    @staticmethod
-    def _reset() -> None:
-        for obj in forever.caches:
-            obj.cache.clear()
+        _reset = __reset_on_thread
 
+    class while_running(memoize):
+        caches = []  # type: List[while_running]
+        kind = "running"
+        caching = False
 
-class reset_on_stop(memoize):
-    caches = []  # type: List[reset_on_stop]
-    kind = "stop"
+        @staticmethod
+        def _start_caching() -> None:
+            while_running.caching = True
 
-    @staticmethod
-    def __reset_on_stop() -> None:
-        for obj in reset_on_stop.caches:
-            obj.cache.clear()
+        @staticmethod
+        def __reset_while_running() -> None:
+            for obj in while_running.caches:
+                obj.clear()
+            while_running.caching = False
 
-    _reset = __reset_on_stop
+        _reset = __reset_while_running
 
-
-class reset_on_prompt(memoize):
-    caches = []  # type: List[reset_on_prompt]
-    kind = "prompt"
-
-    @staticmethod
-    def __reset_on_prompt() -> None:
-        for obj in reset_on_prompt.caches:
-            obj.cache.clear()
-
-    _reset = __reset_on_prompt
-
-
-class reset_on_exit(memoize):
-    caches = []  # type: List[reset_on_exit]
-    kind = "exit"
-
-    @staticmethod
-    def __reset_on_exit() -> None:
-        for obj in reset_on_exit.caches:
-            obj.clear()
-
-    _reset = __reset_on_exit
-
-
-class reset_on_objfile(memoize):
-    caches = []  # type: List[reset_on_objfile]
-    kind = "objfile"
-
-    @staticmethod
-    def __reset_on_objfile() -> None:
-        for obj in reset_on_objfile.caches:
-            obj.clear()
-
-    _reset = __reset_on_objfile
-
-
-class reset_on_start(memoize):
-    caches = []  # type: List[reset_on_start]
-    kind = "start"
-
-    @staticmethod
-    def __reset_on_start() -> None:
-        for obj in reset_on_start.caches:
-            obj.clear()
-
-    _reset = __reset_on_start
-
-
-class reset_on_cont(memoize):
-    caches = []  # type: List[reset_on_cont]
-    kind = "cont"
-
-    @staticmethod
-    def __reset_on_cont() -> None:
-        for obj in reset_on_cont.caches:
-            obj.clear()
-
-    _reset = __reset_on_cont
-
-
-class reset_on_thread(memoize):
-    caches = []  # type: List[reset_on_thread]
-    kind = "thread"
-
-    @staticmethod
-    def __reset_on_thread() -> None:
-        for obj in reset_on_thread.caches:
-            obj.clear()
-
-    _reset = __reset_on_thread
-
-
-class while_running(memoize):
-    caches = []  # type: List[while_running]
-    kind = "running"
-    caching = False
-
-    @staticmethod
-    def _start_caching() -> None:
-        while_running.caching = True
-
-    @staticmethod
-    def __reset_while_running() -> None:
-        for obj in while_running.caches:
-            obj.clear()
-        while_running.caching = False
-
-    _reset = __reset_while_running
-
-
-def reset() -> None:
-    forever._reset()
-    reset_on_stop._reset()
-    reset_on_exit._reset()
-    reset_on_objfile._reset()
-    reset_on_start._reset()
-    reset_on_cont._reset()
-    while_running._reset()
+    def reset() -> None:
+        forever._reset()
+        reset_on_stop._reset()
+        reset_on_exit._reset()
+        reset_on_objfile._reset()
+        reset_on_start._reset()
+        reset_on_cont._reset()
+        while_running._reset()
diff --git a/pwndbg/lib/stdio.py b/pwndbg/lib/stdio.py
--- a/pwndbg/lib/stdio.py
+++ b/pwndbg/lib/stdio.py
@@ -4,11 +4,13 @@
 """
 
 import sys
-from typing import *  # noqa note: TextIO is not abaliable in low python version
+from typing import List
+from typing import TextIO
+from typing import Tuple
 
 
 class Stdio:
-    queue = []  # type: List[Tuple[TextIO, TextIO, TextIO]]
+    queue: List[Tuple[TextIO, TextIO, TextIO]] = []
 
     def __enter__(self, *a, **kw) -> None:
         self.queue.append((sys.stdin, sys.stdout, sys.stderr))


Test patch: