internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: angr/angr

Issue:
VariableRecovery fails with "AttributeError: value"
```
import angr
p = angr.Project('/tmp/a.out')
cfg = p.analyses.CFG()
vr = p.analyses.VariableRecovery(cfg.functions.function(name='main'))
```

for the following code

```
        .globl main
main:
        mov     (%rbx),%rsi
        mov     %rbx,%rdi
        vzeroupper
        callq   *0x8(%rsi)
        mov     %rbp,%rdi
        callq   _Unwind_Resume@plt
```

fails with

```
Traceback (most recent call last):
  File "angr-dev/blah.py", line 4, in <module>
    vr = p.analyses.VariableRecovery(cfg.functions.function(name='main'))
  File "angr-dev/angr/angr/analyses/analysis.py", line 108, in __call__
    oself.__init__(*args, **kwargs)
  File "angr-dev/angr/angr/analyses/variable_recovery/variable_recovery.py", line 416, in __init__
    self._analyze()
  File "angr-dev/angr/angr/analyses/forward_analysis.py", line 557, in _analyze
    self._analysis_core_graph()
  File "angr-dev/angr/angr/analyses/forward_analysis.py", line 580, in _analysis_core_graph
    changed, output_state = self._run_on_node(n, job_state)
  File "angr-dev/angr/angr/analyses/variable_recovery/variable_recovery.py", line 461, in _run_on_node
    concrete_state = state.get_concrete_state(node.addr)
  File "angr-dev/angr/angr/analyses/variable_recovery/variable_recovery.py", line 62, in get_concrete_state
    if s.ip._model_concrete.value == addr:
  File "angr-dev/claripy/claripy/ast/base.py", line 951, in __getattr__
    raise AttributeError(a)
AttributeError: value
```


Discussion:
can I have some compilation instructions for that snippet? trying to compile it with gcc does not reproduce your issue, as cfgaccurate is unable to find main.
I just compiled it with gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28).

Here is the resulting binary: [angr-1264.tar.gz](https://github.com/angr/angr/files/2494986/angr-1264.tar.gz).

Resolving patch:
diff --git a/angr/analyses/variable_recovery/variable_recovery.py b/angr/analyses/variable_recovery/variable_recovery.py
--- a/angr/analyses/variable_recovery/variable_recovery.py
+++ b/angr/analyses/variable_recovery/variable_recovery.py
@@ -483,7 +483,7 @@ def _run_on_node(self, node, state):
                                                      )
         output_states = successors.all_successors
 
-        state.concrete_states = output_states
+        state.concrete_states = [ state for state in output_states if not state.ip.symbolic ]
 
         self._node_to_state[node.addr] = state
 


Test patch:
Repository: angr/angr

Issue:
Cachetools broke their API
There's a new major version of cachetools (providing LRUCache), 3.0.0. This has caused everything to break. I have pinned our version to `cachetools<3` for the time being, but we should migrate.

My guess is that this is because we were using the `missing` argument to LRUCache (in claripy, specifically), and I am fairly sure the intended replacement is to [implement the `__missing__` method](https://cachetools.readthedocs.io/en/latest/#extending-cache-classes). Unsure if there are more implications, which is why this issue is open under angr instead of claripy.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -122,7 +122,7 @@ def run(self, *args):
     install_requires=[
         'ana',
         'sortedcontainers',
-        'cachetools<3',
+        'cachetools',
         'capstone>=3.0.5rc2',
         'cooldict',
         'dpkt',


Test patch:
Repository: angr/angr

Issue:
Crash in Function.local_runtime_values() when state.history.jumpkind is 'Ijk_SigTRAP'
There is no check on the jumpkind of the current block before calling [factory.successors](https://github.com/angr/angr/blob/master/angr/knowledge_plugins/functions/function.py#L362). 

If the state jumpkind is 'Ijk_SigTRAP', this leads to an uncatched `SimEngineFailure`.

In this case, block should probably be skipped or if analysis is desired, `factory.successors` jumpkind could be set to 'Ijk_Boring' I guess.
For now, I go with this patch mostly copypasted from [CFGEmulated](https://github.com/angr/angr/blob/master/angr/analyses/cfg/cfg_emulated.py#L1655)
```diff
diff --git a/angr/knowledge_plugins/functions/function.py b/angr/knowledge_plugins/functions/function.py
index 6664301..b15d370 100644
--- a/angr/knowledge_plugins/functions/function.py
+++ b/angr/knowledge_plugins/functions/function.py
@@ -355,6 +355,12 @@ class Function(object):
             # don't trace outside of the binary
             if not self._project.loader.main_object.contains_addr(state.solver.eval(state.ip)):
                 continue
+            # don't trace unreachable blocks
+            if state.history.jumpkind in {'Ijk_EmWarn', 'Ijk_NoDecode',
+                                          'Ijk_MapFail', 'Ijk_NoRedir',
+                                          'Ijk_SigTRAP', 'Ijk_SigSEGV',
+                                          'Ijk_ClientReq'}:
+                continue
 
             curr_ip = state.solver.eval(state.ip)
 ```




Discussion:
This looks like a good patch! submit a pull request!

Resolving patch:
diff --git a/angr/knowledge_plugins/functions/function.py b/angr/knowledge_plugins/functions/function.py
--- a/angr/knowledge_plugins/functions/function.py
+++ b/angr/knowledge_plugins/functions/function.py
@@ -355,6 +355,12 @@ def local_runtime_values(self):
             # don't trace outside of the binary
             if not self._project.loader.main_object.contains_addr(state.solver.eval(state.ip)):
                 continue
+            # don't trace unreachable blocks
+            if state.history.jumpkind in {'Ijk_EmWarn', 'Ijk_NoDecode',
+                                          'Ijk_MapFail', 'Ijk_NoRedir',
+                                          'Ijk_SigTRAP', 'Ijk_SigSEGV',
+                                          'Ijk_ClientReq'}:
+                continue
 
             curr_ip = state.solver.eval(state.ip)
 


Test patch:
Repository: angr/angr

Issue:
angr.errors.SimEngineError: No bytes in memory for block starting at 0x3000002.
When constructing CFG by CFGFast, encounter SimEngineError:

```
Traceback (most recent call last):
  File "cfg_fast.py", line 61, in <module>
    analyze(sys.argv[1])
  File "cfg_fast.py", line 30, in analyze
    cfg = proj.analyses.CFGFast()
  File "/home/wgh/angr-dev/angr/angr/analyses/analysis.py", line 108, in __call__
    oself.__init__(*args, **kwargs)
  File "/home/wgh/angr-dev/angr/angr/analyses/cfg/cfg_fast.py", line 1026, in __init__
    self._analyze()
  File "/home/wgh/angr-dev/angr/angr/analyses/forward_analysis.py", line 559, in _analyze
    self._post_analysis()
  File "/home/wgh/angr-dev/angr/angr/analyses/cfg/cfg_fast.py", line 1531, in _post_analysis
    self.make_functions()
  File "/home/wgh/angr-dev/angr/angr/analyses/cfg/cfg_base.py", line 1439, in make_functions
    blockaddr_to_function
  File "/home/wgh/angr-dev/angr/angr/analyses/cfg/cfg_base.py", line 1711, in _process_irrational_function_starts
    if block.vex.jumpkind not in ('Ijk_Boring', 'Ijk_InvalICache'):
  File "/home/wgh/angr-dev/angr/angr/block.py", line 152, in vex
    collect_data_refs=self._collect_data_refs,
  File "/home/wgh/angr-dev/angr/angr/engines/vex/engine.py", line 568, in lift
    raise SimEngineError("No bytes in memory for block starting at %#x." % addr)
angr.errors.SimEngineError: No bytes in memory for block starting at 0x3000002.

```
Python code:
```
#!/usr/bin/env python
import os
import sys
import angr
import claripy

def analyze(binary_path):
    proj = angr.Project(binary_path, use_sim_procedures=True, default_analysis_mode='symbolic', auto_load_libs=False)

    # CFG construction: fast
    cfg = proj.analyses.CFGFast()

    # VFG construction
    vfg = proj.analyses.VFG(cfg=cfg)

if __name__ == '__main__':
    analyze(sys.argv[1])
```
[gcc_base.zip](https://github.com/angr/angr/files/2678613/gcc_base.zip)


Discussion:

Resolving patch:
diff --git a/angr/analyses/cfg/cfg_base.py b/angr/analyses/cfg/cfg_base.py
--- a/angr/analyses/cfg/cfg_base.py
+++ b/angr/analyses/cfg/cfg_base.py
@@ -1693,7 +1693,8 @@ def _process_irrational_function_starts(self, functions, predetermined_function_
         :rtype:                             set
         """
 
-        addrs = sorted(functions.keys())
+        addrs = sorted(k for k in functions.keys()
+                       if not self.project.is_hooked(k) and not self.project.simos.is_syscall_addr(k))
         functions_to_remove = set()
         adjusted_cfgnodes = set()
 
@@ -1701,8 +1702,6 @@ def _process_irrational_function_starts(self, functions, predetermined_function_
 
             if addr_1 in predetermined_function_addrs:
                 continue
-            if self.project.is_hooked(addr_0) or self.project.is_hooked(addr_1):
-                continue
 
             func_0 = functions[addr_0]
 


Test patch:
Repository: angr/angr

Issue:
Confusing error with mismatched symbolic value sizes
**Describe the bug**
This is less of a bug report and more of a report on a confusing error message. Essentially, if you have a size mismatch when storing a symbolic value in memory, the warnings that it raises is different depending on how you store it, and in one case it is quite misleading. After speaking to Fish about it, it seemed worthwhile to open an issue.

**Environment Information**
Environment information is a bit weird as this occurred on an internal branch of angr, but angr is version 8.8.10.5. I don't really think the environment is the issue here regardless.

**To Reproduce**
From any state with any binary:
```
fresh_symval = clarity.BVS('somename', 3)
state.memory.store(some_addr,fresh_symval)
```
This will raise 
```
angr.storage.paged_memory | Nothing left of the memory object to store in Simpage
```
The issue here is that a 3 bit symbolic value has no business being stored in byte addressable memory. The warning is a bit misleading here, as it makes it appear to be some sort of paging error, when the issue is actually a size mismatch.

if you instead try:
```
state.mem.int32_t[some_addr] = fresh_symval
```
You get the more useful error message of
```
ValueError: size of expression is wrong for type
```
Which is what helped me originally diagnose the problem.
I think more informative errors in the first case would be very helpful, and that pointing out size mismatch is probably the best way to do this.

Thank you for your time



Discussion:

Resolving patch:
diff --git a/angr/storage/memory.py b/angr/storage/memory.py
--- a/angr/storage/memory.py
+++ b/angr/storage/memory.py
@@ -512,6 +512,11 @@ def store(self, addr, data, size=None, condition=None, add_constraints=None, end
         elif size_e is None:
             size_e = self.state.solver.BVV(data_e.size() // self.state.arch.byte_width, self.state.arch.bits)
 
+        if len(data_e) % self.state.arch.byte_width != 0:
+            raise SimMemoryError("Attempting to store non-byte data to memory")
+        if not size_e.symbolic and (len(data_e) < size_e*self.state.arch.byte_width).is_true():
+            raise SimMemoryError("Provided data is too short for this memory store")
+
         if inspect is True:
             if self.category == 'reg':
                 self.state._inspect(


Test patch:
Repository: angr/angr

Issue:
missing successor in CFG.get_all_successors(node)
**Describe the bug**
`CFG.get_all_successors(node)` is supposed to give all successors of a node(not only direct successors).
But in one of my experiment, it even misses a direct successor of the node.

**To Reproduce**
~~~
import angr

proj = angr.Project('/home/kylebot/src/angr-dev/binaries/tests/cgc/cfe_CADET_00003')
cfg = proj.analyses.CFGFast(normalize=True)

node = cfg.get_any_node(0x80485c1)
target = cfg.get_any_node(0x80485dc)

assert target in node.successors # sanity check. angr can pass this check.

all_successors = list(cfg.get_all_successors(node))
assert target in all_successors # should be True, but no
assert target.addr in [x.addr for x in all_successors] # should be True, but no

~~~


Discussion:
I checked the source code of this function, it directly uses networkx's API like this:
~~~
    def get_all_successors(self, basic_block):
        return list(networkx.dfs_successors(self._graph, basic_block))
~~~
Then I don't know what's wrong.
Should we trust `angr` or `networkx` in this case....

Resolving patch:
diff --git a/angr/analyses/cfg/cfg_base.py b/angr/analyses/cfg/cfg_base.py
--- a/angr/analyses/cfg/cfg_base.py
+++ b/angr/analyses/cfg/cfg_base.py
@@ -342,11 +342,18 @@ def get_all_predecessors(self, cfgnode):
         :return: A list of predecessors in the CFG
         :rtype: list
         """
-
-        return list(networkx.dfs_predecessors(self._graph, cfgnode))
-
-    def get_all_successors(self, basic_block):
-        return list(networkx.dfs_successors(self._graph, basic_block))
+        s = set()
+        for child, parent in networkx.dfs_predecessors(self._graph, cfgnode).items():
+            s.add(child)
+            s.add(parent)
+        return list(s)
+
+    def get_all_successors(self, cfgnode):
+        s = set()
+        for parent, children in networkx.dfs_successors(self._graph, cfgnode).items():
+            s.add(parent)
+            s = s.union(children)
+        return list(s)
 
     def get_node(self, block_id):
         """


Test patch:
Repository: angr/angr

Issue:
angr should not require futures
In [setup.py](https://github.com/angr/angr/blob/c2cf015f78bd060b263e80627f5962b3062e0ea7/setup.py#L145), a dependency on [futures](https://pypi.org/project/futures/) is declared.

However, `futures` is a backport to Python2 of the `concurrent.futures` standard library module available in Python3.

The only occurrences of `futures` at this point are in [exploration_techniques/threading.py](https://github.com/angr/angr/blob/8edb29f5f885f029d2e97fba470063c3d78f7832/angr/exploration_techniques/threading.py).

(Maybe) Relates to #1277 .


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -142,7 +142,6 @@ def run(self, *args):
         'capstone>=3.0.5rc2',
         'cooldict',
         'dpkt',
-        'futures; python_version == "2.7"',
         'mulpyplexer',
         'networkx>=2.0',
         'progressbar',


Test patch:
Repository: angr/angr

Issue:
Code block byte alignment is recognized as a function
Code block byte alignment is recognized as a function. This confused me. Is this the correct analysis?
![image](https://user-images.githubusercontent.com/11703999/64235613-4fbf6f00-cf2b-11e9-9d43-274cb2e9c198.png)

The following is the analysis result.
```
`(Pdb) self.project.kb.functions.function(0x400bcc)
<Function sub_400bcc (4197324)>

(Pdb) self.project.kb.functions.function(0x400bcc)._addr_to_block_node
{4197328: <BlockNode at 0x400bd0 (size 56)>, 4197324: <BlockNode at 0x400bcc (size 4)>}

(Pdb) self.project.kb.functions.function(0x400bd0)
<Function _ZN42CWE416_Use_After_Free__malloc_free_char_43L9badSourceERPc (4197328)>

(Pdb) self.project.kb.functions.function(0x400bd0)._addr_to_block_node
{4197328: <BlockNode at 0x400bd0 (size 56)>, 4197384: <BlockNode at 0x400c08 (size 52)>, 4197436: <BlockNode at 0x400c3c (size 48)>, 4197484: <BlockNode at 0x400c6c (size 32)>}`
```
And can a basic block belong to both functions at the same time?


Discussion:
This is a bug (and probably a regression because I fixed something similar in the past). Can you please post the binary so it's easier for me to download and fix? Thanks!
[binary.zip](https://github.com/angr/angr/files/3573420/binary.zip)

> And can a basic block belong to both functions at the same time?

To answer your second question:

```
ipdb> p list(cfg.functions.function(0x400bcc).blocks)
[<Block for 0x400bcc, 4 bytes>]
ipdb> p cfg.functions.function(0x400bcc)._addr_to_block_node
{4197324: <BlockNode at 0x400bcc (size 4)>, 4197328: <BlockNode at 0x400bd0 (size 56)>}
```

`Function._addr_to_block_node` is just a cache. If you want to enumerate all functions in this function, use `.blocks` or `.block_addrs_set` instead.
And to answer your original question:

```
ipdb> p cfg.functions[0x400bcc].alignment
True
```

Beginning from https://github.com/angr/angr/commit/a03c6a10c191ffbb0b361b26999759b706b5a94e, alignments are not removed from the CFG anymore. Instead they are marked as `alignment`. It's up to the user to decide what to do with them.

Resolving patch:
diff --git a/angr/knowledge_plugins/functions/function.py b/angr/knowledge_plugins/functions/function.py
--- a/angr/knowledge_plugins/functions/function.py
+++ b/angr/knowledge_plugins/functions/function.py
@@ -96,7 +96,9 @@ def __init__(self, function_manager, addr, name=None, syscall=None, is_simproced
         self.prepared_stack_variables = set()
         self.registers_read_afterwards = set()
 
-        self._addr_to_block_node = {}  # map addresses to nodes
+        self._addr_to_block_node = {}  # map addresses to nodes. it's a cache of blocks. if a block is removed from the
+                                       # function, it may not be removed from _addr_to_block_node. if you want to list
+                                       # all blocks of a function, access .blocks.
         self._block_sizes = {}  # map addresses to block sizes
         self._block_cache = {}  # a cache of real, hard data Block objects
         self._local_blocks = {}  # a dict of all blocks inside the function


Test patch:
Repository: angr/angr

Issue:
Strange successors of the return block of a function
I'm analysing a MIPS binary when facing the problem.

The problem exists in the funcition `do_ssc`.

In the following block which has a return statement
![image](https://user-images.githubusercontent.com/8875073/69912053-4fb60900-145f-11ea-9152-4026aab7f033.png)

When I run `node.successors` I got
```
In [103]: end.successors                                                                                              
Out[103]: 
[<CFGNode 0x40a7a8[28]>,
 <CFGNode do_ssc+0x12c [28]>,
 <CFGNode do_ssc+0x4c4 [28]>,
 <CFGNode do_ssc+0x45c [24]>,
 <CFGNode do_ssc+0x2a8 [24]>]
```
Their addresses are `0x40a7a8`, `0x40a33c`, `0x40a6d4` and `0x40a4b8` respectively.

I know the cfg of angr is interfunctional, however, only `0x40a7a8` is an caller of `do_ssc`.

May I know why other threes exist?





Discussion:
[ssi.zip](https://github.com/angr/angr/files/3908027/ssi.zip)
The tested binary.

Resolving patch:
diff --git a/angr/analyses/cfg/indirect_jump_resolvers/mips_elf_fast.py b/angr/analyses/cfg/indirect_jump_resolvers/mips_elf_fast.py
--- a/angr/analyses/cfg/indirect_jump_resolvers/mips_elf_fast.py
+++ b/angr/analyses/cfg/indirect_jump_resolvers/mips_elf_fast.py
@@ -97,7 +97,13 @@ def overwrite_tmp_value(state):
         simgr.run()
 
         if simgr.cut:
-            target = simgr.cut[0].addr
+            # pick the successor that is cut right after executing `addr`
+            try:
+                target_state = next(iter(cut for cut in simgr.cut if cut.history.addr == addr))
+            except StopIteration:
+                l.debug("Indirect jump at %#x cannot be resolved by %s.", addr, repr(self))
+                return False, [ ]
+            target = target_state.addr
 
             if self._is_target_valid(cfg, target):
                 l.debug("Indirect jump at %#x is resolved to target %#x.", addr, target)


Test patch:
Repository: angr/angr

Issue:
`Function.runtime_value()` is broken
**Describe the bug.** / **To Reproduce.**

```python
from angr.project import Project
project = Project('<A BINARY>', auto_load_libs=False) # get a binary containing functions
project.analyses.CFGFast()
function = project.kb.functions.function(name='main') # will fail with any function
function.runtime_values()
```

Leads to several errors:
  * There is no `_cfg` attribute for `FunctionManager`: easily corrected using `kb.cfgs['CFGFast'];
  * But then, `get_all_irsbs` is not an attribute of `CFGModel` (can't even find any occurences of it in the current codebase - although it was here before CFG was split into a model/view).
  * Maybe more, I did not go any further...

https://github.com/angr/angr/blob/0fdab305a121dbc7e0a723a71919676fe9428a61/angr/knowledge_plugins/functions/function.py#L537-L553


Discussion:
Do you need that function? I believe `runtime_values()` can be removed.
No, I don't think I need it.
`VariableRecovery*` seems to give me the information I think I would need ^^

Do you want me to submit the PR to remove it?

Resolving patch:
diff --git a/angr/knowledge_plugins/functions/function.py b/angr/knowledge_plugins/functions/function.py
--- a/angr/knowledge_plugins/functions/function.py
+++ b/angr/knowledge_plugins/functions/function.py
@@ -534,24 +534,6 @@ def local_runtime_values(self):
 
         return constants
 
-    @property
-    def runtime_values(self):
-        """
-        All of the concrete values used by this function at runtime (i.e., including passed-in arguments and global
-        values).
-        """
-        constants = set()
-        for b in self.block_addrs:
-            for sirsb in self._function_manager._cfg.get_all_irsbs(b):
-                for s in sirsb.successors + sirsb.unsat_successors:
-                    for a in s.history.recent_actions:
-                        for ao in a.all_objects:
-                            if not isinstance(ao.ast, claripy.ast.Base):
-                                constants.add(ao.ast)
-                            elif not ao.ast.symbolic:
-                                constants.add(s.solver.eval(ao.ast))
-        return constants
-
     @property
     def num_arguments(self):
         return len(self._argument_registers) + len(self._argument_stack_variables)


Test patch: