Repository: angr/angr

Issue:
CFGFast: NetworkXError: Node not in digraph
Hi,

thanks for fixing that last bug @ltfish , but there is still an Error in the way of my CFGFast on /bin/ls. :) 

```
In [1]: import angr                                                                                                                                                                                                                         
WARNING | 2018-07-03 14:13:20,872 | angr.analyses.disassembly_utils | Your version of capstone does not support MIPS instruction groups.                                                                                                    
                                                                                                                                                                                                                                            
In [2]: p = angr.Project("/bin/ls")                                                                                                                                                                                                         
WARNING | 2018-07-03 14:13:22,746 | cle.loader | The main binary is a position-independent executable. It is being loaded with a base address of 0x400000.                                                                                  
                                                                                                                                                                                                                                            
In [3]: cfg = p.analyses.CFGFast(show_progressbar=True)                                                                                                                                                                                     
WARNING | 2018-07-03 14:13:26,623 | angr.analyses.cfg.cfg_fast | "auto_load_libs" is enabled. With libraries loaded in project, CFGFast will cover libraries, which may take significantly more time than expected. You may reload the binary
 with "auto_load_libs" disabled, or specify "regions" to limit the scope of CFG recovery.                                                                                                                                                   
WARNING | 2018-07-03 14:13:33,234 | angr.engines.successors | Exit state has over 256 possible solutions. Likely unconstrained; skipping. <BV64 global_c000026_86_64{UNINITIALIZED}>
WARNING | 2018-07-03 14:13:34,434 | angr.analyses.cfg.cfg_fast | __libc_start_main is supposed to yield new exits, but it fails to do so.
WARNING | 2018-07-03 14:13:35,639 | angr.engines.successors | Exit state has over 256 possible solutions. Likely unconstrained; skipping. <BV64 global_c00003e_218_64{UNINITIALIZED}>
WARNING | 2018-07-03 14:13:36,330 | angr.engines.successors | Exit state has over 256 possible solutions. Likely unconstrained; skipping. <BV64 global_10_223_64{UNINITIALIZED}>
WARNING | 2018-07-03 14:13:36,766 | angr.engines.successors | Exit state has over 256 possible solutions. Likely unconstrained; skipping. <BV64 global_c000056_240_64{UNINITIALIZED}>
WARNING | 2018-07-03 14:13:37,091 | angr.engines.successors | Exit state has over 256 possible solutions. Likely unconstrained; skipping. <BV64 global_c000066_247_64{UNINITIALIZED}>
WARNING | 2018-07-03 14:13:40,723 | angr.engines.successors | Exit state has over 256 possible solutions. Likely unconstrained; skipping. <BV64 global_c000076_361_64{UNINITIALIZED}>
WARNING | 2018-07-03 14:13:40,804 | angr.engines.successors | Exit state has over 256 possible solutions. Likely unconstrained; skipping. <BV64 global_c000086_369_64{UNINITIALIZED}>
ERROR   | 2018-07-03 14:13:40,877 | angr.engines.vex.statements.dirty | Unsupported dirty helper amd64g_dirtyhelper_FSTENV
ERROR   | 2018-07-03 14:13:40,878 | angr.engines.vex.statements.dirty | Unsupported dirty helper amd64g_dirtyhelper_FLDENV
ERROR   | 2018-07-03 14:13:40,952 | angr.engines.vex.statements.dirty | Unsupported dirty helper amd64g_dirtyhelper_FSTENV
WARNING | 2018-07-03 14:13:42,606 | angr.engines.successors | Exit state has over 256 possible solutions. Likely unconstrained; skipping. <BV64 global_c00009e_438_64{UNINITIALIZED}>
WARNING | 2018-07-03 14:13:43,948 | angr.engines.successors | Exit state has over 256 possible solutions. Likely unconstrained; skipping. <BV64 global_c0000b6_468_64{UNINITIALIZED}>    

[...] # Many more unconstraineds


NetworkXError                             Traceback (most recent call last)
<ipython-input-3-718b8705bb6a> in <module>()
----> 1 cfg = p.analyses.CFGFast(show_progressbar=True)

/home/X/angr-dev/angr/angr/analyses/analysis.pyc in __call__(self, *args, **kwargs)
    106
    107         oself._show_progressbar = show_progressbar
--> 108         oself.__init__(*args, **kwargs)
    109         return oself
    110

/home/X/angr-dev/angr/angr/analyses/cfg/cfg_fast.pyc in __init__(self, binary, regions, pickle_intermediate_results, symbols, function_prologues, resolve_indirect_jumps, force_segment, force_complete_scan, indirect_jump_target_limit, collect_data_references, extra_cross_references, normalize, start_at_entry, function_starts, extra_memory_regions, data_type_guessing_handlers, arch_options, indirect_jump_resolvers, base_state, exclude_sparse_regions, skip_specific_regions, heuristic_plt_resolving, start, end, **extra_arch_options)
    907
    908         # Start working!
--> 909         self._analyze()
    910
    911     #

/home/X/angr-dev/angr/angr/analyses/forward_analysis.pyc in _analyze(self)
    557             self._analysis_core_graph()
    558
--> 559         self._post_analysis()
    560
    561     def _analysis_core_graph(self):

/home/X/angr-dev/angr/angr/analyses/cfg/cfg_fast.pyc in _post_analysis(self)
   1458
   1459         if self.project.arch.name in ('X86', 'AMD64', 'MIPS32'):
-> 1460             self._remove_redundant_overlapping_blocks()
   1461
   1462         if self._normalize:

/home/X/angr-dev/angr/angr/analyses/cfg/cfg_fast.pyc in _remove_redundant_overlapping_blocks(self)
   2737                             self._nodes_by_addr[b.addr].remove(b)
   2738
-> 2739                         self.graph.remove_node(b)
   2740
   2741                         if b.addr in all_functions:

/home/X/.virtualenvs/angr-pypy/site-packages/networkx/classes/digraph.pyc in remove_node(self, n)
    521             del self._node[n]
    522         except KeyError:  # NetworkXError if n not in self
--> 523             raise NetworkXError("The node %s is not in the digraph." % (n,))
    524         for u in nbrs:
    525             del self._pred[u][n]   # remove all edges n-u in digraph

NetworkXError: The node <CFGNode 0x213a0b0L[61]> is not in the digraph
```


Discussion:
I found this bug, too. Haven't got enough time to figure out what was causing it :/
Have this bug as well. Only thing I've figured out so far is that there exist node(s) in `nodes_to_append.values()` but not `self.graph.nodes()` by the time we get to cfg_fast.py:2739

Resolving patch:
diff --git a/angr/analyses/cfg/cfg_fast.py b/angr/analyses/cfg/cfg_fast.py
--- a/angr/analyses/cfg/cfg_fast.py
+++ b/angr/analyses/cfg/cfg_fast.py
@@ -2658,6 +2658,8 @@ def _remove_redundant_overlapping_blocks(self):
                                             thumb=a.thumb,
                                             byte_string=None if a.byte_string is None else a.byte_string[nop_length:],
                                             )
+                        self.graph.add_node(next_node)
+
                         # create edges accordingly
                         all_out_edges = self.graph.out_edges(a, data=True)
                         for _, dst, data in all_out_edges:


Test patch: