Fix #423 IMap Deadlock

Refer to #423 for discussions. 

Thanks to @fantix and my coworker Wenliang Xiao.