Remove inline fragment map

This simplifies inline.rs and makes things easier to understand

r? @pcwalton