Better handling of comment joining

When joining is impossible due to comments, don't fall back to the
default mapping.