Use a multimap where we might have duplicate keys

Thanks to @jwpeterson for catching this!