I guess it's too expensive because it would require to calculate many hash functions in the worst case. I haven't tried it, but the linked list could be replaced by a tree in order to limit the worst case. I a second element is added, it becomes the root of a tree.
Additional elements build the tree.
If an element is removed from the tree, the normal tree algorithms apply. If the first element is removed, and there are other elements in the tree, one tree element must be moved to become the new first element. This algorithm would combine the hash table, list, and tree data structures, so it would require some extra code to handle special cases.