Fix bug where we would only give out colours that had previously been assigned via the hash