Revert "Fix Channel::GetModes, we can never have a negative mode set on a channel so this check is unnecessary"

This reverts commit 678d27fdb74a3c73b696b6feffc00523f2c233b2.
Really fix Channel::GetModes