Only set a loaded channel to temp if it doesn't have an owner yet