Fix the issue that GroupBy may not call 'unsubscribe'

You can take a look at the original discussion in #1959