BUG: Bug in groupby returning non-consistent types when user function returns a None, (GH5992)

closes #5592