Repository: pandas-dev/pandas

Issue:
BUG: time grouper differs when passes as a list and as a scalar
#### Code Sample, a copy-pastable example if possible

```python
data_frame.groupby(pd.TimeGrouper('D', convention='start', key=time_column)).count()
data_frame.groupby([pd.TimeGrouper('D', convention='start', key=time_column)]).count()
```
give below two different outputs

```
            location  price  profit  type
time                                     
2017-07-23         1      1       1     1
2017-07-24         0      0       0     0
2017-07-25         1      1       1     1
2017-07-26         1      1       1     1
2017-07-27         3      3       3     3
2017-07-28         6      6       6     6
```
```
            location  price  profit  type
time                                     
2017-07-23         1      1       1     1
2017-07-25         1      1       1     1
2017-07-26         1      1       1     1
2017-07-27         3      3       3     3
2017-07-28         6      6       6     6
```

#### Problem description

The origin data frame doesn't contain a record in 2017-07-24, you can see if I use time grouper with convention='start' lonely, the 2017-07-24 item has been filled, but if by group list, that item hasn't been filled

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.20.3
pytest: None
pip: 9.0.1
setuptools: 36.2.0
Cython: None
numpy: 1.13.1
scipy: 0.19.1
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
pandas_gbq: None
pandas_datareader: None


</details>



Discussion:
need a copy-pastable example
```python
df = pd.DataFrame(np.random.randn(3), index=pd.Series(['2017-08-09 13:32:23', '2017-08-11 23:23:15', '2017-08-11 22:23:15'], dtype='datetime64[ns]'))

grouper = pd.TimeGrouper('D')
grouped = df.groupby(grouper)
data = grouped.count()
grouped = df.groupby([grouper])
data = grouped.count()

```

you can see the result is different

Sorry that convention parameter doesn't influence this scenario, but the result is what you can see, there is no way to resample and group by another categorical columns in the same time I think?
```python
    def _get_binner_for_grouping(self, obj):
        # return an ordering of the transformed group labels,
        # suitable for multi-grouping, e.g the labels for
        # the resampled intervals
        binner, grouper, obj = self._get_grouper(obj)

        l = []
        for key, group in grouper.get_iterator(self.ax):
            l.extend([key] * len(group))
        pdb.set_trace()
        if isinstance(self.ax, PeriodIndex):
            grouper = binner.__class__(l, freq=binner.freq, name=binner.name)
        else:
            # resampling causes duplicated values, specifying freq is invalid
            grouper = binner.__class__(l, name=binner.name)

        # since we may have had to sort
        # may need to reorder groups here
        if self.indexer is not None:
            indexer = self.indexer.argsort(kind='quicksort')
            grouper = grouper.take(indexer)
        return grouper
```

the second grouper generated by this method, while the first way generate grouper by
```python
binner, grouper, obj = self._get_grouper(obj)
```

it seems that the second way get the same grouper of TimeGrouper as the first way and then sample it once more, which only uses the existed time slices in group axis by
```python
l.extend([key] * len(group))
```
and then do some other operations
So what is the expected behavior? If follow the SQL rule, the missing time slices shouldn't be filled for TimeGrouper, only the resample method can get a complete time axis index. As the resample method call the TimeGrouper to implement the time axis index, may a new parameter needed to control TimeGrouper's behavior?

My target is to group by a categorical column and then resample to get a complete statistic in time dimension, I cannot do this in a groupby operation.
ok, this is a duplicate of #16746, but I like your example better, slightly formatted:

should all result the same [6] and [8] are correct, [7] is the bug.
```
In [5]: df = pd.DataFrame(np.random.randn(3), index=pd.Series(['2017-08-09 13:32:23', '2017-08-11 23:23:15', '2017-08-11 22:23:15'], dtype='datetime64[ns]'))
   ...: 
   ...: grouper = pd.TimeGrouper('D')
   ...: grouped1 = df.groupby(grouper)
   ...: data1 = grouped1.count()
   ...: grouped2 = df.groupby([grouper])
   ...: data2 = grouped2.count()
   ...: 

In [6]: data1
Out[6]: 
            0
2017-08-09  1
2017-08-10  0
2017-08-11  2

In [7]: data2
Out[7]: 
            0
2017-08-09  1
2017-08-11  2

In [8]: df.resample('D').count()
Out[8]: 
            0
2017-08-09  1
2017-08-10  0
2017-08-11  2

```

Should be straightforward to fix if you'd like to dig in.
@jreback I think 7 is also correct if follow the SQL rule, missed data can not be queried, maybe a new parameter needed? After hours debug, I think this situation is very complex and hard to fix, when by scalar TimeGrouper returns a BinGrouper, while by list returns a BaseGrouper that may have other grouping. I'll take some effort in this bug but can't promise to fix it out.
you are diving too deep. look more around [here]( https://github.com/pandas-dev/pandas/blob/master/pandas/core/groupby.py#L2656)

[7] is not a resample, rather a straight groupby by date

```
In [42]: df.groupby(df.index.normalize()).count()
Out[42]: 
            0
2017-08-09  1
2017-08-11  2
```

you could add this example to the groupby docs, but another parameter is not needed.
I think the situation is complex than your thought.
For list groupers, groupings are built for each grouper to make a BaseGrouper, and Grouping class only get binner by *_get_binner_for_grouping* of each grouper, the group_info of BaseGrouper property generated by the labels and group_index of its groupings, and each grouping get them by the **binner**, which is the label list of each row.

```python
# the labels and group_index of Grouping class, with them the BaseGrouper generate group_info
    @property
    def labels(self):
        if self._labels is None:
            self._make_labels()
        return self._labels

    @property
    def group_index(self):
        if self._group_index is None:
            self._make_labels()
        return self._group_index

    def _make_labels(self):
        if self._labels is None or self._group_index is None:
            pdb.set_trace()
            # self.grouper is the label of each row
            labels, uniques = algorithms.factorize(
                self.grouper, sort=self.sort)
            uniques = Index(uniques, name=self.name)
            self._labels = labels
            self._group_index = uniques
```
But for BinGrouper, the group_info property generated in different way, which directly map the labels and group_index.
```python
    def group_info(self):
        pdb.set_trace()
        ngroups = self.ngroups
        obs_group_ids = np.arange(ngroups)
        rep = np.diff(np.r_[0, self.bins])

        rep = _ensure_platform_int(rep)
        if ngroups == len(self.bins):
            comp_ids = np.repeat(np.arange(ngroups), rep)
        else:
            comp_ids = np.repeat(np.r_[-1, np.arange(ngroups)], rep)

        return comp_ids.astype('int64', copy=False), \
            obs_group_ids.astype('int64', copy=False), ngroups
```
So the bug occurs, the binner can only provide the labels of each row in selected axis, and then unique them to get unique labels and group_index, so the missed date time cannot be generated.

I think the way to fix this bug is to provide group_info in *_get_binner_for_grouping* of grouper to get more info? I think this logic change may be bigger? I'll take further look.
@ruiann again, you can simply have a len-1 list-like grouper case turn into the same case as the scalar passed case. its just going down the wrong path.
Then how about the result of ['categorical_axis', TimeGrouper('D', key='time_axis')]? It can not get all time slices for each category item.
In fact I think the
```python
    def _get_binner_for_grouping(self, obj):
        # return an ordering of the transformed group labels,
        # suitable for multi-grouping, e.g the labels for
        # the resampled intervals
        binner, grouper, obj = self._get_grouper(obj)

        l = []
        for key, group in grouper.get_iterator(self.ax):
            l.extend([key] * len(group))
        pdb.set_trace()
        if isinstance(self.ax, PeriodIndex):
            grouper = binner.__class__(l, freq=binner.freq, name=binner.name)
        else:
            # resampling causes duplicated values, specifying freq is invalid
            grouper = binner.__class__(l, name=binner.name)

        # since we may have had to sort
        # may need to reorder groups here
        if self.indexer is not None:
            indexer = self.indexer.argsort(kind='quicksort')
            grouper = grouper.take(indexer)
        return grouper
```
method of BaseGrouper (and BinGrouper) class works wrong, the group_info can be get from the BaseGrouper/BinGrouper instance generated by _get_grouper method, but not to read and sort them again item by item.

I'm trying to return the BaseGrouper/BinGrouper instance generated by _get_grouper method if the Grouping instance is given by Grouper instance, and get labels, group_index from the group_info, but there are still some bugs reported by test case, I'm now trying to fix them.

At last, I think groupby related logics need some refactor (I mean the name, alias, call stack sometimes confuse me, also some duplicated calls happened, maybe convert them all to list groupers could be better, with which only one implement need to deal).
> At last, I think groupby related logics need some refactor (I mean the name, alias, call stack sometimes confuse me, also some duplicated calls happened, maybe convert them all to list groupers could be better, with which only one implement need to deal).

If you want to attempt that, sure. Pls do so w/o adding any features / fixing any bugs, (though you can add xfailing tests). Try to make things pass everything that currently works, then in different commits / PR add things.

Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -256,11 +256,13 @@ def __init__(self, key=None, level=None, freq=None, axis=0, sort=False):
     def ax(self):
         return self.grouper
 
-    def _get_grouper(self, obj):
+    def _get_grouper(self, obj, validate=True):
         """
         Parameters
         ----------
         obj : the subject object
+        validate : boolean, default True
+            if True, validate the grouper
 
         Returns
         -------
@@ -271,7 +273,8 @@ def _get_grouper(self, obj):
         self.grouper, exclusions, self.obj = _get_grouper(self.obj, [self.key],
                                                           axis=self.axis,
                                                           level=self.level,
-                                                          sort=self.sort)
+                                                          sort=self.sort,
+                                                          validate=validate)
         return self.binner, self.grouper, self.obj
 
     def _set_grouper(self, obj, sort=False):
@@ -326,12 +329,6 @@ def _set_grouper(self, obj, sort=False):
         self.grouper = ax
         return self.grouper
 
-    def _get_binner_for_grouping(self, obj):
-        """ default to the standard binner here """
-        group_axis = obj._get_axis(self.axis)
-        return Grouping(group_axis, None, obj=obj, name=self.key,
-                        level=self.level, sort=self.sort, in_axis=False)
-
     @property
     def groups(self):
         return self.grouper.groups
@@ -1733,16 +1730,34 @@ class BaseGrouper(object):
     """
     This is an internal Grouper class, which actually holds
     the generated groups
+
+    Parameters
+    ----------
+    axis : int
+        the axis to group
+    groupings : array of grouping
+        all the grouping instances to handle in this grouper
+        for example for grouper list to groupby, need to pass the list
+    sort : boolean, default True
+        whether this grouper will give sorted result or not
+    group_keys : boolean, default True
+    mutated : boolean, default False
+    indexer : intp array, optional
+        the indexer created by Grouper
+        some groupers (TimeGrouper) will sort its axis and its
+        group_info is also sorted, so need the indexer to reorder
+
     """
 
     def __init__(self, axis, groupings, sort=True, group_keys=True,
-                 mutated=False):
+                 mutated=False, indexer=None):
         self._filter_empty_groups = self.compressed = len(groupings) != 1
         self.axis = axis
         self.groupings = groupings
         self.sort = sort
         self.group_keys = group_keys
         self.mutated = mutated
+        self.indexer = indexer
 
     @property
     def shape(self):
@@ -1888,6 +1903,15 @@ def group_info(self):
         comp_ids = _ensure_int64(comp_ids)
         return comp_ids, obs_group_ids, ngroups
 
+    @cache_readonly
+    def label_info(self):
+        # return the labels of items in original grouped axis
+        labels, _, _ = self.group_info
+        if self.indexer is not None:
+            sorter = np.lexsort((labels, self.indexer))
+            labels = labels[sorter]
+        return labels
+
     def _get_compressed_labels(self):
         all_labels = [ping.labels for ping in self.groupings]
         if len(all_labels) > 1:
@@ -2288,11 +2312,42 @@ def generate_bins_generic(values, binner, closed):
 
 class BinGrouper(BaseGrouper):
 
-    def __init__(self, bins, binlabels, filter_empty=False, mutated=False):
+    """
+    This is an internal Grouper class
+
+    Parameters
+    ----------
+    bins : the split index of binlabels to group the item of axis
+    binlabels : the label list
+    filter_empty : boolean, default False
+    mutated : boolean, default False
+    indexer : a intp array
+
+    Examples
+    --------
+    bins: [2, 4, 6, 8, 10]
+    binlabels: DatetimeIndex(['2005-01-01', '2005-01-03',
+        '2005-01-05', '2005-01-07', '2005-01-09'],
+        dtype='datetime64[ns]', freq='2D')
+
+    the group_info, which contains the label of each item in grouped
+    axis, the index of label in label list, group number, is
+
+    (array([0, 0, 1, 1, 2, 2, 3, 3, 4, 4]), array([0, 1, 2, 3, 4]), 5)
+
+    means that, the grouped axis has 10 items, can be grouped into 5
+    labels, the first and second items belong to the first label, the
+    third and forth items belong to the second label, and so on
+
+    """
+
+    def __init__(self, bins, binlabels, filter_empty=False, mutated=False,
+                 indexer=None):
         self.bins = _ensure_int64(bins)
         self.binlabels = _ensure_index(binlabels)
         self._filter_empty_groups = filter_empty
         self.mutated = mutated
+        self.indexer = indexer
 
     @cache_readonly
     def groups(self):
@@ -2460,6 +2515,19 @@ def __init__(self, index, grouper=None, obj=None, name=None, level=None,
             self.grouper, self._labels, self._group_index = \
                 index._get_grouper_for_level(self.grouper, level)
 
+        # a passed Grouper like, directly get the grouper in the same way
+        # as single grouper groupby, use the group_info to get labels
+        elif isinstance(self.grouper, Grouper):
+            # get the new grouper; we already have disambiguated
+            # what key/level refer to exactly, don't need to
+            # check again as we have by this point converted these
+            # to an actual value (rather than a pd.Grouper)
+            _, grouper, _ = self.grouper._get_grouper(self.obj, validate=False)
+            if self.name is None:
+                self.name = grouper.result_index.name
+            self.obj = self.grouper.obj
+            self.grouper = grouper
+
         else:
             if self.grouper is None and self.name is not None:
                 self.grouper = self.obj[self.name]
@@ -2482,16 +2550,6 @@ def __init__(self, index, grouper=None, obj=None, name=None, level=None,
                                            categories=c,
                                            ordered=self.grouper.ordered))
 
-            # a passed Grouper like
-            elif isinstance(self.grouper, Grouper):
-
-                # get the new grouper
-                grouper = self.grouper._get_binner_for_grouping(self.obj)
-                self.obj = self.grouper.obj
-                self.grouper = grouper
-                if self.name is None:
-                    self.name = grouper.name
-
             # we are done
             if isinstance(self.grouper, Grouping):
                 self.grouper = self.grouper.grouper
@@ -2536,6 +2594,10 @@ def ngroups(self):
 
     @cache_readonly
     def indices(self):
+        # we have a list of groupers
+        if isinstance(self.grouper, BaseGrouper):
+            return self.grouper.indices
+
         values = _ensure_categorical(self.grouper)
         return values._reverse_indexer()
 
@@ -2553,9 +2615,14 @@ def group_index(self):
 
     def _make_labels(self):
         if self._labels is None or self._group_index is None:
-            labels, uniques = algorithms.factorize(
-                self.grouper, sort=self.sort)
-            uniques = Index(uniques, name=self.name)
+            # we have a list of groupers
+            if isinstance(self.grouper, BaseGrouper):
+                labels = self.grouper.label_info
+                uniques = self.grouper.result_index
+            else:
+                labels, uniques = algorithms.factorize(
+                    self.grouper, sort=self.sort)
+                uniques = Index(uniques, name=self.name)
             self._labels = labels
             self._group_index = uniques
 
@@ -2566,7 +2633,7 @@ def groups(self):
 
 
 def _get_grouper(obj, key=None, axis=0, level=None, sort=True,
-                 mutated=False):
+                 mutated=False, validate=True):
     """
     create and return a BaseGrouper, which is an internal
     mapping of how to create the grouper indexers.
@@ -2583,6 +2650,8 @@ def _get_grouper(obj, key=None, axis=0, level=None, sort=True,
     are and then creates a Grouping for each one, combined into
     a BaseGrouper.
 
+    If validate, then check for key/level overlaps
+
     """
     group_axis = obj._get_axis(axis)
 
@@ -2707,7 +2776,7 @@ def is_in_obj(gpr):
 
         elif is_in_axis(gpr):  # df.groupby('name')
             if gpr in obj:
-                if gpr in obj.index.names:
+                if validate and gpr in obj.index.names:
                     warnings.warn(
                         ("'%s' is both a column name and an index level.\n"
                          "Defaulting to column but "
diff --git a/pandas/core/resample.py b/pandas/core/resample.py
--- a/pandas/core/resample.py
+++ b/pandas/core/resample.py
@@ -250,7 +250,7 @@ def _get_binner(self):
         """
 
         binner, bins, binlabels = self._get_binner_for_time()
-        bin_grouper = BinGrouper(bins, binlabels)
+        bin_grouper = BinGrouper(bins, binlabels, indexer=self.groupby.indexer)
         return binner, bin_grouper
 
     def _assure_grouper(self):
@@ -1105,35 +1105,12 @@ def _get_resampler(self, obj, kind=None):
                         "TimedeltaIndex or PeriodIndex, "
                         "but got an instance of %r" % type(ax).__name__)
 
-    def _get_grouper(self, obj):
+    def _get_grouper(self, obj, validate=True):
         # create the resampler and return our binner
         r = self._get_resampler(obj)
         r._set_binner()
         return r.binner, r.grouper, r.obj
 
-    def _get_binner_for_grouping(self, obj):
-        # return an ordering of the transformed group labels,
-        # suitable for multi-grouping, e.g the labels for
-        # the resampled intervals
-        binner, grouper, obj = self._get_grouper(obj)
-
-        l = []
-        for key, group in grouper.get_iterator(self.ax):
-            l.extend([key] * len(group))
-
-        if isinstance(self.ax, PeriodIndex):
-            grouper = binner.__class__(l, freq=binner.freq, name=binner.name)
-        else:
-            # resampling causes duplicated values, specifying freq is invalid
-            grouper = binner.__class__(l, name=binner.name)
-
-        # since we may have had to sort
-        # may need to reorder groups here
-        if self.indexer is not None:
-            indexer = self.indexer.argsort(kind='quicksort')
-            grouper = grouper.take(indexer)
-        return grouper
-
     def _get_time_bins(self, ax):
         if not isinstance(ax, DatetimeIndex):
             raise TypeError('axis must be a DatetimeIndex, but got '


Test patch:
diff --git a/pandas/tests/groupby/test_timegrouper.py b/pandas/tests/groupby/test_timegrouper.py
--- a/pandas/tests/groupby/test_timegrouper.py
+++ b/pandas/tests/groupby/test_timegrouper.py
@@ -623,3 +623,22 @@ def test_nunique_with_timegrouper_and_nat(self):
         result = test.groupby(grouper)['data'].nunique()
         expected = test[test.time.notnull()].groupby(grouper)['data'].nunique()
         tm.assert_series_equal(result, expected)
+
+    def test_scalar_call_versus_list_call(self):
+        # Issue: 17530
+        data_frame = {
+            'location': ['shanghai', 'beijing', 'shanghai'],
+            'time': pd.Series(['2017-08-09 13:32:23', '2017-08-11 23:23:15',
+                               '2017-08-11 22:23:15'],
+                              dtype='datetime64[ns]'),
+            'value': [1, 2, 3]
+        }
+        data_frame = pd.DataFrame(data_frame).set_index('time')
+        grouper = pd.Grouper(freq='D')
+
+        grouped = data_frame.groupby(grouper)
+        result = grouped.count()
+        grouped = data_frame.groupby([grouper])
+        expected = grouped.count()
+
+        assert_frame_equal(result, expected)