Issue 1:
Code Sample

df = pd.DataFrame([{'i': i, 's': str(i)} for i in range(5)])

# this gives two rows with counts of one, as expected
df.iloc[:2].groupby('s').size()

df['s'] = df['s'].astype('category')
# this gives five rows, two of those having counts of one and others of zero
df.iloc[:2].groupby('s').size()

Problem description

I'm using categorical columns to save memory and possibly improve performance when having lots of strings, and would expect grouping by them to behave the same whether the string column is left as-is or converted to category. If that's expected behavior for categorical groupby to keep empty groups, is it possible to at least provide a boolean parameter to groupby like empty_groups? Or maybe even a simpler solution exists, but I couldn't find it.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None

python: 3.6.1.final.0

python-bits: 64

OS: Linux

OS-release: 4.9.16-gentoo

machine: x86_64

processor: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz

byteorder: little

LC_ALL: None

LANG: en_US.utf8

LOCALE: en_US.UTF-8

pandas: 0.20.2

pytest: 3.1.2

pip: 9.0.1

setuptools: 27.2.0

Cython: 0.25.2

numpy: 1.12.1

scipy: 0.19.1

xarray: None

IPython: 6.1.0

sphinx: 1.6.2

patsy: 0.4.1

dateutil: 2.6.0

pytz: 2017.2

blosc: None

bottleneck: 1.2.1

tables: 3.3.0

numexpr: 2.6.2

feather: None

matplotlib: 2.0.2

openpyxl: 2.4.7

xlrd: 1.0.0

xlwt: 1.2.0

xlsxwriter: 0.9.6

lxml: 3.8.0

bs4: 4.6.0

html5lib: 0.999

sqlalchemy: 1.1.11

pymysql: None

psycopg2: 2.7.1 (dt dec pq3 ext lo64)

jinja2: 2.9.6

s3fs: None

pandas_gbq: None

pandas_datareader: None

Issue 2:
Code Sample

pandas.DataFrame.duplicated(subset=None, keep='first')

Problem description

Pandas have a built-in method for denoting duplicated records. However, it is not able to denote duplicated record by some combination of keys.

dataFrame a

index1 index2 val1 val2
a y 1 1
a y 1 2
b z 2 2
c x 3 3

dataFrame b

index1 index2 val3
a y 1
b z 2

there are nodupkey for the first glance.

but this could cause issues when performing merging with validate='1:1' criteria.

pd.merge(a, b, validate='1:1')

we expect that both data set is unique by index1 and index2.

But in dataFrame a, the records are not unique which cause merging failed.

In SAS, duplicate in by group can be found by executing

proc sort data=sample nouniqueky out=dup;
by var1 var2;
run;

Expected Output

Expected add a parameter such as groupby or index

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None

python: 3.6.3.final.0

python-bits: 64

OS: Windows

OS-release: 7

machine: AMD64

processor: Intel64 Family 6 Model 69 Stepping 1, GenuineIntel

byteorder: little

LC_ALL: None

LANG: None

LOCALE: None.None

pandas: 0.21.1

pytest: 3.2.1

pip: 9.0.1

setuptools: 36.5.0.post20170921

Cython: 0.26.1

numpy: 1.13.3

scipy: 0.19.1

pyarrow: None

xarray: None

IPython: 6.1.0

sphinx: 1.6.3

patsy: 0.4.1

dateutil: 2.6.1

pytz: 2017.3

blosc: None

bottleneck: 1.2.1

tables: 3.4.2

numexpr: 2.6.2

feather: None

matplotlib: 2.1.0

openpyxl: 2.4.8

xlrd: 1.1.0

xlwt: 1.3.0

xlsxwriter: 1.0.2

lxml: 4.1.0

bs4: 4.6.0

html5lib: 0.999999999

sqlalchemy: 1.1.13

pymysql: None

psycopg2: None

jinja2: 2.9.6

s3fs: None

fastparquet: None

pandas_gbq: None

pandas_datareader: 0.5.0