Issue 1:
Problem description

The sparse design in Pandas is really very neat. However, I find the sparse documentation to be focused on syntax and structure, while not explaining practical applications of the sparse data structures.

Practical application should include operations on series and dataframes that produce innately sparse results. One such example is pivot (and pivot_table). Pivot induces sparsity by creating elements corresponding to the cartesian product of the new index and column values. It should be able to produce a sparse DataFrame as its output, either using automatic heuristics or by accepting some kind of sparse=bool parameter.

Code Sample

I would like to be able to do:

>>> df = pd.DataFrame({'x': ['a', 'a', 'c', 'a', 'b', 'b'], 'y': ['A', 'B', 'C', 'D', 'A', 'B'], 'val': [5, 6, 7, 8, 9, 10]}).pivot('x', 'y', 'val')
>>> table = df.pivot('x', 'y', 'val', sparse=True) # or sparse='auto'
>>> table.density
0.5

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None

python: 3.5.2.final.0

python-bits: 64

OS: Darwin

OS-release: 17.3.0

machine: x86_64

processor: i386

byteorder: little

LC_ALL: None

LANG: en_AU.UTF-8

LOCALE: en_AU.UTF-8

pandas: 0.20.1

pytest: 3.3.1

pip: 9.0.1

setuptools: 35.0.2

Cython: 0.24.1

numpy: 1.11.1

scipy: 0.18.1

xarray: None

IPython: 5.1.0

sphinx: 1.5

patsy: 0.4.1

dateutil: 2.5.3

pytz: 2016.6.1

blosc: None

bottleneck: 1.1.0

tables: 3.2.3.1

numexpr: 2.6.1

feather: None

matplotlib: 1.5.3

openpyxl: 1.6.2

xlrd: 1.0.0

xlwt: 1.1.2

xlsxwriter: 0.9.3

lxml: None

bs4: 4.5.1

html5lib: 0.999999999

sqlalchemy: 1.0.13

pymysql: None

psycopg2: None

jinja2: 2.8

s3fs: None

pandas_gbq: None

pandas_datareader: None

Issue 2:
Code Sample, a copy-pastable example if possible

I'm currently in the middle of upgrading old system from old pandas (0.12) to the new version (0.23.0). One of the parts of the system is duplicate columns detection in medium-sized DataFrames (~100 columns, few thousand rows). We were detecting it like this dupes = df.T.duplicated() and previously it worked but after the upgrade, it started failing. Simplest snippet to reproduce this locally

import numpy as np
import pandas as pd

data = {}
for i in range(70):
data['col_{0:02d}'.format(i)] = np.random.randint(0, 1000, 20000)
df = pd.DataFrame(data)
dupes = df.T.duplicated()
print dupes

Problem description

To the contrast of the note below, this issue isn't resolved by upgrading to the newest pandas. On the contrary, it is caused by such upgrade :) Old version I've copied below from 0.12 works on a snippet above

def old_duplicated(self, cols=None, take_last=False):
"""
Return boolean Series denoting duplicate rows, optionally only
considering certain columns

Parameters
----------
cols : column label or sequence of labels, optional
Only consider certain columns for identifying duplicates, by
default use all of the columns
take_last : boolean, default False
Take the last observed row in a row. Defaults to the first row

Returns
-------
duplicated : Series
"""

# kludge for #1833
def _m8_to_i8(x):
if issubclass(x.dtype.type, np.datetime64):
return x.view(np.int64)
return x

if cols is None:
values = list(_m8_to_i8(self.values.T))
else:
if np.iterable(cols) and not isinstance(cols, basestring):
if isinstance(cols, tuple):
if cols in self.columns:
values = [self[cols]]
else:
values = [_m8_to_i8(self[x].values) for x in cols]
else:
values = [_m8_to_i8(self[x].values) for x in cols]
else:
values = [self[cols]]

keys = lib.fast_zip_fillna(values)
duplicated = lib.duplicated(keys, take_last=take_last)
return pd.Series(duplicated, index=self.index)

but the new one now fails with

Traceback (most recent call last):
File "/home/modintsov/workspace/DataRobot/playground.py", line 56, in <module>
dupes = df.T.duplicated()
File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/core/frame.py", line 4384, in duplicated
ids = get_group_index(labels, shape, sort=False, xnull=False)
File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/core/sorting.py", line 95, in get_group_index
return loop(list(labels), list(shape))
File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/core/sorting.py", line 86, in loop
return loop(labels, shape)

... many-many lines of the same...

File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/core/sorting.py", line 60, in loop
stride = np.prod(shape[1:nlev], dtype='i8')
File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 2566, in prod
out=out, **kwargs)
RuntimeError: maximum recursion depth exceeded

Which is obviously a regression.

[this should explain why the current behaviour is a problem and why the expected output is a better solution.]

Note: We receive a lot of issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates!

Note: Many problems can be resolved by simply upgrading pandas to the latest version. Before submitting, please check if that solution works for you. If possible, you may want to check if master addresses this issue, but that is not necessary.

For documentation-related issues, you can check the latest versions of the docs on master here:

https://pandas-docs.github.io/pandas-docs-travis/

If the issue has not been resolved there, go ahead and file it in the issue tracker.

Expected Output

I expect no exception and return of bool Series. Example above in old pandas output this

col_00 False
col_01 False
col_02 False
col_03 False
col_04 False
col_05 False
col_06 False
col_07 False
col_08 False
col_09 False
col_10 False
col_11 False
col_12 False
col_13 False
col_14 False
...
col_55 False
col_56 False
col_57 False
col_58 False
col_59 False
col_60 False
col_61 False
col_62 False
col_63 False
col_64 False
col_65 False
col_66 False
col_67 False
col_68 False
col_69 False
Length: 70, dtype: bool

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

pd.show_versions()

INSTALLED VERSIONS

commit: None

python: 2.7.12.final.0

python-bits: 64

OS: Linux

OS-release: 4.4.0-128-generic

machine: x86_64

processor: x86_64

byteorder: little

LC_ALL: None

LANG: en_US.UTF-8

LOCALE: None.None

pandas: 0.23.0

pytest: 3.5.1

pip: 9.0.1

setuptools: 39.2.0

Cython: 0.21

numpy: 1.14.3

scipy: 1.1.0

pyarrow: None

xarray: None

IPython: 5.5.0

sphinx: 1.5.5

patsy: 0.2.1

dateutil: 2.7.3

pytz: 2015.7

blosc: None

bottleneck: None

tables: None

numexpr: 2.6.5

feather: None

matplotlib: None

openpyxl: None

xlrd: 0.9.2

xlwt: 0.7.5

xlsxwriter: None

lxml: None

bs4: 4.6.0

html5lib: None

sqlalchemy: 1.2.7

pymysql: None

psycopg2: 2.7.3.2.dr2 (dt dec pq3 ext lo64)

jinja2: 2.10

s3fs: None

fastparquet: None

pandas_gbq: None

pandas_datareader: None