Issue 1:
With pandas v0.8.0b1

In [90]: u

Out[90]:

2012-05-08 01:45:00 86

2012-05-08 01:50:00 170

2012-05-08 01:55:00 130

2012-05-08 02:00:00 206

2012-05-08 02:05:00 52

2012-05-08 02:10:00 4

Freq: 5T

In [91]: u.append(u)

Traceback (most recent call last):

File "", line 1, in

File "C:\Python27\lib\site-packages\pandas\core\series.py", line 1566, in append

return concat(to_concat, ignore_index=False, verify_integrity=True)

File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 836, in concat

verify_integrity=verify_integrity)

File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 895, in init

self.new_axes = self._get_new_axes()

File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 1066, in _get_new_axes

concat_axis = self._get_concat_axis()

File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 1097, in _get_concat_axis

self._maybe_check_integrity(concat_axis)

File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 1106, in _maybe_check_integrity

% str(overlap))

Exception: Indexes have overlapping values: [1970-01-16 225:45:00, 1970-01-16 225:50:00, 1970-01-16 225:55:00, 1970-01-16 226:00:00, 1970-01-16 226:05:00, 1970-01-16 226:10:00]

Issue 2:
Code Sample, a copy-pastable example if possible

# Your code here
import pandas as pd
import numpy as np

def processCols(x):
x_dtype = str(x.dtype)
if x_dtype=='float64':
y = x.fillna(x.mean()) # replace nan in col with mean
return y
elif x_dtype=='category':
x_top = x.describe().top
y = x.fillna(str(x_top)) # replace nan in col with most occuring cat
return y
elif x_dtype=='int64': #do nothing for now
return x
else: # do nothing
return x

# Test snippet

#pid,typeid,bnum,bcat
#41,A,1.0,3D
#42,A,2.0,3D
#43,B,4.5,3E
#44,A,4.0,3D
#45,A,5.0,4D
#46,B,3.0,3E
#47,A,1.1,4D
#48,B,2.0,3F
#49,A,,
#50,A,,
#51,B,,
#52,B,,

rawData = pd.DataFrame({'pid' : np.array([41,42,43,44,45,46,47,48,49,50,51,52], dtype='int64'),
'typeid' : pd.Categorical(["A","A","B","A","A","B","A","B","A","A","B","B"]),
'bnum' : np.array([1,2,4.5,4,5,3,1.1,2,np.nan,np.nan,np.nan,np.nan],dtype='float64'),
'bcat' : pd.Categorical(["3D","3D","3E","3D","4D","3E","4D","3F",np.nan,np.nan,np.nan,np.nan])
})

groupedBy = rawData.groupby('typeid')

for name, group in groupedBy:
print('By Groups:')
print(name)
print('Before nan correction')
print(group)
transformed = group.transform(processCols)
print()
print('After nan correction')
print(transformed)
print()

# Now process all at once to compare
print()
print('transformed all at once')
transformed_all_at_once = groupedBy.transform(processCols)
print(transformed_all_at_once)

Problem description

When using groupby.transform and a custom function on a full data frame, the groupby key does not get returned in the final glued-together output. However, if this same process is done by iterating over the groups, the individual data frames do return the groupby key. If the groupby key is getting removed, then it requires an external join to reintroduce it. However, the individual replacements for the nan (which is the what the custom function does) is correct in both cases. Thanks.

[this should explain why the current behaviour is a problem and why the expected output is a better solution.]

Note: We receive a lot of issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates!

Note: Many problems can be resolved by simply upgrading pandas to the latest version. Before submitting, please check if that solution works for you. If possible, you may want to check if master addresses this issue, but that is not necessary.

Expected Output

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

INSTALLED VERSIONS

commit: None

python: 3.5.3.final.0

python-bits: 64

OS: Darwin

OS-release: 16.7.0

machine: x86_64

processor: i386

byteorder: little

LC_ALL: None

LANG: None

LOCALE: None.None

pandas: 0.20.3

pytest: None

pip: 9.0.1

setuptools: 36.0.1

Cython: None

numpy: 1.13.1

scipy: 0.19.1

xarray: None

IPython: 6.1.0

sphinx: 1.6.2

patsy: None

dateutil: 2.6.1

pytz: 2017.2

blosc: None

bottleneck: None

tables: None

numexpr: None

feather: None

matplotlib: 2.0.2

openpyxl: None

xlrd: None

xlwt: None

xlsxwriter: None

lxml: None

bs4: None

html5lib: 0.9999999

sqlalchemy: None

pymysql: None

psycopg2: None

jinja2: 2.9.6

s3fs: None

pandas_gbq: None

pandas_datareader: None