Issue 1:
Hello,

I'm running into what I think is a bug in DataFrame.duplicated where it detects duplicates, but the data frame does not actually have any duplicated rows. It seems to only happen with integer columns, and somewhat large datasets (>600,000 rows).

I created a test data set to show the issue:

df = pd.read_table(
"https://www.dropbox.com/s/vkw8bzxp290jitz/test.tab?raw=1",
dtype = {"chrom" : "int64","pos" : "int64"}
)

If you ask for duplicates, it will detect them:

df.duplicated().any() # returns True

However, there are no duplicates:

In [5]: from collections import Counter

In [6]: counter = Counter(zip(df.chrom,df.pos))

In [7]: counter.most_common(5)
Out[7]:
[((0, 13704091), 1),
((0, 201539008), 1),
((0, 8573433), 1),
((0, 127434927), 1),
((0, 247829766), 1)]

If I convert one of the columns to float, and then ask for duplicates, it is correct:

df.loc[:,"pos"] = df.pos.astype("float64")
df.duplicated().any() # returns False

Strangely, converting the first column chrom to float or string does not seem to matter.

I had a difficult time in constructing this data frame to illustrate the example. It seems to only occur:

With at least two columns

One column (integer, string, float) that has a small number of unique values

One column (must be integer) that has a wide range of values, but in many instances, the values are close to each other

The dataset must be somewhat large, but it's hard to pin down. Somewhere on the order of > 500,000 rows, it seems.

From looking quickly at the DataFrame.duplicated code, it looks like it is using a hash table of some kind, and using integer columns differently than other columns - perhaps it's ending up with collisions?

Apologies if I'm missing something obvious here. Please let me know if I can be of any help in investigating further. My pandas version information is below.

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Darwin
OS-release: 14.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.17.0
nose: 1.3.7
pip: 7.1.2
setuptools: 18.4
Cython: 0.22.1
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 3.2.0
sphinx: 1.3.1
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.6
blosc: None
bottleneck: None
tables: 3.2.0
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: 2.0.2
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.7.3
lxml: 3.4.4
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None

Issue 2:
I appear to have some code that is sometimes (inexplicably) running in pure Python (i.e. not Cython-optimized). Specifically, the problem is with loc[] indexing on a boolean Series:

(Apologies for the redacted code and console output, but this project is not an open-source delivery)

For context: world.persons returns a DataFrame with approximately 8 million records long and 15 columns wide. It uses a MultiIndex with 2 levels.

This run() method is being called twice in the software sequence. The first time run() is called, persons.loc[filter_column] takes 15 minutes to execute, while the second time it only takes 5 seconds. What the heck is going on here?

Furthermore, this is NOT the first time in the program that world.persons is being indexed, so I am not sure how caching is being applied.

One other thing: prior to this loc[] call, my system starts to approach memory limits to the point that Windows begins paging.

My OS is Windows 7 x64, and I'm using Pandas version 0.17.0.