BrandonFors/pandas-github-issues huggingface dataset:
BUG: hashtable memory error causes test_factorize_nan crash
State: closed

ObjectVector class resizes its array without reseting its capacity count, so subsequent appends are invalid.

Mac OS 10.9, Python 2.7.6, numpy 1.9.0.dev-ee49411. 

```
==57654== Invalid write of size 8
==57654==    at 0x137F856: __pyx_f_6pandas_9hashtable_12ObjectVector_append (in /Users/mtk/Projects/pandas/pandas/hashtable.so)
==57654==    by 0x139B16F: __pyx_pw_6pandas_9hashtable_17PyObjectHashTable_25get_labels (in /Users/mtk/Projects/pandas/pandas/hashtable.so)
==57654==    by 0x138CA9E: __pyx_pw_6pandas_9hashtable_10Factorizer_5factorize (in /Users/mtk/Projects/pandas/pandas/hashtable.so)
==57654==    by 0xD227E: PyEval_EvalFrameEx (in /usr/local/anaconda/lib/libpython2.7.dylib)

==57654==  Address 0x10095efd0 is 16 bytes inside a block of size 256 free'd
==57654==    at 0x7858: realloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==57654==    by 0x13C0F55: PyDataMem_RENEW (in /usr/local/anaconda/lib/python2.7/site-packages/numpy/core/multiarray.so)
==57654==    by 0x1488DE7: PyArray_Resize (in /usr/local/anaconda/lib/python2.7/site-packages/numpy/core/multiarray.so)
==57654==    by 0x14647A0: array_resize (in /usr/local/anaconda/lib/python2.7/site-packages/numpy/core/multiarray.so)
==57654==    by 0x1396A12: __pyx_pw_6pandas_9hashtable_12ObjectVector_5to_array (in /Users/mtk/Projects/pandas/pandas/hashtable.so)
==57654==    by 0x138CFEE: __pyx_pw_6pandas_9hashtable_10Factorizer_5factorize (in /Users/mtk/Projects/pandas/pandas/hashtable.so)
==57654==    by 0xD227E: PyEval_EvalFrameEx (in /usr/local/anaconda/lib/libpython2.7.dylib)
```


Labels:
name: Bug
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
can u construct a test that fails w/o the fix?

this doesn't fail in numpy 1.9 afaict (in 64-bit linux)

as we test this in Travis 

Added a test case that crashed on linux also.

gr8

can u add more dtypes to your test

eg loop thru object, int64, float 64

and release note (0.14.0)

thanks

Done. Also needed to handle the special case where a vector is resized to 0.

looks good
ping when green

thanks!
BUG/COMPAT: test compat on Python 2.6
State: closed

closes #7152


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
this is good 

these are not tested on 2.6 (except on windows)...that's how I discovered...

running on my 2.6 venv now ... weird that they would only run on windows i would think they run on the slow builds 

oh..sorry....what I meant was that we don't test slow on anything but 1 build on travis

looks good if it passes yours...squash and merge
BUG: allow dup indexing with Float64Index
State: closed

closes #7143


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@jreback should this be in 0.14.0? or not serious enough?

let's hold off on this

looks good....add a whatsnew (0.14.1) note and looks good
TST Fix AttributeError: 'iterator' object has no attribute 'next' on Python 3
State: closed

```
======================================================================
ERROR: test_pie_series (pandas.tests.test_graphics.TestSeriesPlots)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "X:\Python34\lib\site-packages\pandas\tests\test_graphics.py", line 484, in test_pie_series
    expected_texts = list(it.next() for it in itertools.cycle(iters))
  File "X:\Python34\lib\site-packages\pandas\tests\test_graphics.py", line 484, in <genexpr>
    expected_texts = list(it.next() for it in itertools.cycle(iters))
AttributeError: 'iterator' object has no attribute 'next'

----------------------------------------------------------------------
```


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Windows
description: Windows OS

Comments:
thanks!

turns out was not running slow tests on 3.4 on windows, duh!

FYI, announced 0.14.0 rc1: https://groups.google.com/forum/#!topic/pydata/Bx8oiSAZwOc

@cgohlke 

out of curiosity, how do you know when say pandas releases a rc1? (are you scanning the master repo for changes?) are you looking for say a RLS tag?

@jreback  I subscribe to https://github.com/pydata/pandas/releases.atom

all these neat features...cool.thxs

@cgohlke v0.14.0 tagged and released!

@cgohlke thanks for the windows builds!

Let me know if you want to distribute wheels built against older numpy versions (>= 1.6.2).

thanks

I think ok for now 

will let u know if I have requests
improving df.to_csv performance
State: closed

This is a WIP and discuss PR :construction: for #3186 
Adding small Writer class to have short-lived self.writer in CSVFomatter just for the header line. It is closed after the header was written.
Adding new lib.pyx function write_csv_rows_fast that manages file open/close on its own with self.path_or_buf from CSVFormmatter.
Current speed-up is factor 2.5 compared to before. Possibly the fprintf for each item is inefficient, but I thought as fprintf is buffered (in contrast to write) it can't be too bad. Any suggestions what I'm doing horribly wrong?


Labels:
name: Performance
description: Memory or execution speed performance
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO CSV
description: read_csv, to_csv

Comments:
closing for now. pls reopen if you want to update.
ENH/VIS: KDE plot can be cumulative
State: closed

I sometimes want cumulative density plot for distribution comparison. Added `cumulative` to KDE plot, like `matplotlib.pyplot.hist`.

```
import pandas as pd
import numpy as np

s = pd.Series(np.random.randn(100))
s.plot(kind='kde', cumulative=True)
```

![figure_1](https://cloud.githubusercontent.com/assets/1696302/2985394/e88ddcd2-dc3b-11e3-9f77-381c5cb981ec.png)


Labels:
name: Visualization
description: plotting

Comments:
@TomAugspurger didn't we have a discussion about not adding this once before? (as its in statsmodels)

Yep. It's implemented in `seaborn` now. See the very end of [this section](http://www.stanford.edu/~mwaskom/software/seaborn/tutorial/plotting_distributions.html#estimating-the-density-of-the-observations-kdeplot-and-rugplot). It basically wraps statsmodels' / scipy's kde function.

This was the issue: https://github.com/pydata/pandas/issues/2669

Basically decided it was out of scope for pandas, especially since there are so many great plotting alternatives being built.

closing as out-of-scope

Ah, OK. I thought it is kind of `stacked=True` operation though.
BUG/API: info repr should honor display.max_info_columns
State: closed

Closes https://github.com/pydata/pandas/issues/6939

I think this is what we agreed on. The tests are split up. In `test_format` we do more of the following global settings tests. The rest are in `test_frame` which directly call `df.info()` and I'm basically testing that the kwargs there override the global settings.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
I guess this only closes half of https://github.com/pydata/pandas/issues/6939, There's still the second bullet:

> we could add a keyword argument like show_counts (or another name) to specify if you want to show the non-null counts (to be able to override the max_info_rows option for a specific info call)

let's create a new issue for `show_counts` and put in 0.14.1

but this IS an API change yes? as `max_info_columns` took an integer before, and now is boolean/None ?

I didn't change anything with the config. The only API change was to `info` where now the `verbose` kwarg take None/True/False.

the `max_info_columns` is the same as before, but now `info` actually follows it (unless overridden). I do remember a discussion about what to do with `None` in this kind of setting, but I don't think it was for this issue.

@TomAugspurger sorry...you are right....was thinking about something else....ok...gr8 then....

merge when ready / green

(and I would separate out the `show_counts` option to another issue for later)

Ok. I'll open one up. The test failure was a mistake in the test. Should pass now.

excellent!
BUG: cleanup on describe
State: closed

use value_counts instead of `Counter` and make the tests unambiguous.

Came up in https://github.com/pydata/pandas/pull/7088#issuecomment-43034004


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
looks good on windows! thanks!

merge on green

thanks @TomAugspurger
SQL: add release notes for refactor (GH6292)
State: closed

This should (finally) close #6292.

WIP for now, but comments welcome. Have to add to release.rst.

@jreback Should I mark the new sql as experimental? (and if so, where should I do this?)


Labels:
name: Docs
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
that looks good to me

I don't think u need to mark as experimental
what/why do u want to do that?

everything should be back compat yes?

or is some part of the API still a bit experimental?

No, I think it should be OK. But I think you added that in the list of #6292? Only the OO api is not yet well defined, but this is also not publicly exposed in the docs or top-level namespace.

oh ok

just delete the experimental ref

can you merge this in (even if not 100% finished) before RC1? just want to make sure that whatsnew is pretty complete

yes, it can certainly be merged. Will rebase now

go for it

btw I currently have is released set to True (so it makes tags as rc1)

should I change this back to False

and then turn it back in when we actually release?

any idea ?

merge when ready

yep, can be merged, but then it is not included in 0.14rc1?

I am not familiar with the release process, so don't know how the setting of release to True/False works. That's something that maybe should be documented in the wiki.

I'll leave it for now; I have been updating the release checklist (was pretty good anyhow).

This will just update the docs (which the ann points to anyhow).

going to mail you the ANN, pls lmk any comments before I send out
BUG: tzinfo lost when concatenating multiindex arrays
State: closed

Closes #6606. 

The problem is caused by `MultiIndex.append`. Based on current master, `MultiIndex.append` works as below. The fix covers case1 and case2 which the result will be `MultiIndex`. 

The fix is applied to `concat`, and also `pivot_table` work as expected.

```
import pandas as pd

idx1 = pd.Index([1.1, 1.2, 1.3])
idx2 = pd.date_range('2011-01-01', freq='D', periods=3, tz='Asia/Tokyo')
idx3 = pd.Index(['A', 'B', 'C'])

midx_lv2 = pd.MultiIndex.from_arrays([idx1, idx2])
midx_lv3 = pd.MultiIndex.from_arrays([idx1, idx2, idx3])

#1 results in MultiIndex, which nlevels is 2
midx_lv2.append(midx_lv2)

#2 results in MultiIndex, which nlevels is 2, not 3. 3rd line will be disappeared.
midx_lv2.append(midx_lv3)

#3 results in tupled Index.
midx_lv2.append(idx1)

#4 results in tupled Index.
result = midx_lv3.append(midx_lv2)
```


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timezones
description: Timezone data dtype
name: MultiIndex

Comments:
looks fine...pls rebase one more time

thanks!
BUG: Escape `<` and `>` in info _repr_html_
State: closed

Closes https://github.com/pydata/pandas/issues/7105

Before: http://nbviewer.ipython.org/gist/TomAugspurger/4c83a646ffb93d5ee978
After: http://nbviewer.ipython.org/gist/TomAugspurger/81bf15086864ca090bb6

After conosle:

```
In [28]: df
 Out[5]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2 entries, 0 to 1
Data columns (total 2 columns):
0    2 non-null float64
1    2 non-null float64
dtypes: float64(2)
```

FYI, I don't know HTML, but I think this is mostly right.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
rebase on master and you can merge this in when you are ready, I put up the display changes section: https://github.com/pydata/pandas/commit/8a8ead63b15f36efb5fda6e202b158360b21fee2

Perf change vs. master

![formating](https://cloud.githubusercontent.com/assets/1312546/2959404/7d6aa6a4-dab2-11e3-81ed-fcc09fda12d9.png)

Doesn't look like much. The x axis is the size of an NxN DataFrame of floats in log base 2.

@jreback done. Will just wait on Travis.

Merged via 2bb2a8d52b2e37
API: allow option truncate for display.show_dimensions to only show dimensions if truncated (GH6457)
State: closed

closes #6547

add the option 'trunctate'.
Should this be the new default?????

```
dfd = pd.DataFrame(np.arange(25).reshape(-1,5), index=[0,1,2,3,4], columns=[0,1,2,3,4])

# show dimensions only if truncated
with pd.option_context('display.max_rows', 2, 'display.max_columns', 2,  'display.show_dimensions', 'truncate'):
        print(dfd)
```

```
   0  1    
0  0  1 ...
1  5  6 ...
  .. ..    

[5 rows x 5 columns]
```

```
# show dimensions only if truncated
with pd.option_context('display.max_rows', 10, 'display.max_columns', 40, 'display.show_dimensions', 'truncate'):
        print(dfd)
```

```
    0   1   2   3   4
0   0   1   2   3   4
1   5   6   7   8   9
2  10  11  12  13  14
3  15  16  17  18  19
4  20  21  22  23  24
```


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@jseabold @jorisvandenbossche @TomAugspurger @cpcloud 

Looks good on a quick glance.

small detail to nitpick, but `truncated` instead of `truncate` sounds better to me (as the option), as it is not really involved in the action of truncating, only the state of being truncated

@jorisvandenbossche I agree it sounds better, except, that `large_repr` uses `truncate` 

and wanted to be consistent, but ok if you feel strongly

changed to use 'truncated' & example in v0.14.0.txt simpler

Ah, but if `large_repr` already uses the other, I am not feeling strongly about it.

@jreback about if this closes the issue that @jseabold brought up in #6547, he can maybe better say it himself, but I don't think it really does solve his issue he encountered in statsmodels (the use of dataframes to give back the overview result of a model, for which these dimensions are a bit superfluous/distracting). But as also said on the mailing list, not everybody's needs can be satisfied at the same time. 
And this PR provides in any case all needed architecture. Given this, the discussion in the issue/on the mailing list, is only about what should be the default value of `display.show_dimensions`: `True` or `truncated`? `True` would be compatible with 0.13, `truncated` would be compatible with <=0.12 (for small dataframes).

ok..put back to 'truncate'.

like @jseabold to comment (and maybe should send to ML). but I think let's make the default 'truncate'.?

Posted on the [ML](https://groups.google.com/forum/#!topic/pydata/wyhbADpsUss). I tentatively vote for the default to be `truncate(d)`. That really does fix the statsmodels issue.

thanks @TomAugspurger
BUG: GroupBy doesn't preserve timezone
State: closed

Closes #3950. This is a fix for `groupby` issue.
(#7092 is for original report)


Labels:
name: Bug
name: Groupby
name: Timezones
description: Timezone data dtype

Comments:
@jreback OK. Modified `algos.factorize` to meet the behaviour discussed in #7090. Is this what you mean?

yep...thats good..thanks!

always looking to consolidate code / eliminate special cases
BUG: DatetimeIndex with freq raises ValueError when passed value is short
State: closed

When `DatetimeIndex` is created by passing `freq` and `check_integrity=True`, passed `freq` and `inferred_freq` must be identical. But this comparison can fail if length of passed data doesn't have enough length to infer freq. 

Added additional logic to determine whether passed values are on the passed `freq`.


Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
this is fine, rebase and can go in

thanks!
BUG: Let DataFrame.quantile() handle datetime
State: closed

Closes https://github.com/pydata/pandas/issues/6965


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@jreback I'm having trouble with a bit of this.

With a DataFrame

``` python
In [3]: df = DataFrame({'a': pd.to_datetime(['2010', '2011']), 'b': [0, 5]})

In [5]: df.dtypes
Out[5]: 
a    datetime64[ns]
b             int64
dtype: object

In [6]: df.quantile(.5, numeric_only=False)

```

For the implementation, I've got a function `f` that I apply to each column essentially via `df.apply`.
When `f` gets the first column `a` the dtype is `object`, and I'm not able to convert from object back to datetime64.

```
ipdb> arr
0    2010-01-01 00:00:00
1    2011-01-01 00:00:00
Name: a, dtype: object
ipdb> _values_from_object(arr)
array([Timestamp('2010-01-01 00:00:00'), Timestamp('2011-01-01 00:00:00')], dtype=object)
ipdb> _values_from_object(arr).view('i8')
*** TypeError: Cannot change data-type for object array.
ipdb> arr.convert_objects(convert_dates=True)
*** TypeError: Cannot change data-type for object array.
```

What's to best way to get from these timestamps to viewing as `i8` so that the quantiles can be computed?

just iterate over with iteritems() and concat the results

apply has a lot of heuristics that u don't need

Thanks. That's a lot better.

no prob...this looks fine otherwise
BUG: tz info lost by set_index and reindex
State: closed

Closes #6631. Closes #5878. Regarding #3950, original problem can be fixed by this, but `groupby` problem isn't.

Also, this includes the fix for `MultiIndex.get_level_values` doesn't retain `tz` and `freq`, as the method is used in `set_index`.

```
# current master
>>> import pandas as pd
>>> didx = pd.DatetimeIndex(start='2013/01/01', freq='H', periods=4, tz='Asia/Tokyo')
>>> pidx = pd.PeriodIndex(start='2013/01/01', freq='H', periods=4)

>>> midx = pd.MultiIndex.from_arrays([didx, pidx])
>>> midx.get_level_values(0)
<class 'pandas.tseries.index.DatetimeIndex'>
[2012-12-31 15:00:00, ..., 2012-12-31 18:00:00]
Length: 4, Freq: None, Timezone: None

>>> midx.get_level_values(1)
Int64Index([376944, 376945, 376946, 376947], dtype='int64')
```

```
# after fix
>>> midx.get_level_values(0)
<class 'pandas.tseries.index.DatetimeIndex'>
[2013-01-01 00:00:00+09:00, ..., 2013-01-01 03:00:00+09:00]
Length: 4, Freq: H, Timezone: Asia/Tokyo

>>> midx.get_level_values(1)
PeriodIndex([u'2013-01-01 00:00', u'2013-01-01 01:00', u'2013-01-01 02:00', u'2013-01-01 03:00'], freq='H')
```


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Period
description: Period data type

Comments:
OK, added `_simple_new` to `Index`. Also, I noticed #5878 can be solved by the same fix.

gr8! thanks
ENH/CLN: Add factorize to IndexOpsMixin
State: closed

As pointed in #7041, I prepared a PR to add `factorize` to `IndexOpsMixin`.

As a side benefit, `Multiindex.from_arrays` can preserve original `DatetimeIndex.freq` and `tz`. (Related to #3950 and #6606.  These issues are not solved yet because these use different methods to create `MultiIndex`). 

I would like to confirm following points before adding more tests.
- What `Index.factorize` and `Series.factorize` should return as unique values, `ndarray` or `Index`? I think it should return `Index` to preserve `DatetimeIndex` attributes (`freq` and `tz`).
- Is this should be added to `api.rst`? 


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Categorical
description: Categorical Data Type

Comments:
this seems reasonable; factorize for a `Index/Series` should return an `Index` (of course using it on an ndarray just returns an ndarray). So the api is preservered. You can add it to the `api.rst` for completeness.

rebase 1 more time (I think just rlease notes conflict then good to go)

@sinhrks thanks!

@jreback Thanks to merge. Added more tests and added descriptions to `api.rst`
SQL: datetime writing: add tests + issue with pymysql
State: closed

There were apparantly not yet tests for writing datetime data. So added a test case for both query/table.

For MySQL using the pymysql driver, writing a datetime column is not working ... It gives following error:

```
In [65]: import pymysql
In [66]: engine_mysql = sqlalchemy.create_engine('mysql+pymysql://root@localhost/pandas_nosetest')
In [67]: df = pd.DataFrame({'A': pd.date_range('2013-01-01 09:00:00', periods=3),
    ...:                    'B': np.arange(3.0)})
In [68]: df.to_sql('test_datetime', engine_mysql, if_exists='replace')
Traceback (most recent call last):
  File "<ipython-input-68-95961c7ce232>", line 1, in <module>
    df.to_sql('test_datetime', engine_mysql, if_exists='replace')
...
  File "C:\Anaconda\envs\devel\lib\site-packages\pymysql\converters.py", line 24, in escape_item
    encoder = encoders[type(val)]

KeyError: <class 'pandas.tslib.Timestamp'>
```

The data are feeded to the database using `df.itertuples()` and then `np.asscalar()`, and apparantly this gives data of the `Timestamp` class. And for some reason the other drivers can handle this (because it is a subclass of datetime.datetime?), and pymysql not. Which is not good...


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
does this handle NaT / None conversions?

I don't know if we need a new item, but pandas is good at handling timezone aware dates, but mysql has no knowledge of them.  I don't know if we can lay down some suggestions/code for dealing with that data.  For example, an IS_DST field along with a MySQL DATETIME field.  It's unfortunate that the capability of MySQL is so limiting in this respect.

No, not yet tested NaT/None conversions (and there are certainly things not working with that, see #2754). 
@rockg Indeed we should see what we do with timezones, but here, even just a simple datetime isn't even working with mysql (but not due to mysql, but pymysql I think). For timezones, I think we should first ensure it works (does not error, maybe by storing it as utc), and then later maybe see to add options to store it in some other way. If you have concrete ideas of proposals for that, certainly let that know. Although I think eg storing it as a seperate column is up to the user to do this themselves.

Merging this, and I will open a new issue for the issues with MySQL/pymysql
BUG: read_fwf colspec should treat None like slice
State: closed

Closes https://github.com/pydata/pandas/issues/7079


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
can u add. test for none in an earlier (than last)
and multiple ones 

thxs 

Yeah I forgot to add those.

@TomAugspurger ok when green
SQL: resolve legacy mode + deprecate mysql flavor (GH6900)
State: closed

- removed warning for not using sqlalchemy (as sqlite DBAPI connection is fully supported, no warning needed)
- added deprecation warning for 'mysql' flavor
- removed necessity of providing flavor kwarg (no warning if not provided, assumed to be sqlite3)
- removed `flavor` kwarg from execute and read_sql_query (because a) are not database specific functions + b) would only be needed for mysql, but this will is deprecated so no need to introduce it)
- updated the tests to reflect this
- updated docs and docstrings to reflect that sqlite3 is the only but fully supported DBAPI connection (no 'legacy')

Closes #6900.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@hayd @danielballan @jreback This can use some carefull review I think.

Looks thorough and correct. I can't find an issue.

OK, then I am going to merge this. Further comments are always welcome after-merge! (but it's a bit difficult to futher work on the sql code with a lot of PRs open)