BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fixes color selection in andrews_curve
State: closed

Fixed bug introduced in b5265a5b84ae1aeaf5fff88f1f9e5872a9b404e6 which change from picking a single random color for the class to picking a random color for each line.

This returns the color selection to a per-set basis.  

See http://stackoverflow.com/questions/19667209/colors-in-andrews-curves


Labels:

Comments:
Please post an image of output you get from this.

(and nice to have reproducible example, even if it's just copying from SO question)

```
import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(100,3), columns=['A','B', 'C'])
df['X'] = np.random.choice(['Alpha', 'Beta', 'Theta'], size=100)
pd.tools.plotting.andrews_curves(df, 'X')
```

![so](https://f.cloud.github.com/assets/199813/1434164/502f6770-410f-11e3-932b-c45889f5bddc.png)

thanks for posting that - this looks good to me (going to try to pull down and see if I get the same thing) - anyone have any comments?

Looks great now; thanks! 

Let me know if this needs to be a separate request, but consider: 

```
plt.ylabel(r'$f_x(t)$')
plt.xlabel('t')
ax.xaxis.set_ticks([-pi, -pi/2, 0, pi/2, pi])
ax.xaxis.set_ticklabels([r'$-\pi$', r'$-\frac{\pi}{2}$', '0', r'$\frac{\pi}{2}$', r'$\pi$', ])
```

Everyone loves properly labeled plots. 

![so_andrewscurves_03](https://f.cloud.github.com/assets/324454/1441531/b9485062-41ac-11e3-83a2-5efb37f456ce.png)

Can you post another image? :)

Let's keep this to the colors for now and can deal with labeling
separately.

Just updated my last comment with an image. :)  I think having the grid lines in multiples of pi makes a world of a difference. 

The colors change is minimal enough that I'm willing to merge it (and it seems clearly correct). We're in the middle of putting out a release so, in terms of changing labels or ticks, I'd prefer to hold off on that for a week or two. You can put up a separate PR that we can consider though. 

It looks like you have Travis set up but that it didn't build this. If you think Travis is set up can you run `git commit --amend -C HEAD && git push --force` to get Travis to notice the build? 

I hadn't enabled it to build.  Travis is running now.

And it looks like it built my master branch (which is just master).   

Need to check out the `fix_andrews...` branch

Sorry, I am really confused by your travis set up (I am used to mpl where PRs are auto-magically built by the matplotlib travis account).

you did it :)

If there's a way to do that in Travis, we probably should - that would be much easier for contributors.

Can you add a quick note to release.rst? After that I'll merge.

Done.

Squashed the commits.

If you could, could you change the commit name to something like: "BUG: Fix color selection in andrews curve" and then you can put "fixed bug introduced in b5265a5
which change from picking a single random color for the class to
picking a random color for each line."

In the commit message?  Makes it easier to follow what's going on.

like that?

exactly. once it passes Travis I'll merge - thanks for the final cleanups!

thanks!
repr for PeriodIndex does not handle <=2 elements well (GH5372)
State: closed

closes https://github.com/pydata/pandas/issues/5372


Labels:

Comments:
pls try to name the issues and commits to something related to the issue (and just reference the issue, e.g. GH5372)

@jreback sorry, changed name to more appropriate

gr8

can you add a release notes entry refing this issue? (doc/source/release.rst)

pls also rebase and squash: https://github.com/pydata/pandas/wiki/Using-Git

@jreback  hope I did it right

the release notes look fine, but you need to squash...

```
git rebase -i <commit hash before your first one>
```

then `s` or `f` on the commits

then 

```
git push myfork thisbranch -f
```

@jreback another try, sorry, used github only occasionally before

@RomanPekar thanks!
TST: Better handle np.array_equal() edge cases
State: closed


Labels:

Comments:
ok...

@JeffReback we want to check each element if array_equal doesn't work, right?

I think that if array_equal returns False and they are the same shape/dtype, and both can hold nan (e.g. float/object), then coerce to float and check for nan, otherwise I think its correct, just return False

Well, examples of things that fail:

```
np.array_equal(np.array([], dtype='M8[ns]'), np.array([], dtype='float64'))
```

should 0-Len be equal regardless of dtype

if yes then this is trivial

I think so. Also think assert_almost_equal should ignore dtype and callers
should check that. This handles the case shown above.

agree
ENH: HDFStore.flush() to optionally perform fsync (GH5364)
State: closed

Pull request for #5364.

The `fsync=False` parameter was added to `flush` given this offers utility for non-distribution use cases, such as a user just wanting to ensure their data is regularly flushed to disk.

I could not see any mocking library in use in the Pandas tests, so I skipped adding one just to verify `os.fsync` actually gets called.


Labels:

Comments:
For now, probably not worth it, because we don't have much that really needs it (might be nice for network tests but a little too late now).

thanks for this!
fix: to_datetime returns KeyError: 'p' when %p specified in format string (GH5361)
State: closed

closes https://github.com/pydata/pandas/issues/5361


Labels:

Comments:
pretty sure most of these tests are in

pandas/tseries/tests/test_timeseries.py

pls hook up Travis as well (see contributing.md)

I realize that there is no real test coverage for to_datetime with various formats (there is SOME though). would you be up for moving this test (and the existing 2-3 tests) to a new class (within test_timeseries), then adding some coverage for formats? (a test like you did with the %p is enough).

fixed and added tests, but tests with PM failed on one of Travis configurations (python 2.6), don't know what to do with that, so I commented tests

do you want to try to debug the 2.6 error, you can use a virtual env

@RomanPekar can you add a release notes entry for this? (under bug fixes for 0.13.1)

@RomanPekar can you add release notes notes for this?

@jreback Sorry had absolutely no time for last 2 months, spend all time with my family, I'll try to add notes ASAP

@RomanPekar can you rebase?

@jreback I'll try to find time tomorrow

merged via https://github.com/pydata/pandas/commit/b5535b91278cb5712eb3b0f977e41b12c460d8d0
CLN: Add axis kwarg to Series wrapper for compat
State: closed

Now flex methods can take axis=0/axis='rows'/axis='index'. Also updated
tests to cover full range of Series flex ops and check that they all
accept an axis argument.


Labels:

Comments:
@jreback want to take a quick glance? I think this is fine.

I guess u don't actually use the axis arg in the series ops methods
ok on this
API: provide compat for Timestamp  now/today/utcnow class methods (GH5339)
State: closed

closes #5339


Labels:

Comments:
cc @JanSchulz , cc @cancan01 any other methods you think are missing?

@jreback it's cancan101 !

`fromtimestamp`,  `utcfromtimestamp`, `strptime` and `combine` are also broken.

@cancan101 sorry...typo!

@cancan101 ...want to add them? should be straightforward...just add tests and push to this PR!

@cancan101 pls create a new issue for these other methods
BUG/CLN: Cleanup sanitize column and abstract broadcasting
State: closed

Also fixes small bug from #5321


Labels:

Comments:
@jreback A few points here:
1. This function never checked for column uniqueness previously, it always tried the lookup and then tested if it was a DataFrame or not. Thus it can't be a big performance hit (and my own perf testing shows that it isn't...)
2. It can't check for  uniqueness because of issues with multilevel columns. E.g. consider this example:

```
df = pd.DataFrame(zip(range(-5,1), range(-5,1)), columns = pd.MultiIndex.from_arrays([['a', 'a'], ['x', 'y']]))
df['a'] = df.index
```

regarding your last comment....

yeh..perf may not be a problem....I am always wary of heavily called functions that have to deal with unique/non-unique...almost always I separate these cases (because of perf)

if its a multi-index it doesn't matter(so `self.index.is_unique and not not isinstance(self.index,MultiIndex)` should suffice (fyi....I would maybe add a method to do exactly this in your refactor (or could even override is_unique for mi, maybe)

@jreback just thinking about it again, there's no reason to have a separate branch for Index, just needs to do an instance check for Index and ndarray first, then can use is_sequence.

Is there a reason to prefer `is_sequence()` over checking `np.isscalar(val)`?

@jtratner that's fine...but then need to fix `is_sequence` (for the perf hit of `iter`, e.g. you can special case Index/Series in there I think``

you can try `is_scalar`....there is a prob a case where they differ (e.g. a Timestamp would hit is_sequence and not isscalar)

@jreback wha? Timestamp is true for `_is_sequence`?

ah, I see, is_scalar is False for timestamp, got it.

yep strings/Timestamp looks like lists and isscalr fails

@jtratner if perf checks out ok...this looks good!

I ran the perf check 4 times on base commit and tip of this PR, everything came out about even on average, and the few that were particularly different were because 1 run of that particular benchmark took _much_ more time than the other three, so I think it should be fine.

ok good to go then
Docs for error_bad_lines, and warn_bad_lines options to pd.read_*
State: closed

Hi,

Added some docs for error_bad_lines & warn_bad_lines, as currently there aren't any found here: http://pandas.pydata.org/pandas-docs/dev/generated/pandas.io.parsers.read_csv.html


Labels:

Comments:
Seems to me the docstring template used for all the methods makes it difficult to track 
which args apply where, but this is good.

merged. Thanks!
DOC: Added/fixed some docstrings, esp Panel flex
State: closed

1. Panel flex docstrings had str representation of functions like
   `<builtin-func add..>` (not pretty). Fixed to be name
2. Added some docstrings to miscellaneous properties and clarified
   meaning of info_axis/keys.
3. Wrap 0.13's release.rst at 80 characters per line.

I'm happy to revert (3) above...just was bothering me (most of the doc
breaks at 80)

Key change is here - http://jtratner.github.io/example-pandas-docs/html-minor-doc-fixup-10-25/api.html#id17


Labels:

Comments:
Any objections to this? I think it's a good time to cleanup release.rst too.

ok
BUG: empty series not printing name in repr (#4651)
State: closed

Implemented as suggested in #4651


Labels:

Comments:
@jtratner this ok I think? (I can rebase on merge)

needs to check if it has a name, no?

something like:

``` python
attrs = ", ".join("%s: %s" % (attr, val) for attr, val in [('dtype', self.dtype), ('name', self.name)] if val is not None)
return "Series([], %s)" % attrs
```

Also, that will eventually make it easier to handle metadata if we want (I guess)

well, these are always defined on series, so I think its ok (name could be None of course)

Right, but it doesn't show it in normal Series repr if None:

``` python
In [2]: ser = pd.Series([1, 2])

In [3]: ser
Out[3]:
0    1
1    2
dtype: int64

In [4]: ser.name = "apple"

In [5]: ser
Out[5]:
0    1
1    2
Name: apple, dtype: int64
```

so empty series repr would be:

``` python
ser = pd.Series([], dtype=object)
ser
Series([], dtype=object)
ser.name = "apple"
Series([], name="apple", dtype=object)
```

should fix the ordering that I posted there though.

Also, I prefer to put dtype in quotes so you can always copy paste, so that would be a good addition (don't do `%r`, just `'%s'`)

@jtratner this look ok to me.....I understand you want dtype in quotes, but that is a big change (as its not done elsewhere)

Not a big deal to me either way - and generally you're not copy/pasting
Series/Frame reprs anyways.

@goyodiaz can you rebase ?

I think we omit name if it is `None`, can you make this change?

```
In [1]: Series(name='foo')
Out[1]: Series([], name: foo, dtype: float64)

In [2]: Series()
Out[2]: Series([], name: None, dtype: float64)

In [3]: Series(dtype=object)
Out[3]: Series([], name: None, dtype: object)

In [4]: Series(dtype=object,name='foo')
Out[4]: Series([], name: foo, dtype: object)

In [5]: Series([1,2])
Out[5]: 
0    1
1    2
dtype: int64
```

I will try to look at this tomorrow.

thanks!
CLN: Fix return type for initObjToJSON()
State: closed

Make it so that it always returns the same thing as numpy, so that it
matches the right signature whether in or not in the error condition.

Closes #5326.


Labels:

Comments:
Here's the in-a-nutshell of why this is necessary: http://stackoverflow.com/questions/10509400/difference-between-pymodinit-func-and-pymodule-create

In Python 2.X the module init function could return void, in 3.X needs to return `PyObject*` (or NULL). I guess numpy assumes you'd use `import_arrays()` in a function whose signature is `PyMODINIT_FUNC` and therefore conveniently includes the appropriate return that you'd need on an error importing numpy multiarray. We weren't actually handling the non-error case correctly - thanks to @toddrme2178 for catching this!

Thanks @jtratner for digging into this. Fix looks good.

fyi...this builds on windows! so that is good

good, and it ought to build on openSUSE too...

I can confirm this fixes the problem on openSUSE.  Thank you for the prompt
fix.  I have backported the patch to the 0.12.0 release.

On Sun, Oct 27, 2013 at 1:41 AM, Jeff Tratner notifications@github.comwrote:

> good, and it ought to build on openSUSE too...
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5334#issuecomment-27158343
> .

No problem - thanks for reporting this! - it was a subtle error (and it was
helpful to look into the numpy C API further).  It also speaks to the
utility of the openSUSE build QC. :)
ENH: Add option to highlight NaN cells
State: closed

Useful for quickly inspecting data in an ipynb
- Add display.highlight_nan option (default True)
- Add argument to DataFrame.to_html
  and HTMLFormatter.__init__
- Add test


Labels:

Comments:
Thanks for the PR, but this solution is too specific. The actual feature needed is
conditional formatting and it needs to be solved in a more general way, one
which will support this as a private case.

closing.

related https://github.com/pydata/pandas/issues/3190?source=cc

This is nearly exactly what I have in mind.  (I would like to de-emphasize NaN cells.)

My idea was to add a class to empty cells:

``` html
<td class="pandas-nan">
```

Then, one can use CSS for arbitrary styling.  (No, even CSS3 does not have selectors yet based on _content_ which would allow to select NaN-cells.)

Since CSS allows many other types of conditional formatting already, I would say that this is a useful addition that does not interfere with other enhancement ideas (templates, ...)?!

I like you're solution much better @hmeine but it's for @y-p to decide.

@hmeine @sburns We might be waiting quite a while for @y-p to decide since he's stepped out of pandas dev. I think may be too specific. We don't really want to add anymore arguments to `to_html`, though I understand the need to be able to customize HTML output. If one of you wants to take the lead on #3190, that would be most welcome.
TST/BUG: allow invalid Timestamps to pass thru as datetimes when operating with offsets
State: closed

closes #5312


Labels:

Comments:
@jtratner, cc @cancan01

this provides tests for this on each class, though skips on some of @cancan01 newer offsets as they need custom creators

@cancan101 - at some point would be nice to go back and make them either
raise or return a valid datetime.
BUG: bug when trying to display an embedded PandasObject (GH5324)
State: closed

closes #5324

There was a bug where this wasn't display at all, so this PR fixes

Do we care about formatting _embedded_ pandas objects? (as we don't recommend this anyhow)

```
In [15]: df1 = DataFrame({'a':[1, 2, 3], 'b':[3, 4, 5]})

In [19]: df2.dtypes
Out[19]: 
0    object
dtype: object

In [16]: df1
Out[16]: 
   a  b
0  1  3
1  2  4
2  3  5

In [17]: df2 = DataFrame([df1,df1+10])

In [18]: df2
Out[18]: 
                                         0
0             a  b
0  1  3
1  2  4
2  3  5
1      a   b
0  11  13
1  12  14
2  13  15
```


Labels:

Comments:
@y-p @jtratner ?

:+1: 

This apparently is what 0.10.0 displayed, which is wrong (because it appears they are lists), but then again
does show a reasonable display.....

```
        0
0  [a, b]
1  [a, b]
>>> 
```

to contrived?

```
In [4]: DataFrame([ DataFrame({'A' : [1,2], 'B' : 0}), Series([1,2,3]) ])
Out[4]: 
           0
0  DataFrame
1     Series
```

you should just use numpy's array_repr for Series

I'd say this falls under "undefined behaviour". No objections to making this change though.
PERF: vbench for time-series index assignment in frame (GH5320)
State: closed

PERF: direct index assignment in a frame was doing lots of work

closes #5320

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_assign_timeseries_index                |   0.7157 | 398.8634 |   0.0018 |
```


Labels:

Comments:
This introduced a bug because it didn't broadcast the value on a matching index - I'm working on a fix right now.

``` python
import pandas as pd
df = pd.DataFrame(zip(range(5), range(5), range(5), range(5)),
                  columns=['C1', 'C2', 'C1', 'C3'],
                  index=['a', 'b', 'c', 'd', 'e'])
df['C1'] = df.index # fails because doesn't broadcast the value
```

dups ok let me take a look (prob was not tested before)

@jreback no, the issue is that you have to apply the broadcasting logic at the end of the if clause you took it out of. I already have a fix. no big deal.

When adding cases to already hairy conditional logic It's useful to add a
comment referencing the orig issue (GH1234) to help locate the orig. discussion. 
pep8 storms sometimes make using `git blame` inconvenient for this.

I picked this up originally from wes's code and I think it's a useful convention to adopt.

my 2c

That makes sense. Btw - I recently noticed that fugitive for vim lets you
traverse back through the history of a particular line - incredibly useful
feature.  If we try to keep line widths to 79 characters, that would also
help the problem :P
ENH: add basic postgresql support to io.sql
State: closed

I added in basic postresql support


Labels:

Comments:
Thanks for the contribution, unless @hayd wants to do this now, probably would push this to 0.14 (planning on a release candidate for 0.13 in the next few days). We are working on refactoring sql for the 0.14 release so hopefully we can incorporate some of this.

sounds good, I could always lend a hand if he would like some help with postgresql there.  I'd really like to see psycopg2.copy_from method make it into 0.14

I will absolutely take you up on that :). 

i just imported an 18 column, 54k row dataframe using that commited code, took 20 mins to update my remote server using executemany, bulk inserts would be a very nice thing to have

Btw, maybe easier to do would be to select subset of columns you care
about, then doing df.values.T, convert to list of lists and then pass to
executemany. (unless it can already work with ndarray). Don't have a
computer in front of me to test with right now though.

@jtratner, @hayd, there's a jungle of sql PRs to sort through, help?

We need one committer to take charge of this - I think my alternative
caused this to go off track. @hayd willing to do that?

I'm game to get back on top of this, I think we got stuck bike shedding before, and ought to push forward hard for 0.14.

I made bit of a mess of this with all these commits, I continued adding postgresql copy support to sql_legacy.py in PR #5323. Hope this helps some

@rla3rd, does that mean this is superceded by #5323 and can be closed?

If sql.py is to become sql_legacy.py in upcoming versions, and sql.py migrates to sqlalchemy under the hood, then yes, it is superseded by #5323.  I havent really done any work under the hood using sqlalchemy yet.

@mangecoeur, perspective re #5950?

@y-p this is superceded by the sqlalchemy support, we keep sqlite and mysql only for legacy support (since these were the only ones supported before)

@rla3rd acknowledged that the new SQL functionality in 0.14.0 supercedes this.
Rather then having two ways to dath same things, we'll stick to the more general one in #5950.

closing. Thanks for the PR none the less @rla3rd .
BLD: 0 13 pickles
State: closed

closes #4796


Labels:

Comments:
cc @TomAugspurger

can you generate a pickle on the mac for us?

pull down this branch, build like: 

```
python setup.py build

vagrant@precise32:~/0_13_pickles/build/lib.linux-i686-2.7$ python ../../pandas/io/tests/generate_legacy_pickles.py 0.13.0 ../../pandas/io/tests/data/legacy_pickle/0.13.0/
/usr/local/lib/python2.7/dist-packages/pytz-2013b-py2.7.egg/pytz/__init__.py:35: UserWarning: Module pandas was already imported from ./pandas/__init__.pyc, but /usr/local/lib/python2.7/dist-packages/pandas-0.12.0-py2.7-linux-i686.egg is being added to sys.path
  from pkg_resources import resource_stream
This script generates a pickle file for the current arch, system, and python version
  system version: 
  output version: 0.13.0
  output dir    : ../../pandas/io/tests/data/legacy_pickle/0.13.0/
created pickle file: 0.13.0_i686_linux_2.7.3.pickle
```

then commit and do a PR on this branch.....thanks

Getting an import error.

```
 pandas-dev  tom  dhcp80fff527  ~  Envs  …  python2.7  site-packages  pandas  jreback-0_13_pickles  $ python pandas/io/tests/generate_legacy_pickles.py 0.13.0
Traceback (most recent call last):
  File "pandas/io/tests/generate_legacy_pickles.py", line 146, in <module>
    write_legacy_pickles()
  File "pandas/io/tests/generate_legacy_pickles.py", line 114, in write_legacy_pickles
    import pandas.util.testing as tm
  File "./pandas/util/testing.py", line 40, in <module>
    from pandas import _testing
ImportError: cannot import name _testing
```

I'll see what I can figure out.

did you rebuild ? that was just added.....

I had used `python setup.py develop` in that virtualenv previously. Doing `develop` this time worked.

OK so I've pushed it [here](https://github.com/TomAugspurger/pandas/tree/jreback-0_13_pickles)

I want to PR against this PR right? I hope I did that correctly: https://github.com/jreback/pandas/pull/7

@TomAugspurger thanks....everything working so far.....cross-fingers

any one have a windows 3.3 build setup can can contribute a pickle?

any other relevant archs/combinations think are needed?
DOC add get_dummies to reshaping.rst
State: closed

examples shamelessly from Wes' book

closes #4444


Labels:

Comments:
@jreback Did I do something wrong here? This doesn't appear at bottom of web page: http://pandas.pydata.org/pandas-docs/dev/reshaping.html#tiling

nope

doc build stuck somehow (as of oct 11)

@wesm ping!

docs are updated! http://pandas.pydata.org/pandas-docs/dev/reshaping.html
(there was a typo here which I just fixed...fyi)