BrandonFors/pandas-github-issues huggingface dataset:
DOC: Mention date and time formatting available throughout MPL
State: closed

The visualization documentation only mentions plotting using pandas' interface. This patch adds another section mentions that it is also possible to plot directly with MPL and that date and time formatters are automatically available through all plots in MPL. An example shows matplotlib's errorbar function where points and errors on the x axis are respectively timestamps and offsets.


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
the error plotting is going to be supported by #5638 

Ok, I didn't know. I'll prepare another example: is fill_between (planned to be) supported by pandas?

no....that would be fine

For your convenience, this is the example plot, now using MPL's stackplot.

![stackplot](https://f.cloud.github.com/assets/1579083/2477667/bfbe77a6-b073-11e3-9a83-e0d03ab4515e.png)

@jorisvandenbossche look ok ?

@agijsberts I think this is certainly a good addition to the docs. 
Although I was thinking, the fact that sometimes you need to plot directly with matplotlib, and the date formatting are to somewhat seperate facts that both can be under this title. But maybe seperate them a little bit? (eg just in two paragraphs)?

Regarding stackplot example, I've prepared PR #6656. I've used `fill_between` to outplot it.

Bad news for me, good news for pandas ;)

I updated the patch with a simple finance related example.

![bollinger](https://f.cloud.github.com/assets/1579083/2492463/3269552a-b22f-11e3-8386-b890752a3452.png)

Hopefully this will do.

@agijsberts can you squash this down to a single commit...otherwise looks ok

@jorisvandenbossche @TomAugspurger 

@jreback Sorry about that, still new to git. Should be ok now; if not, please let me know.

OK!
REGR: fixing Timestamp/Series subtraction, resolves #6648
State: closed

closes #6648.

Admittedly the conditional in `__sub__` is a little gross now.

I didn't add release notes as this is really just patching up a yet-to-be released item, but please let me know if I should.

Also, the test only runs on numpy >= 1.7, is that okay?


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Timedelta
description: Timedelta data type

Comments:
this looks fine.

can you do a quick perf check?

Perf test is clean.

thanks!
PERF: Speed up DatetimeConverter by using Matplotlib's epoch2num when possible...
State: closed

closes #6636

This fixes the performance bottleneck described in #6636 by using the vectorized epoch2num for DataIndex arrays. The included test ensures that the DatetimeConverter produces the same results as the Matplotlib's date2num for individual Timestamp objects as well as DatetimeIndex array objects.


Labels:
name: Visualization
description: plotting
name: Performance
description: Memory or execution speed performance

Comments:
Will do. A quick question though: I just noticed there are already tests in pandas/tseries/test/test_converter.py. Would that be the correct place to add them instead, rather than pandas/tests/test_tseries.py?

ok to put then in the converter test suite

Maybe also add a vbench speed benchmark?

I created a simple vbench benchmark for datetimeindex conversion in vb_suite/timeseries.py. Before I commit though, 2 small questions:
1. What should I use as start_date? I would imagine the date that DatetimeConverter was added to the project, any way to find out when that was?
2. DatetimeConverter depends on matplotlib. Should I deal with possible import failures?

you can use 1/1/13 is fine
vbench will catch import failures so just import 

vbench run:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
datetimeindex_converter                      |   0.8059 | 411.0680 |   0.0020 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [dc95c4c] : Added vbench performance benchmark for speed up of DatetimeConverter.
Base   [4216178] : Merge pull request #5505 from jsexauer/fix5505
```

@TomAugspurger ok with this?

Yep this looks good. Should I merge it?

@agijsberts just need a release note

Latest commit implements comments and adds release notes. Ok like this?

Sorry, some tests fail in the last commit. If I replace `from pandas import Series`  with `from pandas.core.series import Series` all is fine. Anybody knows why this is?

the first causes a circular import 

Good to know, thanks!

@TomAugspurger you have the honors!

Thanks @agijsberts.
BUG: Bug in DataFrame.reindex(labels,level=0) with with reordered labels (GH4088)
State: closed

closes #4088


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
@jreback Can you explain why you closed it for future reference (issue is still open)?

I really couldn't get this to work. got tricky. I'll revisit at some point.

was closed by #8948
CLN: Fix many indentation errors found in #6643.
State: closed

Changes should only affect whitespace.

closes #6643


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@dsm054 can you rebase (and remove the sql stuff)...then can merge

@dsm054 can you rebase this....

@dsm054 want to rebase this?

Okay, this is the next one up.

ok make a one liner for that elif change pls
API: A tuple passed to DataFame.sort_index will be interpreted as the levels of the index (GH4370)
State: closed

closes #4370

```
In [3]: df = DataFrame({'one':[2,1,1,1],'two':[1,3,2,4],'three':[5,4,3,2]})

In [7]: df.columns = MultiIndex.from_tuples([('a','one'),('a','two'),('b','three')])

In [8]: df
Out[8]: 
     a           b
   one  two  three
0    2    5      1
1    1    4      3
2    1    3      2
3    1    2      4

[4 rows x 3 columns]
```

These are now equivalent. Used to be able to do only the 2nd.

```
In [9]: df.sort_index(by=('a','two'))
Out[9]: 
     a           b
   one  two  three
3    1    2      4
2    1    3      2
1    1    4      3
0    2    5      1

[4 rows x 3 columns]

In [10]: df.sort_index(by=[('a','two')])
Out[10]: 
     a           b
   one  two  three
3    1    2      4
2    1    3      2
1    1    4      3
0    2    5      1

[4 rows x 3 columns]
```


Labels:
name: Docs
name: API Design
name: MultiIndex

Comments:
anyone think this is a bad idea?

Looks like a good idea at first sight. Should this also be implemented for `sort`?

`Dataframe.sort` just passed thru to `sort_index`...but'll change the doc-string
ENH add base arg to value_counts
State: closed

Often I find myself wanting to throw away values which aren't in a given set when doing `value_counts`...

This basically just does this but inside value_counts (before possibly sorting):

```
pd.value_counts(x).reindex(base, fill_value=0)  # same as Series method
```

_I previously thought this was trickier to implement, so feel a bit stupid, but anyway could be useful....?_


Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
I. theory this could be done in value_counts itself (the actual cython code) and would make it faster 

Agreed. Will check it out. (Also, random build error)

FYI started to change `lib.isimember` at the same time to use hashtables rather than set, I think perf is 10x better.

ahh sexy 

nice on perf with `lib.ismember` - sounds good :). I'd leave the check for this with the result. I don't think the perf difference will be that great and it'll leave the value_counts method less complicated. [the reindex would cause a memory copy and you'd be trading counting a few less items for potentially having to do a lookup for every single item (and/or checking for the existence of base every iteration)]

So IIRC (I need to look back at this next week / when I have some time) this was pretty trivial/clean to change in the cython code (there's already a method to lookup an array in a hashtables).

ping!

ping

let's move to 0.15

nice idea but stale

I killed the last wip (performance) commit, I still think base arg could be a good idea.

I've rebased - let's see if it still passes...

dang it, apparently you can't force push to a closed PR (you have to reopen first). Submitting new PR.
BUG: Bug in fillna with limit and value specified
State: closed

from SO: http://stackoverflow.com/questions/22379343/change-first-occurrence-of-nan-in-a-specific-dataframe-row-to-a-new-value/22380225#22380225


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Looks good to me.

thanks....Panel is now NotImplemented, because its ambiguous which direction to run the limit filling. Not sure how to deal with it (as really need another argument I think).
ENH: Allow use of longtable package in to_latex.
State: closed


Labels:
name: Enhancement
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
What's the difference in the last two Travis builds? How do you escape a backslash in a docstring with all the magic?

pls add a release note (and mention in v0.14.0 if you think its warranted). otherwise ok

Was hoping these were automatically linked now. Added.

I don't like the auto-linking actually. It creates just about the same amount of work as you have to categorize in any event.

merged via https://github.com/pydata/pandas/commit/98f51f53ecfa217a381938db84270e52ca59c9cb

It's really not more work, but I've already made my case for why it works with examples of it working elsewhere.

if you want to implement it go for it.

Yes, I probably will end up doing it when I have an afternoon. Here I was trying to alleviate the burden on occasional contributors.

always a plus...
BUG/CLN: fix iloc when positional indexer matched Int64Index key
State: closed

closes #6612

As discussed in #6612, `reindex(..., takeable=True)` as released in 0.13.0 didn't do anything that's not achieved by `.take`, `.iloc` and `.copy` attrs/methods, but also contained a bug which propagated to `iloc` when the latter was changed to use _reindex/takeable_ internally.

The proposed solution is to remove _reindex/takeable_ functionality altogether. According to @jreback, it was never supposed to be a part of public API, so probably no deprecation procedure is necessary.


Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
I'll add tests & release notes later tonight.

looks good
simple release note is fine

I've added this as a bug fix as it technically fixed a bug with `iloc` (and also renamed the issue).

thanks!
FIX str.match uses na flag
State: closed

fixes #6609


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
travis failing is a refused connection with fred, unrelated to this pr (you'd think)

https://travis-ci.org/pydata/pandas/jobs/20585139

hmm are the Fred calls wrapped with network. decorator?
that should catch and skip those tests

actually these network errors are 'ok'. because in theory if the URL had changed that's what we'd get 

so if it keeps repeating....can investigate further

I was wondering, should it be noted in the docstring that by default it gives NaN for non-strings? (because you could also assume this gives False)

@jorisvandenbossche this fix makes it inline with the current behaviour of contains (et al?), I guess we could change the docstring too (not sure which others this is used in).

...tbh I had assumed that it was equivalent to s.astype(str).str.contains(...) but they aren't (i.e. you can't search for digits of a number).

(re fred, this branch passed on hayd/travis, so it's something intermittent)

@jorisvandenbossche I kind of think I should merge this in first, then have sep issue for the docstring. Maybe adding it to docs example of vectorised string methods would be enough...

Oh goodness, when I implemented `get_dummies` I used `.astype(str)` to get around this problem, I had assumed this was how they all worked... I suspect that's going to raise sometimes? :s

Yeah, but I would certainly add something to the docs, since you also expected non-strings to be converted to strings instead of giving NaN.

For further discussion see also #6634

What's weird is that I'm still convinced I'd tested it (on the other methods)!
ENH: Allow update to use an on keyword. Allow one to many update.
State: closed

Scratch an itch I had. Use case

```
df = DataFrame([[np.nan, 'A'],
                [np.nan, 'A'],
                [np.nan, 'A'],
                [1.5, 'B'],
                [2.2, 'C'],
                [3.1, 'C'],
                [1.2,' B']], columns=['number', 'name'])

df2 = pd.DataFrame([[3.5, 'A']], columns=['number', 'name'])

df.update(df2, on='name')
```

The tests fail though because when you reset the index, the column order is not preserved and `self = self[col_order]` doesn't seem to do what I'd expect it to do. Maybe there's a better way to go about doing all of this?

Also, I dropped reindex_like because you were just iterating through columns of NaNs in `other`.


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
can you put the output you are expecting up as well?

NaNs "updated" to 3.5.

```
In [10]: df.loc[df.name.isin(df2.name)] = df2

In [11]: df
Out[11]: 
   number name
0     3.5    A
1     3.5    A
2     3.5    A
3     1.5    B
4     2.2    C
5     3.1    C
6     1.2    B
```

This is just a toy case. What if on is multiple columns? This also only updates where there are NaNs in df. There might be overlap in df and df2 and this sanity checks that.

true.....`update` really needs to be rewritten actually (and pushed into internals).....but that's a bit more drastic....

shouldn't you merge these then set?

Sounds like half a dozen of one to me... Merge, conditionally set `_x` for each column then delete `_y` and rename everything? 

moving a bit away from adding things to `.update` and stale
SQL: move df.to_sql to generic + update with new functionality
State: closed

Initial commit for this.

Questions:
- To catch Panels (and raiseNotImplementedError), should I do this in sql.to_sql or in NDFrame.to_sql?
- To support Series, I can just call `to_frame` on it. Is this OK?

And should add tests for this.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
I would to this in `sql.to_sql` (then `NDFrame.to_sql` is pretty generic)

you can call `to_frame` on Series, sure

added tests
FIX: Bug whereby array_equivalent was not correctly comparing Float64Ind...
State: closed

Currently, 

```
>>> import pandas.core.common as com
>>> com.array_equivalent(Float64Index([0, np.nan]), Float64Index([0, np.nan]))
False
```

Although the current pandas code base does not use `array_equivalent` to compare Float64Indexes, leaving `array_equivalent` in its current state may be a bug waiting to happen. 

This PR attempts to fix the problem by using `pd.isnull` for all arrays of dtype `object`. In a previous PR I tried this and got terrible perf results. Since then I've discovered that my machine does not have enough memory to run the full perf test suit without page faults. If I rerun `test_perf.sh` for just a few Benchmarks, I can avoid the page faults and get consistent results.

Running `/usr/bin/time -v ./test_perf.sh -b master -t fix-equivalent` yielded two tests with ratio > 1.1. 

``` -------------------------------------------------------------------------------
reindex_fillna_pad                           |   0.5784 |   0.5034 |   1.1490 |
packers_write_pack                           |  15.2360 |   7.1851 |   2.1205 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

which I believe were due to page faults. When I reran perf on just these tests using
`/usr/bin/time -v ./test_perf.sh -b master -t fix-equivalent  -r "reindex_fillna_pad|packers_write_pack"`

I got

``` -------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
reindex_fillna_pad_float32                   |   0.4633 |   0.4590 |   1.0093 |
packers_write_pack                           |   7.9544 |   7.8390 |   1.0147 |
reindex_fillna_pad                           |   0.7290 |   0.7180 |   1.0154 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thank you sir!
Gbq unicode support
State: closed

Resolves Issue https://github.com/pydata/pandas/issues/5940 . 


Labels:
name: Enhancement
name: Unicode
description: Unicode strings

Comments:
can you add a release note under improvements

also want to rebase away the merge of master (I can do it when I merge in any event)

I rebased and added a note in the 0.14.0 release docs.
Improve performance for custom business days (GH6584)
State: closed

closes #6584

Tests are passing. Incrementing datetime with custom dateoffset is at about twice the speed now. This could probably be optimized more. 


Labels:
name: Performance
description: Memory or execution speed performance
name: Frequency
description: DateOffsets

Comments:
can you add a vbench . look for agood place in the vb_suite dir

Sure, might take a few days though. I'll let you know when I had the time to look into this. 

I've added a small vbench to vbsuite/timeseries.py. 

Btw, what is the standard way to run vbsuite on the last commit? I did

%run test_perf.py -t 6305492393 -b 391ed321528 -r timeseries

timeseries_custom_bday_incr                  |   0.0157 |   0.0353 |   0.4437 |

Perhaps it would be good to add some hints on how to use vbench, which script to run etc. 

https://github.com/pydata/pandas/wiki/Performance-Testing

pls add a release notes entry under enhancements, referencing #6584

I've added the release note and rebased. Do you want me to squash this into one commit? 

Also, I have started to implement `pandas.offsets.CustomBusinessMonth`. Should I open a separate PR for this or keep going here? 

pls squash 

open a new PR/issue for different `CustomBusinessMonth`

Ok done

thanks!
BUG: replace iterrows with itertuples in sql insert (GH6509)
State: closed

Fixes #6509.

Should still add a test


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jreback I added a `check_exact` keyword. Is this what you had in mind?

@jorisvandenbossche perfect.

assert_numpy_array_equal should prob be a module level function too (it just calls np.array_equal in any event), but no biggie.

in theory all functions should actually be on the TestCase, but too much work to fix

pls add a release note then merge 

Should there be a release note? As this fixes something (the new sqlalchemy based sql functions) that is not yet released?

oh right.....though like to reference the closed issue (maybe just add it to an existing one)

Actually, it was also fixed in the legacy code. Anyway, added a line to release notes.
FIX raise when groupby selecting cols not in frame
State: closed

KeyError part of #5264


Labels:
name: Bug
name: Groupby

Comments:
rebase on master .....setuptools had changed so build was failing

the 2.6 failure is real though...

lol to the failure, similar to me just claiming was an issue in 2.6. Silly.

maybe slightly more intuitive, but yours pretty self exaplanatory

```
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
index f0ee01d..fce8947 100644
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2579,8 +2579,8 @@ class DataFrameGroupBy(NDFrameGroupBy):
             raise Exception('Column(s) %s already selected' % self._selection)

         if isinstance(key, (list, tuple, Series, np.ndarray)):
-            if len(self.obj.columns.intersection(key)) != len(key):
-                bad_keys = list(set(key).difference(self.obj.columns))
+            if self.obj.columns.isin(key).sum() != len(key):
+                bad_keys = Index(key)-self.obj.columns
                 raise KeyError("Columns not found: %s"
                                % str(bad_keys)[1:-1])
             return DataFrameGroupBy(self.obj, self.grouper, selection=key,
```

subtle improveent is that yours doesn't allow for repeats... whhich is not supported... should it be :s (rabbit hole?)

hmm

I suppose that should technically be allowed
so go with yours 

No methods work at the moment (as you're reindexing by a dupe), will have a little think.

the reindexing could be fixed, but i think you are right, it opens a big can of worms.So pls add a check and test for dups (just raise ValueError) for now.

Now I think about it, not tested axis=1...at all (I'm guessing this won't work, I could just raise as NotImplemented ) :S

more tests on this for axis=1? release notes?

ping!

ping

@hayd ping....need to get this in ASAP

this looks fine. anything else on this? give ok and I'll merge it (I want to update the release note a bit)

Will have some time for this late tomorrow (and wed and thurs), sorry been swamped recently, need to pull finger out on many PRs lined up mañana.

I see point on axis=1, I'm worried machinery is different... and not played with this.

ping!
CLN: Change assert_([not] isinstance(a,b)) to specialized forms
State: closed

Work on #6175. Changes instances of assert_([not] isinstance(a,b)) to specialized assert[Not]IsInstance(a, b).


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I will work on the Travis problems. (These did not get picked up locally, as I am running with 2.7.)

rebase on master
these r already fixed 

I fetched latest master and rebased (also, changed one superclass, to use tm.TestCase instead of unittest.TestCase). The two errors I am still seeing on Travis are "ImportError: No module named pandas".

Travis is being flaky today...

@bwignall rebase on master and travis should be ok now.....setuptools had changed and caused the install to fail

@bwignall thanks!