BrandonFors/pandas-github-issues huggingface dataset:
CLN: Converting assert_'s to specialized forms, in pandas/*/tests
State: closed

Work on #6175, in pandas/*/tests. Tests still pass locally. Finishes most (all?) of the remaining vanilla assert_(x == y) conversions


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
just noticed this...can you rebase?

I will give it a try.

Just tried my first rebasing. How does it look, @jreback ?

git checkout assert_6175_6
git rebase master
[resolve one merge conflict]
git add pandas/io/tests/test_sql.py
git rebase --continue
git push origin assert_6175_6
[Error about "Updates were rejected because the tip of your current branch is behind its remote counterpart." Consult Google, and find suggestion...] 
git push -f origin assert_6175_6

Your rebase is okay assuming your local master is synced with upstream master. Personally, I like to keep a remote called upstream and rebase on top of that right after fetching from it

looks gr8 @bwignall 

if anything more would appreciate the update, otherwise ping and we can close the issue out
PERF: various performance optimizations
State: closed

PERF: enhance tseries getitem indexing
PERF: micro optimzations in inference
API/CLN: core.internals.apply now only takes kwargs; makes code a bit simpler
CLN: wrap core/common/isnull into more direct Block.isnull
PERF: fast path series block operations

closes #6264

original timings

```
In [11]: %timeit com.is_integer(13)
100000 loops, best of 3: 1.29 µs per loop

In [12]: %timeit com.is_integer(np.int32(2))
100000 loops, best of 3: 3.32 µs per loop

In [13]: %timeit com.is_integer(np.int64(2))
100000 loops, best of 3: 1.86 µs per loop

In [14]: %timeit com.is_integer(14L)
100000 loops, best of 3: 1.29 µs per loop

In [15]: %timeit com.is_integer("1")
100000 loops, best of 3: 2.2 µs per loop
```

withi this PR

```
In [2]: %timeit com.is_integer(13)
10000000 loops, best of 3: 72.5 ns per loop

In [3]: %timeit com.is_integer(np.int32(2))
1000000 loops, best of 3: 620 ns per loop

In [4]: %timeit com.is_integer(np.int64(2))
1000000 loops, best of 3: 315 ns per loop

In [5]: %timeit com.is_integer(14L)
10000000 loops, best of 3: 72.4 ns per loop

In [6]: %timeit com.is_integer("1")
10000000 loops, best of 3: 78.8 ns per loop
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
closing in favor of #6269
CLN: Work on converting assert_'s to specialized forms, in pandas/tests
State: closed

Work on #6175, in pandas/tests. Tests still pass locally. A work-in-progress, but--with #6261--removes many/most cases of assert_(x == y) in this directory

Replacements:
- assert_(x == y) ==> assertEqual(x, y)
- assert_(x == True) ==> assertTrue(x)
- assert_(x == False) ==> assertFalse(x)


Labels:

Comments:
thanks on this one too!
CLN: More work on converting assert_'s to specialized forms
State: closed

Work on #6175 to specialize assert_'s. Tests still pass locally. A work-in-progress.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
See #6262 for rest of first pass through other files in pandas/tests.

looks good..thanks...keep em coming...

this is a good size of PR/changes
BUG: bug in xs for a Series with a multiindex (GH6258)
State: closed

closes #6258
closes #5684


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
I've got a commit ready adding the test from https://github.com/pydata/pandas/issues/5684. It's similar, but uses labels instead of integers for the level, so it may be useful.

For the PR will I submit it against this branch, `jreback/pandas/xs_bug` instead of `pydata/pandas/master`?

sure

I think I did that right: https://github.com/jreback/pandas/pull/8

yep...you did gr8

I merged it into the PR, which I didn't need to do as that commit just shows up on the PR itself. So then rebased it away (and had to add in a release notes entry). but all good.
BUG: Fix ImportError on Python 3.4b2 and 3.4b3
State: closed

See http://bugs.python.org/issue20166

```
ERROR: Failure: ImportError (dynamic module does not define init function (PyInit_json))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "X:\Python34\lib\site-packages\nose\failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "X:\Python34\lib\site-packages\nose\loader.py", line 402, in loadTestsFromName
    module = resolve_name(addr.module)
  File "X:\Python34\lib\site-packages\nose\util.py", line 311, in resolve_name
    module = __import__('.'.join(parts_copy))
  File "X:\Python34\lib\site-packages\pandas\__init__.py", line 44, in <module>
    from pandas.io.api import *
  File "X:\Python34\lib\site-packages\pandas\io\api.py", line 7, in <module>
    from pandas.io.excel import ExcelFile, ExcelWriter, read_excel
  File "X:\Python34\lib\site-packages\pandas\io\excel.py", line 14, in <module>
    from pandas import json
ImportError: dynamic module does not define init function (PyInit_json)
```


Labels:
name: Build
description: Library building on various platforms
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
cc @Komnomnomnom

@cgohlke thanks 

we don't test py3.4 yet (soon though)

Noted, thanks for the fix @cgohlke! I'll try and get this pushed upstream to ujson, as I imagine they'll have the same problem.

@Komnomnomnom n.b. we don't actually have a python3.4 build yet (soon though)
DOC/PY3: Python 3 docs example compat for range, StringIO and datetime
State: closed

related is #6212

Fixes examples in documentation to be compatible with Python 3. Also tested it locally under Python 2.
- Turned occurrences of `(x)range()` to `range()` for iteration and to `list(range())` for  list concat.
- Fixed leading zero argument(s) as in e.g. `datetime(2013, 01, 27)`.
- Removed explicit `StringIO` and `cStringIO` imports in examples (uses suppressed pandas.compat imports) and added a note on py2/3 library reorganization.
- Fixed `StringIO` for binary data to the more explicit `BytesIO`.
- Fixed `NameError` for missing `pd.` in `pd.Series`.
- Added note for failing `numpy.unique` example, see also #6229.  

I am not sure about the `BytesIO` though. Maybe good to put a note saying this is Python 3 way to do it and users on 2.x should use the old `StringIO` instead. In general, should we favor Python 3 doc versions in such cases and add notes for Python 2.x users? What do you think?


Labels:
name: Docs

Comments:
seems ok to me...@y-p ?

Is the conversion of `xrange()` to `list(range(..))` necessary? Why not just to `range()`? 
This works both on both python 2/3 in a list comprehension, and seems easier.

as long as the `pandas.compat` imports are limited to wrappers for known classes
it's not a problem. Looks fine.

Wherever the code does not explicitly require a sequence for random access e.g. foo[4],
the `list(range(foo))` is not required as joris pointed out.

yep you can just use range (if u r iterating over it)

use list(range) if you actually need a list

Adapted.

If this looks ok to you now, I could go ahead and squash...

go ahead and squash 

done.

@phaebz , please get rid of all the `list(range())`` where not required, I think that's
everywhere.

@y-p regarding the PR, I think it is OK now. The remaining list in `list(range(1,24))+[np.NAN]` is needed for being able to add the nan. 
But elsewhere in the docs there will be still some unneeded `list(range())`s, but that can maybe be left for another PR.

You're right, good. merging.
TST: run clipboard tests explicitly
State: closed

closes #6317


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
@y-p u ok with this ... very simple

Thanks. It's quieted down and it seems correlated with me logging in after all. 
Let's wait, it might have been me ruining things. disabling tests is a last resort.

ok np

Looks liks the errors were caused by logging in during building. closing for now.
The race window is real, but probably very small in practice.

@jreback Here's the PR.

righto thabks

@cpcloud can you rebase this...these fail a lot on windows too....let's just disable
BLD: Cut Travis-CI build times by 30-50%
State: closed

 and some travis.yml cleanups after recent bug fixes by the travis guys.


Labels:

Comments:
just perusing Travis docs

do we have fast finish set?

we could create a build(s) that is allowed to fail (not because we expect it to fail but because it would take more build time)

possibilities:  

2.7 with numpy dev, plus no versioning of most packages so they get the latest releases
the wheels are not useful here (maybe exclude scipy/bottleneck as it takes a long time to build)
thinking latest of tables/Numexpr/dateutil/pytz things that have suprised in the past

python 3.4

What's the benefit of an allow-fail in that case?
The only issue is the 5 concurrent build limit on travis. If we add one more build, we might
as well add 5 more.
Kepping iterations fast really matters. a 25 minute build (we've been there, before wheels),
means a DOC commit could mean the following one could take 50 minutes to green. 
You might add the additional builds only in the run up week or two to a release. That would
be quite a good thing. If you want to add more regardless, that's ok too. I might grumble
ocassionally, no more.

cc @cpcloud , some numexpr wheels are missing, If you get a chance...

Finally. 

48 min -> 26 min

:winning:

and nice to finally see green days are here again. I thought we'd never get rid
of those bogus network errors.

awesome!

Reinvesting the time gained into extra coverage actually makes a lot of sense.
We were in an ok place before this (~10min), if we can stay there with more covered 
that'd be good.

in 0.14 would be prudent to have an sql heavy build
for various flavors (to the extent we have specific code ) eg SQLite , Postgres 

Very true. We need to sql special attention in 0.14.
BLD: setup.py version strings. handle shallow clones and installing from sdist
State: closed

Travis exposed a corner case in the handling of version strings since the `git-describe` shakeup.

Previously (0.10.1), version strings were of the form:

```
v0.10.1 # release
v0.10.1.dev # dev, no git available
v0.10.1.dev-abcd1234 # dev, git available
```

Now that we use git-describe, here's the case breakdown:
- release
- dev, no git 
- dev, no git and pandas/version.py exists (installing from sdist) # *
- dev, git available
- dev, git but shallow clone (no tags) (travis) # *

the \* cases were hard to see coming  (I didn't) and so the current logic does not handle 
them gracefully, and weird version strings result.

Behavior summary following this PR:

```
v0.13.1 # release
v0.13.1.dev # dev, no git available, matches old behavior
v0.13.1.dev-abcd1234 # dev, git available but no tags (pip tarball), matches old behavior
foo # dev, no git available, pandas/version.py exists and contains foo
0.13.0-509-ga134eb6 # dev, git available, tags available (normal dev situation), normal new behavior.
```

Slightly dangerous to merge shortly before a release, but it would be worse
to keep this brokenness for months until 0.14 IMO.


Labels:

Comments:
On second thought, only affects dev version strings anyway, pushing to 0.14.
CLN: PEP8 cleanup of rpy/tests
State: closed

Fixes pep8 errors in test_common.py in rpy/tests.

Ran autopep8, then fixed remaining errors manually. Test still passes locally.


Labels:

Comments:
Should this issue be closed? Can revisit another time, if desired.

yeh....pep8 are a complicated issue.....generally we will just do late in the release when noone else is touching code....so closing for now....but can revist later
BUG: dont ignore use_index kw in bar plot
State: closed

Closes https://github.com/pydata/pandas/issues/6209

Here's the fixed output for the same example from the issue: `df.plot(kind='bar', use_index=False)`:

![line_false_fixed](https://f.cloud.github.com/assets/1312546/2053850/603f044e-8aad-11e3-86e3-3f3bf38a5683.png)

I thew it in `.13.1`, can push to `.14` if need be.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
ping we when its green.....

@jreback I had a `u` prefix that python 3.2 didn't like. Green now.

EDIT: hmm, I was looking at a different build I guess. Still running.

looks green to me....merge?
DOC: add links to other pages to contributing.rst
State: closed

Awaiting for that the different developer pages could be reworked and added to this page, I thought to at least add some links here. So if people go from the docs to this page, they get redirected to the relevant pages.


Labels:

Comments:
@jorisvandenbossche do we have an open issue to incorporate the developer pages here (and then link back on pydata)? if not ...let's create one

Not yet I think, will do now. Merging this.
BUG: bug in nanops.var with ddof=1 and 1 elements would sometimes return inf rather than nan (GH6136)
State: closed

closes #6136


Labels:

Comments:
@y-p I could not reproduce, but this seems likely the problem

Thanks. Could this be a numpy issue? should we ping someone there?

It comes down to precision I think;

The following 2 cases end up slightly different if the numerator is exactly 0 or not (which is why it was 'randomly' failing)

I changed it so it doesn't do the computation if the denomincator is 0, problem solved.

```
In [9]: Series(0.,index=np.arange(5)).values/0.
Out[9]: array([ nan,  nan,  nan,  nan,  nan])

In [10]: (Series(0.,index=np.arange(5)).values+0.1)/0.
Out[10]: array([ inf,  inf,  inf,  inf,  inf])
```

That makes sense. Still, the same data behaves differently on the _same_ platform
and numpy. You have to cut fp some slack, but is this really an instance of that?

I'll open a numpy issue just in case. My numerical-fu isn't good enough to call it.

not sure I never could reproduce
even with the same exact specs

are you on conda? they use MKL for numerics.

this was a grohkle 1.8 MKL build for windows

Same here (didn't realize those were compiled against MKL). strange.
small py3 fixes
State: closed

@y-p correcting those small fixes I mentioned in the comments of your commit.


Labels:

Comments:
Thanks, can you test 3d155cc on PY3?

Works, thanks! :+1:

Wait, so the docs compile on py3 now?

Yes, they do with the latest master.

(I see you added the fixes yourself)

I thought the code examples were not py3 compat?

Anyway, I'll test doc code fixes on PY3 in the future as to not break things again. 
Thanks for reporting.

Yes, the code examples in the documentation are still not py3 compatible yet. I am just talking about the doc build process here.

OK, thanks, it would be great if you guys tested any changes to these sphinx extensions and the make.py script on Python 3 in the future (so that we don't have to redo the same work).
BUG/TST raise a more detailed error when GH6169 occurs, added a test
State: closed

Raise a detailed error when a `columns` argument is passed through 'where' to
select a multiIndexed Dataframe from an HDF store. Wrote a test
showcasing the bug

```
modified:   pandas/io/pytables.py
modified:   pandas/io/tests/test_pytables.py
```


Labels:

Comments:
closing in favor of #6202 

@glyg I took your tests and fixed the issue, was a bit non-trivial as the column selection is done in a non obvious way

thanks!

you're welcome!

and as always...pls report any further issues!
CLN/TST: #6175, specializing assert_ (WIP)
State: closed

Helping with the specialization request. Work-in-progress. Updated tests pass locally.


Labels:

Comments:
Looks good. I'd merge on green but will wait until 0.13.1 is released, out of caution.
Please have a look at guidelines on commit subjects in [https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md](CONTRIBUTING.md). If you're comfortable with git,
you may amend, if not leave them as is and we'll take care of it.

If you wish to do more, I suggest you open a seperate PR with more work. 
Keeping PRs small makes it easier for us to review and merge with confidence.

Thanks for contributing.

Thanks for your comment, y-p. My apologies on the mistake; I am still learning my way around git. I will open another PR with more on this topic shortly, hopefully that goes more smoothly.

On the contrary, if that's the case you've done very well.

https://github.com/pydata/pandas/pull/6198 superceded this.

@bwignall, it's good practice to always start a new PR by branching against the current
git master. If you include the commits from one PR in another (sometimes one PR depends
on another) please explicitly state that you've done so and why, or close the superceded
one if it has been.

Git takes some getting used to, don't beat youself up. Thanks again for contributing.

@y-p,

My apologies (again) for the elementary mistakes. I appreciate your patience and your suggestions.

I am happy to help. I will endeavor to do so more efficiently in the future.
ENH: Import testing into main namespace.
State: closed

Unfortunately I won't be able to use this in production until 2020 when we require 0.14, but this would make some things a bit easier for projects that use pandas as a library.


Labels:
name: API Design

Comments:
what r u going to use this for?

Testing without an extra import and using the make\* functions interactively for examples, bug-reporting, etc.

E.g., we need the pandas test function for testing DataFrame equality. 

Incidentally, I started to add to this suite some useful functions (DataFrame approximately equal and allclose) until I found that these functions are undocumented and most are written in Cython. Will have to stick to 3+ function calls to test until I find some time for a PR.

their is a new `.equals` function which should work for you (handles all the nan vs. nan issue)
(also `array_equivalent for arrays`); could be modified with a precision i think

http://pandas.pydata.org/pandas-docs/dev/whatsnew.html#api-changes

For precision issues cf. `np.testing.assert_allclose`. Would be a good target. I don't know what `array_equivalent` is.

Hmm, equals is an interesting addition. Still nice to keep testing functions in the test namespace, though right?

`equals` is not really for testing (though it may be used that way). Its a first class function; comparing frames with nans for equality is not possibly right now (without the expressions equals uses); and object arrays are a disaster. see the examples.

why do you want to import testing to the main namespace, can't you import into statsmodels ? (if that's where you are using I assume)

Numpy imports testing into the main namespace, and I got tired of typing two imports for pandas everywhere. Also for bug reporting. Everyone comes up with there own `pd.DataFrame(np.random.random...`. Would be simple to just point people to pd.util.testing.make\* functions. What's the harm? The kitchen sink is already imported in the `__init__` and it doesn't add another actual namespace to the main one.

ok...np then....do you want in 0.13.1 (ok with me)...if so go ahead and merge

Thanks. Doesn't really matter to me. I'm usually on dev version, and like I said I won't ever actually be able to use this in production until our min version is wherever it ends up (which feels like never these days...). 

ok...let's wait for 0.14 then

@jseabold , I thought you were an academic. What constitutes "production"? is there
a GMM capture-the-flag colloquium you run or something?

@jseabold don't you want to do (in `__init__`)

```
from pandas.utils import testing as testing
```

as I presume them you want to do

```
panel = pd.testing.makePanel()
```

?

I was leaving it in the util namespace so that the changes are really very minimal, but flat is better than nested, and I'm fine with it if y'all are. 

@jseabold this is fine
append to existing table with mulitindex fix
State: closed

Appending to an existing table with a multiindex failed as data_columns is set to [] but compared with None


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore

Comments:
looks fine
can u add a release note for 0.13.1 (ref this pr as it doesn't have an issue) and squash 
also mark in the code what u r testing

@wabu beautiful....you can do more of these!
BUG: long str unconvert fix 
State: closed

When storing long strings in hdf5, they get truncated at 64bytes when converted back to strings. The fix computes the itemsize of the strings and uses it before converting to strings. It would be possible to use the attribute information of the table, but that would require changes in the calling code.

Test that failed without the patch is included.


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore

Comments:
this was a fixed bug in numpy 1.7.2
https://github.com/numpy/numpy/issues/2485

needs a perf check

iirc this was an odd bottleneck

course it only matters if the data is encoded

I did an update on numpy and it fixed the issue. If you think the I should pref check the patch for numpy <= 1.7.1, I could do this, otherwise you can close the pr

no...I like this PR!

I don't think its actually an issue (the issue is in decoding strings, which is why I used this method in the first place which is quite fast). True you have to figure out the max length of the string...

tell you what...why don't you put a conditional based on numpy <= 1.7.1 and then do your way, otherwise leave it? (use LooseVersion)

I did the perf test, nothing changed significantly, and also implemented the check for the numpy version 

Did the changes and added release notes.

This change is actually very tricky and subtle because of exactly 3.3 / 2.7 interact with encodings....

still failing on 3.2 on 1.7.1.....

https://github.com/jreback/pandas/commit/6c268de1be38a239e9a114aaf1723a85e44146f2

I am not sure how to specify a dtype on 3.2 like 'S100'....maybe you can figure this out?
its essentially a unicode dtype

what about `np.dtype((str,itemsize))`? Results in '<U{size}' for python 3 and '|S{size}' for python 2, which seems to be ok?

do u have the ability to try in py3.3 on windows
that was the one throwing the error originally (weird but not on Linux)

u can use the just released 3.1 and try modifying the source 
as compiling in windows is a bit tricky

if u want to try compiling on windows I can help though

can't help with this, have no windows machine around.

ok no problem
pls update with the encoding tests and your change and I'll give a whirl in a couple of days

I did a PR on your branch with some fixes to make this work on windows: https://github.com/wabu/pandas/pull/1

@wabu pls incorporate my changes when u can

@wabu can you incoporate the changes and rebase?

closing in favor of #6821
4304r if_exists == 'fail' should actually work
State: closed

fixes #4110 #4304 
#4304 rebased and passes travis.

cc @y-p


Labels:

Comments:
Did you review the code for sanity?

I haven't looked at the SQL code. ever.

@hayd looks good....(and nice that you added some more tests).

Must take care to not drop this when merging sql stuff in 0.14.

cc @mangecoeur will see this on rebasing....

Need to test this with the new sql branch. Will come up when bring back the old sql test file... (which atm I can't get to pass...)

@hayd Can you take this on? To resurrect the old test suite?

@jorisvandenbossche @hayd I thought I fixed this with the new code? Pretty sure it's in the new tests too - it should test each allowed value of if_exists. Should also raise ValueError if invalid if_exists arg passed, though could probably do with more robust checking in code and tests

@mangecoeur It is indeed already tested in the new tests. But that doesn't mean that we shouldn't try if the old tests still pass?

@jorisvandenbossche I got the impression on reviewing the legacy tests that they were a bit of a mess, including testing undocumented APIs such as the "retry" option. That was why i ported the coverage to the new test architecture rather than extending the legacy ones - i tried to keep the test concepts rather than the test code.

OK, fine with me. I think @hayd is more familiar with the previous code base/tests, so maybe he can give a final tought on that.