BrandonFors/pandas-github-issues huggingface dataset:
FIX sql handling of timedelta64 columns (GH6921)
State: closed

Closes #6921

Converting to an int was actually also how it was now with sqlite DBAPI fallback (as timedelta64 is subclass of integer), so this solution is also backwards compatible and consistent between sqlalchemy and fallback mode.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Timedelta
description: Timedelta data type

Comments:
@danielballan @jreback 

don't you need to explicity cast before you write to the db itself? (e.g. doing `s.astype('int64')`, essentially converting it to an integer column)

Not really necessary, as it is an int subclass, and in the sql write code `np.asscalar` is used to convert the numpy scalars to python types:

```
In [29]: np.asscalar(np.timedelta64(10000, 'ns'))
Out[29]: 10000L
```

Ah, and it seems I have to skip the tests for older numpy versions. Updated.

looks good

def put this as a warning in the docs though

OK, added a note in the docs.
Fix Xl(sx|wt)Writer always using date_format even if a datetime is supplied.
State: closed

In the `XlsxWriter` and `XlwtWriter` implementations there is a minor bug resulting in the `datetime_format` being discarded (i.e. https://github.com/pydata/pandas/blob/master/pandas/io/excel.py#L662):

```
if isinstance(cell.val, datetime.datetime):
    num_format_str = self.datetime_format
if isinstance(cell.val, datetime.date):
    num_format_str = self.date_format
```

Since `datetime.datetime` derives from `datetime.date` the second `if`-clause in the  will always trigger, leading to the `num_format_str` being overwritten by the `date_format`.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO Excel
description: read_excel, to_excel

Comments:
can you add a test for this? 

pls add a release note as well (reference this PR) as this doesn't have an associated issue?

Adding a test is difficult, as this only shows in Excel itself (by showing `2014-05-08` instead of `2014-05-08 15:23` in a timestamp column). In particular rereading the file will give proper `datetime` or `Timestamp` objects again.

I'll add a release-note.

ok then

ping when pushed

Ping.

thanks!
BUG: PEBKAC bug in Float64Index
State: closed

closes #7066


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@jreback ok with this? passed on my travis 

can I add a tests of a 2 elent index that has a nan and a value and uses isin (and then check that the isin values is in and another when it's False)

you add (typo)!

I saw u r giving. talk on Pandas!
cool - u have slides? what r u talking on?

Topic is all the awesome features of pandas since v0.11. IPython notebook in the works. 

sounds awesome!

will see if I can make it

That'd be great!

@jreback good 2 go? test added

looks good
Added dropna to docstring for HDFStore.put()
State: closed

HDFStore.put() method accepts dropna boolean parameter as it is passed
directly to the internal _write_to_group() method, but it was not
explicitly documented (as it is for append).  Also settable via option.
Just added 2 lines to the put() method docstring to mention it explicitly - copied from append()


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
great.....can you squash to 1 commit: https://github.com/pydata/pandas/wiki/Using-Git
otherwise looks fine

you didn't need to close the other and open a new one

you can simply force push which overwrites the commits (this is what squashing then pushing with `-f` will do, FYI)

@davidljung did you close this?

@jreback - yes - I didn't realize.  I'll try to get it right when I have another moment.  Obviously, fairly new to GitHub and my attempt to squash from your link doesn't appear to have worked.  I'll figure it out. (Sorry for so much trouble for a 2-line change - but I'll know for next time)

hah!

git-fu is tricky at first.....that's why its nice to figure it out on a smallish change!

you can reopen one of these and just force push to it

gr8 thanks!
Added dropna to docstring for HDFStore.put()
State: closed

HDFStore.put() method accepts dropna boolean parameter as it is passed
directly to the internal _write_to_group() method, but it was not
explicitly documented (as it is for append).
Just added 2 lines to the put() method docstring to mention it explicitly.  


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
I wondered about that, since the append() method looks to have logic specifically looking at that option, but the put() method does not (- I've not actually tested it nor dug into the methods it calls to see if it was picked up there)

no they go the same place, guess this wasn't documented :)
Remove tzinfo from datetime before writing cell with xlsxwriter
State: closed

fixes #7056


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel
name: Timezones
description: Timezone data dtype

Comments:
see here: https://github.com/pydata/pandas/wiki

you need a test that fails w/o the fix and passes with
it will test in all excel engines

put in io/tests/test_excel.py (just copy the test format)
add the issue reference 

@sboehler tests?

Apologies, I don't think I will find the time to create tests, I guess I
should pull back the request.

2014-08-05 19:11 GMT+02:00 jreback notifications@github.com:

> @sboehler https://github.com/sboehler tests?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/7060#issuecomment-51228638.
DOC: cleanup vis docs
State: closed

DOC: Give KDE a section header

DOC: remove extra slice in hexbin example

DOC: move notes to look nicer

A few changes that came up in https://github.com/pydata/pandas/pull/7027


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
OK!

:+1: 

seems 'pie' is missing from the Other Plots ?

also maybe make it `bar` (with the link), or `barh` (with a link to the same place)
w/o the quotes? 

http://pandas-docs.github.io/pandas-docs-travis/visualization.html#other-plots
API: update nth to use the _set_selection_from_grouper makes first==nth(0) and last==nth(-1)
State: closed

closes  #6732

`nth` sets the index appropriately and the same as first/last.

this becomes less like head/tail in that `as_index` determines when you have an index

here's the revised behavior:

```
In [1]: df = DataFrame([[1, np.nan], [1, 4], [5, 6]], columns=['A', 'B'])

In [3]: df
Out[3]: 
   A   B
0  1 NaN
1  1   4
2  5   6

[3 rows x 2 columns]

In [2]: g = df.groupby('A')

In [9]: gni = df.groupby('A',as_index=False)
```

```
In [5]: g.first()
Out[5]: 
   B
A   
1  4
5  6

[2 rows x 1 columns]

# this was a regression from 0.13.1 (in that before this PR, this was returning like ``as_index=False``)
In [6]: g.nth(0)
Out[6]: 
    B
A    
1 NaN
5   6

[2 rows x 1 columns]

In [7]: g.nth(0,dropna='all')
Out[7]: 
   B
A   
1  4
5  6

[2 rows x 1 columns]
```

```
In [10]: gni.nth(0)
Out[10]: 
   A   B
0  1 NaN
2  5   6

[2 rows x 2 columns]

In [11]: gni.nth(0,dropna='all')
Out[11]: 
   A  B
0  1  4
1  5  6

[2 rows x 2 columns]
```


Labels:
name: Bug
name: Groupby
name: API Design

Comments:
@hayd @jorisvandenbossche pls review for saneness

The question is a little bit if we see `nth` as a filtering function, or a reducing function? It looks like #6569 changed it be filtering, while this does revert it again to reducing?

I get it, the reason I 'changed' it was two-fold:
- `nth` is a _reducer_, much like _first_ and _last_, in fact first == nth(0)
- you can now pass `as_index=False` to get a filter, so this preserves the behavior while making the default a more natural, show me the nth element.

so to answer your question, this is really a reducer (as it can return at most 1 per group!)

`head/tail` can return _multiple_ per group (so they are filters)

@hayd ping!

@hayd any comments?

Looks good, thanks for putting this together, apologies for being AWOL of late!

glad 2 have u back!
ENH/BUG: partial string indexing with PeriodIndex
State: closed

Closes #6716.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Period
description: Period data type

Comments:
can you put a small example in v0.14.0.txt and in http://pandas-docs.github.io/pandas-docs-travis/timeseries.html (maybe need a sub-section `PeriodIndex`) with a `partial string indexing` section. 

also if you want to throw in some docs about creating PeriodIndex would be great

OK, I'll try. I found an issue in `Index` partial slicing, thus please leave it a while.

@jreback I think I've finished. Pls check.

Regarding the document, there is a section about creating `PeriodIndex` already. What is the topic should be added?

thanks...I think the docs are fine, you put them in the right place

that said, after this buidls on master, pls check out the built docs for correctness

This fails on numpy master for some reason, can you investigate: https://travis-ci.org/pydata/pandas/jobs/25073475
BUG: unstack fails in PeriodIndex
State: closed

Closes #4342.

Also changed `Categorical` and `PeriodIndex.factorize` to make index sorted like other indexes do.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Period
description: Period data type

Comments:
did anything change because of the sorting change?

@sinhrks fyi, in general ping me when you think something is ready for merging (esp if its marked for 0.14). try to make green! if its not, ok to ping as well and will take a look

Sorting change only affects to `Categorical` creation with `PeriodIndex`. Because:
- `Categorical` calls `Index.factorize` if it has the method. Currently, `PeriodIndex` is the only class which has the method. Thus, any other classes and indexes are not affected.
- `PeriodIndex.factorize` use `sort=False` as default (which derived from `core.algorithms.factorize`). Thus, any other function which use `PeriodIndex.factorize` will get the same result as before.

ok; looking at this code, I think you can  define `factorize` for `Index/Series` in core/base.py,

then just do the test `hasattr(labels,'factorize')`

ok, i think you can rebase this and then good?

or close after #7090?

Should be rebased, I will finish it soon.

@jreback Rebased, and get green.

thanks!
PERF: optimize Index.delete for dtype=object
State: closed

This should close #6933.

The downside of this patch is that we'll lose type inference that might change index type in rare occasions when index items have different dtypes. This is the same issue that occurred in #6440, and it was ruled out as infrequent and worth dropping for extra performance.

Note though, that the benchmark results are not exactly what I've expected:

``` python

In [39]: idx = tm.makeStringIndex(100000)

In [40]: timeit idx.delete(-1)
1000 loops, best of 3: 1.46 ms per loop

In [41]: timeit np.delete(idx, -1)
1000 loops, best of 3: 1.05 ms per loop

```

The result is faster, but not as fast as I'd expect. This is probably because of object refcounting overhead, because MultiIndex fares a lot better with its Categorial-like implementation:

``` python
In [42]: midx = pd.MultiIndex.from_product([[0], idx])

# <..snip..>

In [45]: timeit midx.delete(-1)
1000 loops, best of 3: 225 µs per loop

```


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
ok...looks fine...merge?

Yeah, it's ready.

I wasn't sure about combining release notes, but they did seem relevant. If that's ok, let's merge it.

yeh...combing is fine when pretty related
TST/CLN: centralize common validation methods in test_graphics
State: closed

Create base class for plotting related test, and organized data creation and validation function.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@TomAugspurger @cpcloud pls review

One general comment. Anywhere that you have a pattern like

``` python
for result, expected in zip(ax.lines, expected_lines):
    assertEqual(result, expected)
```

could you add a test at the start or end to ensure that the two things being zipped have the same length? I don't think we have any generators here, so this should be doable. For example, this would pass

```
In [74]: for k, v in zip([], ['a']):
   ....:     assert k == v
```

since zip only zips up to the shortest.

@TomAugspurger Thanks, I've modified except last point. Could you check?

Thanks for this. Good work!
TST: nose.SkipTest on RemoteDataErrors in tests for io.data.Options
State: closed

Prevents tests from failing if data isn't able to be downloaded from Yahoo Finance.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
looks good.pls squash down and add a release note (ref this PR), just so we can keep track of this

Squashed and added a release note.

make that small change and rebase; ping when green
BUG: fix reading multi-index data in python parser
State: closed

partial fix for #6893

The python parser has a problem reading data with a multi-index specified in the row following the header, for example

``` python
In [3]: text = """                      A       B       C       D        E
one two three   four
a   b   10.0032 5    -0.5109 -2.3358 -0.4645  0.05076  0.3640
a   q   20      4     0.4473  1.4152  0.2834  1.00661  0.1744
x   q   30      3    -0.6662 -0.5243 -0.3580  0.89145  2.5838"""

In [4]: pd.read_table(StringIO(text), sep='\s+', engine='python')
Out[4]: 
                           E
one two three   four        
a   b   10.0032 5     0.3640
    q   20.0000 4     0.1744
x   q   30.0000 3     2.5838

[3 rows x 1 columns]
```

(the C parser doesn't make it this far, see #6893)

This PR fixes the bug in the python parser:

``` python
In [4]: pd.read_table(StringIO(text), sep='\s+', engine='python')
Out[4]: 
                           A       B       C        D       E
one two three   four                                         
a   b   10.0032 5    -0.5109 -2.3358 -0.4645  0.05076  0.3640
    q   20.0000 4     0.4473  1.4152  0.2834  1.00661  0.1744
x   q   30.0000 3    -0.6662 -0.5243 -0.3580  0.89145  2.5838

[3 rows x 5 columns]
```


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv
name: MultiIndex

Comments:
ok, pls add a release note (even if it doesn't completely solve)

@jreback OK, release note added. OK to open another PR for the C parser fix or at least to detect the condition?

go ahead and open a new issue for the c-parser (reference the original issue too and this fix)

put the release note with the other changes for the parser (just move right below)
are their any tests for this? or were the others just disabled?

> go ahead and open a new issue for the c-parser (reference the original issue too and this fix)

would it make sense just to leave the original issue open, since this is only a partial fix?

> are their any tests for this? or were the others just disabled?

sort of... `test_read_table_buglet_4x_multiindex` was previously in `ParserTests` but I copied it to `TestPythonParser` in #6889 since it had been falling back to python. For the C engine we currently just test that it produces a `CParserError`. However the test only checks that the index is read correctly and so it didn't expose the bug fixed in this PR. Now I've added a bit to the bottom that does cover this (testing now, will commit soon).

ok...we can leave the original issue open, no prob

let me know when you have rebased and green

@jreback green!

thanks!

ok....so come back to the open issue at some point (its in 0.14.1), the projected bug-fix release
so have some time (prob 1 mo after 0.14)
API: adds weekday_common accessors (GH6936)
State: closed

closes #6936


Labels:
name: Datetime
description: Datetime data dtype
name: API Design

Comments:
prob nice to add these for DatetimeIndex as well

can you rebase this

Two remarks:
- needs some docs + docstring (how is it different from `week`?)
- is it necessary that we have both? I don't know why we have now both `week` and `weekofyear`, but if it is mere historical reason we maybe should not repeat that? (but if this was done for convenience, then it is OK of course)

In theory if `week` (I think `weekofyear` is just an alias) was a function, then this would be easier. But that just breaks everything. 

To be clear, with my last point I meant having both `week_common` and `weekofyear_common`, instead of just choosing one of the two. And not about adding this one (although I don't really like the name, but I haven't directly a better idea)

oh....then that's ok (I agree name needs some thought though)

@trailsquirrel can you rebase this (and move release notes to v0.14.1)

@trailsquirrel defering to 0.15.0. have to think about these names

@jorisvandenbossche what did you think about these names?

- I am not really liking the name `week_common`, as I think the ISO standard is maybe not always intuitive, but it is AFAIK the 'common' standard to give week numbers (eg used in business, governments, ..)
- For the more intuitive week number, why this convention and not another one?
  - I think this PR implements it as 1-7 january = week 1, 8-14 january = week2, irregardless of the day of the week?
  - another possibility would be to let each week start on Monday (or Sunday), but the week with firs tmonday (sunday) of the week- = week 1. And all days before that get week number 0. This is how `strftime/strptime` in the datetime module does it eg: https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior (`%U` and `%W`).  
    Just to ask the question: is there a common definition for a non-ISO week number?

closing as stale
BLD: use sqlalchemy 0.7.1 in 2.6 build (GH6340)
State: closed

closes #6340


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: CI
description: Continuous Integration

Comments:
@jorisvandenbossche 

ok.....good thing we tested!

this is 0.7.1 for sqlalchemy, and slightly older versions of pymsql and psycopg2

https://travis-ci.org/jreback/pandas/jobs/24212656

strange, the integer columns fail. We convert it to a string (https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L586), but for some reason sqlalchemy converts it back to an int. Have to investigate.

ok...my pr is up their...lmk when you want to merge (or you can simply take and rebase on fix ontop)
up 2 you

also ok with simply skipping that test on a too old version of sqlalchemy (seeing as its the only 1 that failed).

But possibly also a bug on our side (as here https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L520 we don't convert the column names to strings when constructing the dict which is passed to `execute`). Will look at it later.

In any case, say it is only this that is not working for sqlalchemy 0.7. Do we say they need 0.8, or just say 0.7 is possible but that does not work? Can we eg skip that one test when using sqlalchemy 0.7?

I would just say min in 0.7.1 and leave it at that

then skip this test and/or fix. good enough (in 0.14.1 can extend the testing if you'd like)

@jorisvandenbossche you are going to cover the build change in another PR, yes? or want me to keep this open

bumping this; for 0.14.1 can work this in (and test with MySqlDB as well), https://pypi.python.org/pypi/MySQL-python/1.2.5

@jreback Can you rebase this? Normally tests should now pass after #7456
Refactoring: DataFrameWidget(QWidget) -> DataFrameWidget(QTableView)
State: closed

I think that deriving DataFrameWidget from the QTableView is more logical than from QWidget


Labels:
name: Visualization
description: plotting
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
did you open 2 PR's fro this? #7021, #7020 pls close one of them

here are some git tips: https://github.com/pydata/pandas/wiki/Using-Git

Nope. I don't know why but when I did `git pull-request` via GitHub `hub` tool it copied the old message. In this pull request I refactoried some code.

Or I should wait until my first PR will be accepted? I'm not an expert in git yet...

ok, before you go further we need to create a test structure. That is important to retain backward compatibility.

I agree with you but I need to submit labs in mathematical modeling :smiley: So I'll write my code. When someone would get ideas how to unit test PyQt application I'll add tests.

https://github.com/ChrisBeaumont/qt-travis-test

OK, I'll try it later. I don't know how to deal with constantly changing version of PyQt. May be we should just install PyQt from repositories? I don't know how does Travis work but I think it should be possible. Other problem is: PySide, PyQt4 and PyQt5 should not be mixed. I don't know how to make several build environments.

they can be done on different builds. All I need are instruction to install from repositories and/or pip.

need tests for these, maybe should be taken out of pandas
PyQt 5 support added
State: closed

Added imports for PyQt 5


Labels:
name: Visualization
description: plotting
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
best to make this back-compatible; you should import the new libraries in a try: except: and rename them as needed to the old

can you add some tests?

I did it exactly as you said (via try except approach).

I don't now how to write tests for imports.

their don't seem to be any tests at ALL for anything qt..

ideally install qt4 and qt5 in separate virtual environments and test

I think we need a setup of baseline tests..so this is a bit more involved

can you post what is required to run qt (e.g. if I have a blank environment, what pip needs to install)?

I've version of PyQt5 installed from my openSUSE repos. The canonical way is on page: http://pyqt.sourceforge.net/Docs/PyQt5/installation.html, I've tried it, it works. I didn't try to install it through `pip`.

ok, the trouble is need to do this via travis. can you investigate? (IPython I am sure has some good resources on this as well)

http://docs.travis-ci.com/user/languages/python/

essentially you can only install really really basic things via apt-get as the travis boxes are 2.7 ubuntu. I'll bet their is an easy way to install qt though.

We CAN build it as a wheel (we do this for almost all libraries), that auto installs the dependencies.

OK, I've done little search in IPython repository and... didn't find any sign of Qt testing. Look: https://github.com/ipython/ipython/blob/master/.travis.yml (no Qt) and: https://github.com/ipython/ipython/pull/5458 (no tests).

It works, I guarantee it! :smiley: 

ok....maybe search github/travis for other projects....surely people do this!

>  Failed — The Travis CI build failed · Details

Hey! That's not my fault! I think master was broken before me. The error is not related to Qt at all.

you need to rebase on master, but their are no tests for this.

So need to fix that.

actually the 3.2 build already installs qt4, so just testing is needed (and qt4 would be straightfoward)

We (IPython) don't run the Qt section of our tests on Travis. I think it should be possible to install PyQt\* from apt, but I don't know if it can create the widgets without an X server running. xvfb lets you run a headless X server - maybe that works on Travis.

@takluyver thanks for the info!

we do run an `xvfb` (to test clipboard via `xsel` and such). so testing `qt` should be straightforward!

How is it going? Is the Qt testing infrastructure ready? Should I rebase and make another pull request?

@rominf You will need to create the testing infrastructure for this. I don't believe really any of this exists.

I've rewritten the current qtpandas model and uploaded it to my pandas fork:
https://github.com/datalyze-solutions/pandas/tree/qt4/pandas/sandbox

Its still in sandbox. An example can be found in tests named TestApp.py. I just testet Qt4 and python2.7, Qt5 might be interesting.

Any reviews are welcome! :smiley: 

See also #8788

closing as this would need a whole testing infrastructure.

this should prob be pulled out of pandas anyhow.