BrandonFors/pandas-github-issues huggingface dataset:
Better treatment for complex numbers
State: closed

This fixes #1083.  I checked that the number of failing tests (SKIP=38, errors=5, failures=34) do not increase with these change sets.

BTW, in form_blocks why not simply do

``` python
    if float_dict:
        ...
```

instead of calculating length?:

``` python
    if len(float_dict):
        ...
```


Labels:

Comments:
Oops I have NOSE_DOCTEST_TESTS=t and NOSE_WITH_DOCTEST=t and that's why I had failing tests.  There is no fail/error when I remove these envvars.

The bug I fixed in bae81555be2c16cb191b5c9f2b9f9f305eea53d5 is very serious one and it could have effect even without complex number implementation I added.  It is probably a bug in numpy.  Try the following code and observe sorting fails.  I checked with numpy 1.6.1.

``` python
>>> from numpy import dtype
>>> sorted([dtype('bool'), dtype('float64'), dtype('complex64'),
...         dtype('float64'), dtype('object')])
[dtype('bool'),
 dtype('float64'),
 dtype('complex64'),
 dtype('float64'),
 dtype('object')]
```

BTW, I am using complex64 here but it should be complex128, right?
I noticed just now.

Yes, should be complex128

Merged this in. Should make a sweep and change complex64 -> complex128 at some point, though
datetime stuff on python 3
State: closed

See issue #1066. This is enough to get pandas to import in Python 3; I've not yet tried beyond that.


Labels:

Comments:
Thank you, Thomas, this is excellent. 

A few more fixes from digging into the tests, but I'm not yet making much headway with the tests (98 errors with the non-slow tests).

Latest finding: in some circumstances, the index code uses `np.searchsorted` when the type of the search term doesn't match the index type (e.g. searching for an integer in a string index). This fails with a TypeError on Python 3, because different types aren't orderable by default.

I had a real blast refactoring the time series code today. May have to do some of the transcription here by hand if it doesn't merge properly. 

I've rebased this, but I've realised the problems I'm still seeing aren't caused by Python 3. I think there's an integer overflow on my 32 bit platform, where IIUC `long` is only 32 bits. I'm trying to dig this out - if I get it, I'll make a separate PR.

Can't fix it at the moment (I don't do much with C/Cython), but I'll file an issue for it.
BUG Deprecating use of bool() on DataFrame, GH #1069
State: closed

Calling bool() on DataFrame raises issues described in #1069. This
change causes DataFrame to raise a ValueError when called with bool(),
and instead provides a new property, dataframe.empty.


Labels:

Comments:
This will cause client code to break, but I think it will prevent more bugs in the long run. I'll include it in the upcoming 0.8.0 (May, sometime)
fixing the error in io/sql.py
State: closed

Not able to find an testing module in for io/sql.py. And I don't have time to create one now. So I paste the testing code below, which can be added into pandas later on.

```
#!/usr/bin/env python

from pymysql import connect
from pandas.io.sql import read_frame

con = connect(host='genome-mysql.cse.ucsc.edu', user='genome', db='mm9')
sql = 'select geneName as symbol, name as refSeq, chrom, strand, txStart, txEnd from refFlat limit 5'
df = read_frame(sql, con)
print df
```


Labels:

Comments:
Hi pengyu, you can check out /pandas/io/sql/tests/test_sql.py for the testing module for sql.py. Thanks.

I see the following test code for testing read_frame. Yet, it can't catch the error that I fixed. I'm not familiar with the testing in pandas. Could anybody improve the testing code so that it can catch this error?

```
 36         result = sql.read_frame("select * from test", con=self.db)
```

instead of passing in self.db you can use your own con object.
if you search around in the file you'll see that self.db is part of the initial test setup.

You misunderstood what I mean. I don't understand why line 36 did catch the error.

I tried my testing code on sqlite3, it works the same as pymysql and MySQLdb. So, theoretically speaking, even the testing code in line 36 is for sqlite3, as long as there is converge for the error, it should catch it.

But it seems that it doesn't cover the error. I can not quickly add some code (just based on sqlit3 instead of pymysql) to cover the error, as I'm not familiar with the testing framework.

Thanks. Changed this to only iterate over the result of `cur.fetchall()` if it's a non-list
Check for openpyxl
State: closed

I added a try/except catch for the lack of an installed openpyxl. The user gets a nice message that she needs to install it or to save the .xlsx file as .xls.
I also added a test for it and a test.xlsx file that I saved from test.xls.
Furthermore I noticed an ugly break of the DRY principle in ExcelFile.parse and gave an alternative with a nice dictionary that provides the handle for the required method depending on the 'use_xlsx' flag.


Labels:

Comments:
thanks a lot
adding ddof parameter back into std and var
State: closed

Issue #934


Labels:

Comments:
I just got bit by this, and I think it's a pretty serious bug. If someone will merge this, I will write some tests.

Just curious what your application is that needs population std measures?

I need sample std measures for calculating standardized (internally studentized) residuals in a linear model.

If that's the case shouldn't the default pandas behavior be correct?

No. I need sample statistics. In any event, this is a bug.

Oh, you need ddof > 1 in your use. Got it.

Yeah ddof = model degress of freedom in this case.

+1 Need this as well 

merged in and tested. need to update the docstrings. someday
Added support for read_fwf() as in R.
State: closed

I added a simple function like R's read_fwf().

It also made sense to refactor read_csv() and read_table() to redirect to a new function _read() which takes the parser's class as input. I needed this for the FixedWidthFieldParser which needs to override the TextParser in order to provide a custom reader for the fields. Please review; the tests pass.

Also, I made the column spec ("widths") 
- zero-based, and
- inclusive on both the from and to fields (e.g., a colspec of (1,2) has two characters)
  I'm not sure that's the best; I think in practice a lot of column format specs start counting at one, but I haven't done a survey or anything, feel free to change it.  Maybe we should support both.


Labels:

Comments:
A broader point about the behaviour: how should it handle non-ascii data, especially with variable width encodings like UTF-8?

At present, I think this will do different things according to the version of Python:
- In Python 3, it will decode the file using the specified encoding you pass to the file, and count by character across the line (i.e. `字` is one character). If you don't specify the encoding, a platform default encoding is used, and unexpected bytes are replaced with the replacement character.
- In Python 2, it will completely ignore the encoding parameter and count bytes, so `字` is three bytes (if the file is UTF-8).

I guess counting bytes is more consistent with what other tools mean by fixed width, but we should decode strings in line with the encoding parameter.

I've made changes recommend by Thomas. 
I've added more tests.
I've done some careful refactorings of the read_\* functions for clarity.
Tests pass.
Please pull.

OK, I've made a couple more minor comments. I'll like someone with a fresh pair of eyes look over this - pinging @wesm and @adamklein.

This all looks pretty good to me. I'll take it for a spin in the next couple of days and merge it in. Thanks for the PR!

Sweet thanks all for this, very nice work, merged it in. 

@blais you'll want to fetch pydata/master now and `git reset --hard upstream/master`  (where upstream is the remote for pydata/pandas) to fix up your master
scatter plot matrix
State: closed

I wanted an easy way to create a scatter plot matrix from a data frame. It doesn't have the labels along the diagonals like R's ipairs does, but I don't think that would be too much work if you guys preferred that. I saw there was a scatter_matrix function in tools/plotting.py, but I wasn't sure what that was for.

I made sure it displayed data with NA values correctly, but that's about as much testing I could think of.

![Example](http://i.imgur.com/BuZDm.png)

Update: I didn't notice issue #857, I guess if this is already in the works then please ignore this.


Labels:

Comments:
Thanks so much. I factored this into a method in pandas.tools.plotting while I figure out how to organize all this plotting functionality (I'd rather not have everything as instance methods quite yet)
Add sort_columns parameter to plot to allow unsorted plots
State: closed

I was using pandas for the first time and when plot()ing I noticed it was sorting the columns alphanumerically.

I felt like the ordering I had defined in the DataFrame was a pretty decent proxy for how I wanted the subplots to be ordered, so I think this should be an option so I can over-ride the sorting and have the plot go in the direction specified.


Labels:

Comments:
Looks good to me, but as it touches the API I'll let Wes have a look. Thanks for the pull req

Looks good to me

Awesome!

Also I didn't want to push my luck too much, but is there a reason that the default is sorting columns in a plot?  It certainly is a nice feature to have and I imagine for a lot of datasets it is a really helpful feature, but as a first time user it was strange to me that my plots were showing up in a different order than they were in my dataframe, especially given how wonderfully easy it is to re-order and take subsets of a dataframe.
Patched _safe_fetch to ensure a list of tuples is returned.
State: closed

When MSSQL is used via pyodbc, Cursor.fetchall() returns a list of pyodbc.Row instances.  This just casts the rows as tuples so they don't give a TypeError in DataFrame.from_records.


Labels:

Comments:
@aman-thakral i took an alternate approach in the above commit that will not result in performance problems for other use cases. this will be in 0.7.2--  let me know if you have any more issues

This is still happening as of 0.11.0, 

---

TypeError                                 Traceback (most recent call last)
/home/user/<ipython-input-29-ef2471aa99a1> in <module>()
      1 conn = pyodbc.connect(conn_str)
----> 2 psql.read_frame('select count(*) from table_x', con=conn)
      3 conn.close()

/usr/lib/pymodules/python2.7/pandas/io/sql.pyc in read_frame(sql, con, index_col)

/usr/lib/pymodules/python2.7/pandas/core/frame.pyc in from_records(cls, data, index, exclude, columns, names)

/usr/lib/pymodules/python2.7/pandas/core/frame.pyc in _list_to_sdict(data, columns)

/usr/lib/pymodules/python2.7/pandas/_tseries.so in pandas._tseries.to_object_array (pandas/src/tseries.c:67122)()

TypeError: Expected list, got pyodbc.Row
Bugfix: Couldn't subclass Panel
State: closed

I was trying to create a subclass from Panel and came into this snag. I'm not sure if there was a reason Panel was using a hard coded _constructor method, but I changed it to be more generic. I'm new to open source, so constructive criticism is welcome. Am I doing this right?


Labels:

Comments:
Welcome to open source! You're on the right lines, I've just highlighted a few potential problems. Other people who're more familiar with the design of panel may give you more suggestions. Remember, although the comments might come across as quite terse, we are grateful that you're using and improving the code.

Thanks for the feedback. I changed _constructor to return type(self), and the _constructorGroupBy thing was a consequence of sloppy search and replace, so I fixed that. Should I submit another request or keep this one?

Merged in. Welcome to the project!

Future reference, it's easier (on you) if you work in a non-master branch. When you update, you'll have to "git reset --hard <upstream>/master"

@joshuaar a more fundamental question is: why are you subclassing Panel? Often when people ask me about subclassing I talk them out of it (viz "favor composition over inheritance")
Adding links to change log bullets
State: closed


Labels:

Comments:
@adamklein thanks for the handholding.  Sorry for being a bit boneheaded about this.  Let's just keep my ignorant comments between you and me ;-)

No worries :)

I pulled the changes. In the future, it's helpful if you make the changes in a new branch, easier to rebase. Thanks!

Though not the end of the world, do `git reset --hard upstream/master` to get synced up to pydata/pandas 

Thanks for your patience!

Travis

On Mar 1, 2012, at 5:54 PM, Wes McKinneyreply@reply.github.com wrote:

> Though not the end of the world, do `git reset --hard upstream/master` to get synced up to pydata/pandas
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/845#issuecomment-4273930
References Linked
State: closed

Quick edit to make references to issues into links.


Labels:

Comments:
The other docs just use the trailing underscores e.g. GH220_ 

I've just used this convention.

Sorry if I'm not being clear - unless I'm missing something, you've only used the pointer "GH220_", but not the necessary link that the pointer "GH220" deferences.  If you look at another whatsnew.rst, you should see what I mean (at the bottom of the file are the full hyperlinks that get will get dereferenced by the sphinx parser)
Bug Fix for Issue #513 Fractional seconds lost in HDFStore for datetime indexes
State: closed

PyTables Time64Col is in fact a float64 alias so adding fractional seconds and using the correct dtype fixes this issue
#513


Labels:

Comments:
Looks pretty good to me. Will let Wes look before I merge it in...

indeed this looks fine. thanks a lot!

hi @lespaul, because you made a pull request from your master branch, you will want to `git fetch upstream` and then `git reset --hard upstream/master`
DataFrame_repr_html changes according to #772 discussion.
State: closed

cc @ellisonbg

Implemented changes as discussed in #772.
Not so crazy on the scrollable div. For now i left the div max sizes as they were, could use suggestion on _good_ values.

I don't use the notebook often, what i noticed just now is that in combination with the IPython notebook `get_terminal_size()` returns the size of the terminal where the notebook was launched, not the size of the notebook display area. Don't know if there is a way to get the notebook size.


Labels:

Comments:
I'm pretty sure we haven't implemented any way to get the notebook size - in any case, the window can always be resized later, or the notebook could be reopened on another screen.

@lodagro can you add an option to set_printoptions in the print configuration that will disable the HTML output (and any other notebook-enhanced reprs, eventually)? It may be as simple as raising an exception

By default notebook will use _html repr_, but you can do `set_printoptions(notebook_html_repr=False)` to revert back to _plain text repr_ in the notebook (available in this pull request). 

Oh cool, you are way ahead of me. thanks
New offsets
State: closed

These are the new offsets and frequency strings posted on the wiki.

I haven't included any deprecations because I'm not clear how this should work yet.


Labels:

Comments:
Quick question: one of your tests fails, 

datetime(2007, 1, 1) + MonthBegin(-1) is datetime(2007, 1, 1) and not datetime(2006, 12, 1) as you suggest in the test. I think it should indeed be datetime(2007,1,1) (b/c add with n<0 is rollforward and add). can you just confirm?

NM, I say the typo that led to the behavior (in MonthBegin, it was =+ instead of +=.  Lovely)  Now all the tests pass.
Issue #806 DataFrame/Series.align can now specify fill_value or fill_method
State: closed


Labels:

Comments:
you'll want to do work like this in a branch, see:

http://pandas.pydata.org/developers.html#working-with-the-code

After this gets merged you'll want to fetch upstream and do

```
git reset --hard upstream/master
```

so you get rid of those merge commits above. git takes some git-ing used to

Ahhh the reset step. Thanks! Will be up to speed soon :)
On Feb 23, 2012 2:08 PM, "Wes McKinney" <
reply@reply.github.com>
wrote:

> you'll want to do work like this in a branch, see:
> 
> http://pandas.pydata.org/developers.html#working-with-the-code
> 
> After this gets merged you'll want to fetch upstream and do
> 
> ```
> git reset --hard upstream/master
> ```
> 
> so you get rid of those merge commits above. git takes some git-ing used to
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/807#issuecomment-4142848

Also, u get a slap in the back of the head for using a "git" pun :P
On Feb 23, 2012 2:08 PM, "Wes McKinney" <
reply@reply.github.com>
wrote:

> you'll want to do work like this in a branch, see:
> 
> http://pandas.pydata.org/developers.html#working-with-the-code
> 
> After this gets merged you'll want to fetch upstream and do
> 
> ```
> git reset --hard upstream/master
> ```
> 
> so you get rid of those merge commits above. git takes some git-ing used to
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/807#issuecomment-4142848

the test suite didn't pass with these changes-- I patched things up and looking good now
'Mixed depth' hierarchical column labels
State: closed

Modifications to column-hierarchical behaviour as per "'Mixed depth' hierarchical column labels" discussion on pystatsmodels.

If column label tuples contain only empty strings at the lower levels, allows the DataFrame to be indexed with only the higher levels. So for example if a column has the hierarchical label ('a','',''), and there are no other columns with 'a' at their top level, then this change allows the column to be pulled out with df['a'], rather than df['a','','']. It also names the resulting series 'a'. 
The use case for this is when some columns have a deeper hierarchy than others, and you want to treat the shallower columns as if the lower levels of the hierarchy weren't there.

If column label tuples contain only empty strings at the lower levels, allows the DataFrame to be indexed with only the higher levels. So for example if a column has the hierarchical label ('a','',''), and there are no other columns with 'a' at their top level, then this change allows the column to be pulled out with df['a'], rather than df['a','','']. It also names the resulting series 'a'. 
The use case for this is when some columns have a deeper hierarchy than others, and you want to treat the shallower columns as if the lower levels of the hierarchy weren't there.


Labels:

Comments:
Could I ask you to add a couple of tests? `pandas/tests/test_multilevel.py` would probably be the right place

Sure, it will take me a short while to figure out the test framework, but as I'm sure you'll attest it's something I should be doing anyway!

Well, there's not much to it. You already did the hard part which was modifying the code so that your use case works-- so take your test case, put that in a test function, and throw in some asserts that the results are what you expect

Ah, sorry, earlier commits broke other tests. All tests now pass...except for `test_daterange.TestDateRange.test_with_tzinfo_ambiguous_times`, but that fails for me without my changes.

thanks Chris, this looks great. I imagine that test is failing because of missing or outdated pytz

Cheers! Thanks for including it.
Handle subclasses of dicts in DataFrame constructor, with tests
State: closed

Minor change to allow passing subclasses of dict into DataFrame constructor.  I assume there is no way to pass these directly to cython, we have to make a copy first?


Labels:

Comments:
I think the solution is just to use the regular pure Python alternative to `fast_multiget`. I'll make a quick patch to avoid the copy-- surprised I don't have an initialization from `collections.defaultdict` somewhere already in the codebase

alright, played some tricks to not have to copy the dict and added a defaultdict test also for good measure
BUG: code to address #742, DataFrame.rank
State: closed

#742


Labels:

Comments:
Hmm, I'm not sure this needs to be so complicated. Why not just have a `numeric_only` flag (default True, unless all the data are non-numeric) and call either `rank_2d_generic` on the values (if it happens to be `dtype='O'`)? If you do numeric_only=False then it will attempt to call whichever `rank_2d` method on the values matrix is appropriate (and letting failure raise an exception if there are incomparables in the DataFrame)

This solution is also more consistent with other methods that have a numeric_only flag

About your comment - I wondered about this - but what if some object columns/rows are rank-able and others are not? Won't it then fail unnecessarily (during the 2d argsort/sort in numpy)

In that case it _should_ fail. I'm saying the logic should probably be
- `numeric_only=None` (default)
  - attempt to rank. if fail only rank numeric data

otherwise if numeric_only=True use only the numeric data (current behavior) otherwise only use all data (and fail if there are any incomparables)

Makes sense, will make those changes
Prototype string functions
State: closed

This is a prototype for functions to work with series of strings. There's a lot more to be done if we like the approach, but I wanted to check in before I spent too much time on it.

You can easily map a function onto a series:

```
In [4]: s = Series(["hi","hoot", "far"])

In [5]: strings.startswith(s, "h")
Out[5]: 
0   True
1   True
2   False
```

You can also use other series to give arguments; they're not reindexed to the main series yet, but this is NaN aware:

```
In [6]: s = Series(["hi","hoot", "far", np.nan])

In [7]: strings.search_re(s, "h(.*)", groups=Series([0,1,1,0]))
Out[7]: 
0    hi
1    oot
2    NaN
3    NaN
```


Labels:

Comments:
I'll go ahead and merge this in. I'm going to move the strings modules to the `tools` subdir, I think. We'll want to start building a test suite, of course, to permit refactoring

OK, thanks. I wasn't actually expecting this to get merged as is. I'll find time to improve on it.

Does little harm-- I want to hack on it too sometime in the next few weeks
BUG: resolve left/right key names overlap in merge
State: closed

Here is an attempt at fully resolving name conflict in merges when `left_on` or `right_on` keys are passed explicitly.

I am not sure my fix is correct but I include a new non-regression test to highlight the issue and the resolution does not introduce any regression findable by running `nosetests`.


Labels:

Comments:
Rebased & merged

Thanks!
Pass figsize into _grouped_plot -- please review
State: closed

To still allow default matplotlib size for the figure I added handling of figsize="default"... may be altogether it should just drop custom default size here?

also another commit on **kwds -- depends on what was intended, but at least figsize should be passed 


Labels:

Comments:
Thanks. I also pulled in the pyplot.subplots function to improve compatibility with earlier versions of matplotlib.

you'll want to reset to upstream/master (since you did pull request from master) but since you're a git guru you probably already know that ;) 

THANKS!  yeah -- I hope to figure my ways around -- thanks for the note ;)

I think you have missed to adopt 837854554bbb4b3076b13860cc7335ca0cd60f5d  which took care about passing figsize from DataFrame.boxplot into plots.boxplot (actually it just was to pass whole **kwd)

Oops. Thanks
Make _get_terminal_size_linux() work inside Emacs
State: closed

I'm using ipython inside Emacs. Detection of the console width and height is not working with the current implementation. ioctl seems to return (0,0). Therefore, performing an explicit check for cr == (0,0) (in addition to the check for not cr) before falling back to environment variables is a quick fix that I think shouldn't make any negative impact on non-emacs users.
I've also added the import of os.environ line since env previously wasn't defined.


Labels:

Comments:
thanks! Appears not to cause any issues