BrandonFors/pandas-github-issues huggingface dataset:
CLN: cleanup tox.ini, remove stale pandas/setup.py, fix tox warning
State: closed

I always find it hard to get Packaging issues right, i'd appreciate if someone 
would back me up on pandas/setup.py being superfluous.

I made sure that setup.py install/develop still works, and that introducing
a failed test is  caught by tox as you would expect,

Also see #2029 for much faster tox runs.


Labels:

Comments:
thanks!
BUG: fix Series repr when name is tuple holding non string-type
State: closed

Not sure if this is in line with latest unicode work, @y-p?


Labels:

Comments:
on git master the test fails with

>  File "src/pandas/pandas/core/format.py", line 88, in _get_footer
>    series_name = "('%s')" % "', '".join(self.series.name)
> TypeError: sequence item 0: expected string, int found

the test in b837101 already passes with (PR #2005).
as a side note, the PR excises stringify and stringify_seq in favor of a new `pprint_thing()`,
which should handle more corner-cases and is clearer about unicode/str issues.
a lot of that PR is migrating things over to using that, so it'd be nice if they weren't reintroduced by 
new patches if possible.

I vote to keep the test of course.

So what do we do? Is it ok that you add the test in #2005?

Done.  11dff0d  
note that I rewrite the PR as i go along to keep things tidy, so the SHA may change.

Ok, closing this one as it is redundant now.
Hopefully #2005 gets merged in soon, which will also close #2051.

Thanks. I'll review and merge #2005 as soon as I can
Uncentered
State: closed

@wesm can I get a second pair of eyes to skim over this PR real quick? All tests pass but touches a lot of core modules.


Labels:

Comments:
Should this be merged for 0.10? Since I moved around the cython code this won't merge cleanup but shouldn't be too hard to fix up

It's fine to wait. No urgency on this functionality. I'll merge it after the release

On Dec 1, 2012, at 10:36 PM, Wes McKinney notifications@github.com wrote:

> Should this be merged for 0.10? Since I moved around the cython code this won't merge cleanup but shouldn't be too hard to fix up
> 
> —
> Reply to this email directly or view it on GitHub.

will have to have someone pick up if really intereseted
Added hourofyear for pivoting calcualtions
State: closed

I do not know if this is possible. But very much needed.


Labels:

Comments:
AFAIK hoy is not an existing field. We'd love to merge a complete PR if you can implement it and add a test case. 
Thanks for the feedback. Please keep the feature requests/bug reports coming!

I am just trying to improve the pivoting of time series.

Especially:
#1953

and

#833 (#736)

But I feel quite lost as my array knowledge is limited.
Also:
Where's the source for 

```
import pandas.lib as lib
```

?

I have now the required function, purely python:

```
def hoy_dtobj64(dt_obj64):
    """
    calculate hour of year for a 'datetime64[ns]' object
    """
    doy = dt_obj64.dayofyear
    second = ((doy-1)*3600*24 + dt_obj64.hour*3600 + 
                dt_obj64.minute*60+dt_obj64.second)
    hoy = (second / 3600) + 1

    return hoy

```

Where shall we place it?

As an additional property of the index?

You're almost there :)

the definition for Timestamp resides in pandas/src/datetime.pyx
you can add that method in a similar manner as dayofweek/dayofyear etc.

Finally you'll need a few test cases to be added in to pandas/tseries/tests/.

mmh.

I must admit, I do not understand the code in pandas/src/datetime.pyx

I would need a helping hand.

my pure python function above is much more understandable..

stalled.

what means stalled?

It means "inactive for a long time".

let's close it then:
#2183
To csv infs
State: closed

This branch makes inf/-inf different from nan/None (GH #1919), and also fixes a few bugs with INF and NaN values (GH #2026, #2041).


Labels:

Comments:
I'll take a crack at merging this soon. Will have to do so tenderly because of all the merge commits =/ 

my apologies if I've made this overly complicated. let me know if i can clarify things and what to do differently in the future.

Generally merging master into a feature branch is discouraged because it makes merging back or cherry-picking messy. Rebasing, if anything at all, tends to be preferable
ENH: add arguments for scatter_plot
State: closed

Modifies pandas/tools/plotting.py
- xlim, ylim, sharex, sharey, xlabelsize and ylabelsize arguments are added
- To handle the common arguments for axes decoration, _decorate_axes() function has been added (currently, only scatter_plot uses this function).
- kwds argument added which is passed to ax.scatter()
- Some test cases are added.

This pull request should be merged to following pull request, but I've mistaken to merget the latest master branch.
https://github.com/pydata/pandas/pull/1947


Labels:

Comments:
Postponed til after 0.10. Can you fix the bugs here and let me know when you have the test suite passing? Thanks

MIA.
fixed color keyword in time series plot - issue
State: closed

Fix 33bb84b was to series.plot not timeseries.plot. This is the same fix for timeseries.plot.
Not sure why the def _maybe_add_color(kwargs, style, i) occurs in both _make_plot and _make_ts_plot?
#1890


Labels:

Comments:
`_maybe_add_color` :  they shouldn't be <i>defined</i> in both

oops, left a comment to myself in the code
(much) faster testing
State: closed

after priming the cache, the extensions and cythoning will only recompile if files
change. otherwise they are retrieved from the cache.

the caching is disabled by default just in case it's buggy, but i've had no problems
so far.

with caching enabled and using tox_prll.shl it now takes 25 seconds to test all
2.x and about a minute and a half to test everything.


Labels:

Comments:
looks good. thanks
Fix for EOF issue on c-parser branch
State: closed

Fix EOF issue on c-parser branch due to parser_handle_eof() not explicitely returning a value if state is START_RECORD. This results in the return value sometimes incorrectly set to -1 on Mac OS (seems to work okay on linux though). As a result, a parser exception is raised even though the file is read correctly.

Patch returns 0 if parser state is START_RECORD, since that should mean that a line has just been finished being parsed correctly and it's at a valid end of the file. Also added a default return value of -1 if none of the other tests pass.


Labels:

Comments:
Good catch. I'm surprised I never hit this issue (I've been developing on Linux and Mac)
ENH: pivot_annual with a better exception message
State: closed

Without the extended message the user hardly understand why no integers are allowed.

Actually, I do not even understand it by now.

Can we not use _values.fill(np.nan)_ with integers?


Labels:

Comments:
I pushed a change to allow series with integer dtypes in pivot_annual
You cannot use values.fill(np.nan) with integers because np.nan is a float.
The new change ignores the input dtype and returns float dtypes
Rplot
State: closed

The examples in documentation did not render very nicely.


Labels:

Comments:
Thanks @orbitfold this is really great stuff. We'll review and merge in this month, to be released as part of upcoming pandas 0.10.

Assigning this to Chang-- sorry that we haven't been able to get to merging this yet. Very busy couple of months

Well, once you need me to fix anything, or add new features let me know.

bump for next sprint. 

This is going to happen. I promise, lol

can this be considered for 0.11? it's been bumped in the past, and 
the submitter put in significant effort. I feel bad.

Yes let's merge. This should be clearly indicated as an experimental API that is subject to change however. 

I merged. This will need some cleanup and tweaking but is OK as a WIP. 

Just one note: this has an unprotected import of matplotlib, which shouldn't be a hard requirement

Fixed in git master

fixed integer division failing tests on py3
ENH: Allow DataFrame.update to work on non DataFrame object if coercible
State: closed

On issue #1988, a new Panel.update function should accept a dict of DataFrames.  Right now, DataFrame.update can't accept a dict of Series (or a dict of arrays for example).  To me, it wouldn't make much sense for Panel.update to have the functionality while a DataFrame does not.

Currently:

```
>>> df.update(s) #s is a dict of arrays or of Series
...
AttributeError: 'dict' object has no attribute 'reindex_like'
```


Labels:

Comments:
Merged in also. thanks
ENH: autocov function for Series
State: closed

Add an autocov function to Series.

Default is single lag w/ bias, but those are configurable.


Labels:

Comments:
May want to add a see also in the docs for the statsmodels version, sm.tsa.acovf?

Aside: I imagine we're going to continue to see duplicate efforts on all of these little convenience methods.

I think that's a good idea.  I've updated the docstring.

I think we are deferring to statsmodels on this...if you disagree pls reopen/open thanks
Fix #1982
State: closed

fixes #1982

Had to hemstring another test so the fix will not break the tests, see 
first patch in series I think it's the right thing to do.

There was another fix for a similar unicode problem in  703bbba7ff3bd, 
there seems to be some Unicode whack-a-mole going on...

I'm willing to try my hand at fixing the unicode issues more seriously, 
but that probably involves many changes throughout the codebase,
and some pain until things settle down. 

would you consider such a big PR?


Labels:

Comments:
I'm definitely aware of the Unicode whack-a-mole. It's used up a huge amount of development time. Having to deal with different console encodings is also a big pain. I'd be willing to accept a big PR cleaning this up after 0.9 is out; I guess we have a test suite and CI to protect us on Python 2 and Python 3.
ENH: Add default_index method to DataFrame
State: closed

This partially addresses #1894. Just a method to reset the index in the sense of give me the default np.range index.


Labels:

Comments:
Looks like it does `df.reset_index(drop=True)`?

``` python
In [27]: df
Out[27]: 
          A         B         C         D
1  0.748854 -0.183158 -0.224337 -0.321686
4  1.000946 -1.383009 -0.950255  0.067827
6  0.445805 -0.994903 -0.152020 -1.445314
9 -0.900290  1.865405 -0.290509  0.138487

In [28]: df.reset_index(drop=True)
Out[28]: 
          A         B         C         D
0  0.748854 -0.183158 -0.224337 -0.321686
1  1.000946 -1.383009 -0.950255  0.067827
2  0.445805 -0.994903 -0.152020 -1.445314
3 -0.900290  1.865405 -0.290509  0.138487
```

If so, then this should be clearer in the documentation for reset_index. 

```
drop : boolean, default False
    Do not try to insert index into dataframe columns
```

As I mentioned in the issue, I always assumed that reset_index should do what I want it to do, but I don't get that from the above.
ENH: Allow referencing of Excel columns by their Excel column names.
State: closed

I find it more convenient to refer to the columns as they are in Excel by their string representations.

The excel2num() function I found on StackOverflow. Does that need to be referenced/acknowledged?

I'm happy to put in some unit tests but couldn't find a natural home for it. Most of the ExcelFile tests sit in test_frame.py but I think something like test_io.py or test_parsers.py would be a better fit.


Labels:

Comments:
All you have to do is push additional commits to the branch (well, your master branch in this case-- see http://pandas.pydata.org/developers.html#working-with-the-code for the future). Would you mind adding a unit test that addresses this new behavior (you can use one of the .xls files that are already used in the test suite)

Thanks Wes. I wanted that feature for myself so I just started hacking at it. I will follow the guidelines in future.

I've made the change I pointed out above and added a unit test for the 'str' behaviour that replicates the existing parse_cols unit tests.

Please review and see if you're happy with it.

cherry-picked the relevant commits. thanks!
FDPanel object, a 4 dimensional pandas container (version 2)
State: closed

FDPanel is like a Panel object, but provides 4 dimensions: labels, items, major_axis, minor_axis

(version 2: much of the code changes now occur in panel.py; fdpanel.py is a thin wrapper over an n-dim 
panel like obj; supporting NDPanel like objects with named higher dimension will be very easy)

instead of using a dict of Panels to hold data, the FDPanel provides a convenient represenation in pandas space
with named dimensions to allow easy axis swapping and slicing

see examples/4D Example 2.pdf for creation and usage
## motivation

io/pytables currently provides a nice representation of a table to store a Panel object
directly (assume item x major (dates) x minor)
FDPanel allows many named tables to represent a collection of these tables in a single hdf5 file
## testing

tests/test_fdpanel.py provides a similar methodology to test_panel.py

FDPanel required an overhall of many methods in panel.py and one change in core/index.py (regarding multi-indexing)
almost all methods in a Panel are extended to FDPanel (with the exception in that Panel now allows a multi-axis on axis 0)

docstrings need to be refreshed a bit and made a bit more general

all tests that are not skipped pass

join works but merge testing code not complete yet (pandas.tools.merge)


Labels:

Comments:
Awesome. I'll have to review this in more detail after 0.9 final is out

3 obvious extensions to this:
1. allow a class method on NDFrame to set/reset axis properties (e.g. set all of the _AXIS_\* stuff, and recreate the axes) - essentially a way to rename (or provide alias) for different axes - e.g. instead of label,items,major_axis,minor_axis, could use latitude,longitue,altitude,time for example - should be straightforward to do this
2. could rename FDPanel -> Panel4D to be consistent, create Panel5D (5th dimension could be called by default ???)
3. a ND class creator - take an existing class and add a dimension to it  
   e.g. to create Panel4D I could do something like:
   `Panel4D = NDCreator(Panel, axes = ['labels','items','major_axis','minor_axis'])`  
   Panel5D then is trivial e.g.
   `Panel5D = NDCreator(Panel4D, axes = ['high_dim','labels','items','major_axis','minor_axis'])`  
   
   Of course this is whole exercise is non-trivial:
   - a higher dimensional object is not so easily specified, as the plane/join axes need to be computed (which I think could be done automagically)
   - this new object needs to be inserted in the indexing.py (thinking multi_take here) - but I think this could also be handled by changing multi_take to inquire to the object if it can handle this.....could get a bit tricky)
   - the constructor_slice needs to be created properly (e.g. in order to create a Panel5D, you have to have a Panel4D, which in turn requires a _compatible_ Panel - e.g. same labels/alias for sliced objects) 

Panel4D pull request supersedes this
ENH: Series item access via attribute #1903
State: closed

Hack based on http://stackoverflow.com/questions/12383436/access-pandas-series-item-with-dot-notation-like-dataframe
Some tests included.

First contribution to this great project !
I hope this could help.


Labels:

Comments:
I wonder what the performance penalty of this is in attribute lookup time

`%timeit s.index` goes from ~130-150ns to ~350-400ns with this change. Tabling this until 0.10 (at least) 

could probably add df.c.colX and df.r.rowX without performance hit and cleaner namespace
in ipython tab completion.

superceded by #3045, closing in 3..2..

1.

I closed for a PR which I withdrew, let's keep this open for future discussion.

this was covered by #3482, closing.

this was closed by #3482 (will note it soon)

```
In [1]: s = Series([1,2,3],index=['a','b','c'])

In [2]: s.c
Out[2]: 3

In [3]: s.b
Out[3]: 2
```
Allow setting quoting
State: closed

Allow setting the quoting type for to_csv. Adds "qouting" argument to the to_csv method of DataFrame and pass this on to the csv.writer. Can then use the defaults from the standard library to set the desired  type of quoting, like so:

import csv
...
dataframe.to_csv(filepath, quoting=csv.QUOTE_NONNUMERIC)
...


Labels:

Comments:
FYI this commit breaks the test suite. I'm cleaning it up, but keep in mind for the future

Thanks. Will remember to run the test suite next time.
Pytables support for FDPanel
State: closed

add pytables branch to support FDPanel

this is on top of my previous fdpanel branch
supporting put/append/select of FDPanel to hdf5 tables (and makes selection a bit more generic, as store.select() now works).

FYI - the included FDPanel definitions (eg fdpanel.py) are v1 - so merge FDPanel (v2) first
 Then cherry pick this merge (eg only need pytables.py and tests and included examples)


Labels:

Comments:
this is deprecated in implementation - updated pytables coming in future PR
FDPanel, a 4 dimensional pandas container
State: closed

a FDPanel is like a Panel object, but provides 4 dimensions: labels, items, major_axis, minor_axis

instead of using a dict of Panels to hold data, the FDPanel provides a convenient represenation in pandas space
with named dimensions to allow easy axis swapping and slicing

see examples/4D Example 2.pdf for creation and usage
## motivation

io/pytables currently provides a nice representation of a table to store a Panel object directly (assume item x major (dates) x minor)
FDPanel allows many named tables to represent a collection of these tables in a single hdf5 file
## testing

tests/test_fdpanel.py provides a similar methodology to test_panel.py

FDPanel is completely self contained (in core/fdpanel.py), with the exception of a single change that was needed in core/indexing.py (regarding multi indexing)

all tests that are not skipped pass (some were skipped due to deprecation of some panel like terms that are somewhat confusing in a 4D representation, e.g. minor_xs,major_xs); arithmetic operations not really tested much - almost always reduce the dimensionaliity (to panel/frame/series) before these types of operations
some code in panel.py could be changed to avoid some method duplication in fdpanel.py (e.g. _prep_ndarray, hard coding of some axes)

this was based originally on 0.7.3 (but currently working on 0.9.dev)


Labels:

Comments:
Cool. This begs the question: maybe time to have an N-dimensional table/panel object and have Panel/FDPanel as special cases (i.e. the axes have names)? 

NDFrame is almost there - a couple of tweaks to avoid explict names in dimensionality drops is all that really required (e.g. just like you have _constructor, you need a _sub_constructor) - though this might have a performance impact (which you could avoid by hard coding in Panel/FDPanel).

You might have a use case where you take a multi-index dataframe and expand the dimensions to make a NDFrame
I do this explicity and just represent many things as panel, as I prefer the syntax and usage of panels versus multi-index

see fdpanelv2 pull request
Update pandas/__init__.py
State: closed

Removed try/except around import pandas.lib

The try/except hid a dateutil import error.


Labels:

Comments:
Wouldn't it be better to have the current exception?  Perhaps you can check for whether the `ImportError` came from `pandas` or not...

IMO, it is better to not have any special handing of this import error, since the except block can raise an error with wrong information.

In my case, I didn't have python-dateutils in my virtual env. When doing "import pandas" I got the "C extensions not built" error, instead of "ImportError: No module named dateutil".

I agree it would be preferable to have a better error message. Even checking explicitly for the presence of the .so/.pyd files (to determine whether the extensions have been built) would be better.

I added a narrower check that will make dateutil import failures pass through, but still get a more helpful error message if the C extension isn't built at all
fixed row/col orientation for 2D arrays
State: closed

Pytables uses the opposite convention to pandas for row/col. Using the numpy transpose .T corrects for this without copying the data and keeping the orientation of 1D arrays intact.


Labels:

Comments:
Is solution for https://github.com/pydata/pandas/issues/1824

my apologies, it appears I was a little premature with this pull request as the files it writes can't be read again. Should I close this request or edit it in-place as/when the problem is resolved?

earlier problems are fixed by adding a new _v_attr signalling the transpose has been applied. It's not that pretty, but it's logical and appears to work.

Thanks. I think the `_v_attr` hack also will make it so legacy files can be read

Yes it should do, because it defaults to false if it's not found.
print exception when importing pandas.lib
State: closed

I was trying to install pandas and kept getting the error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/pandas/**init**.py", line 16, in <module>
    raise ImportError('C extensions not built: if you installed already '
ImportError: C extensions not built: if you installed already verify that you are not importing from the source directory

I couldn't figure out what was happening until I made this change and saw from the printed out exception that I  didn't have the dateutils package installed.


Labels:

Comments:
Thanks (I moved the print statement til after the exception is grabbed out of the sys module (I can't remember how that ended up like that instead of assigning the exception when it's caught))
Fix tostring unicode
State: closed

I'm not sure if this is perfect (or where else things like this might come up), but this fixes the failing test I added and the rest of the test suite passes. The issue arises when you give a string containing non-ascii characters. You have to decode to unicode (which I force to be utf-8...). I don't know the Python 3 issues, if any (it's in a Python 2 protected block of code). See issue #1620.


Labels:

Comments:
Looks good over here. Thanks
Handle NoneType docstrings to support the -OO interpreter option
State: closed

This fixes the issue I described in #1741.

When running uWSGI with the `--optimize` option set to `2`, it passes the `-OO` option to the interpreter and then there are no docstrings and pandas errors out during import.

I also noticed there was a note about this [here](https://github.com/pydata/pandas/blob/master/pandas/util/decorators.py#L74).


Labels:

Comments:
Thanks. I cherry-picked this commit so you should `git reset --hard origin/master` after fetching the pydata upstream branch
Add expanding moment functions and related tests.
State: closed

Implement `expanding_` analogues to the `rolling_` moment functions available in `pandas/stats/moments.py` by calling the `rolling_` function with `window=len(arg)` and `min_periods` defaulting to the minimum number of data points to compute the statistic (`1` for most, `4` for kurtosis, etc.). Resolves Issue #849.

The only potentially surprising behavior is that some `expanding_` functions differ from their NumPy equivalents in whether they preserve `NaN`s. For example, `expanding_max(s)` is equivalent to `rolling_max(s, len(s), min_periods=1)`, but both produce different output from `s.cummax()`, as the latter always returns `NaN` for every `NaN` in `s`. I don't have a strong opinion on which output is correct, aside from thinking `rolling_` and `expanding_` should agree.


Labels:

Comments:
This is great, thanks. I'd held off on adding the expanding window functions but they're useful to have and will reduce the number of questions I get ;) It would be nice to add something to the documentation about them