BrandonFors/pandas-github-issues huggingface dataset:
BUG: del df[k] with non-unique key
State: closed

This touches some delicate functions to be messing with, so I've split off the 
underlying logic into a function in common and added an exhaustive test. 
Would welcome review none the less.

Also, I tried to test for nagative perf. There must something
wrong with the test since comparing the commit after just adding a test to the baseline, 
shows 50% degredation in a bunch of things. As I said, I don't trust this, ideas?

```
timeseries_asof_nan                       23.4131    15.7471 1.4868
datetimeindex_normalize                 1153.1918   774.8010 1.4884
timeseries_asof                           22.3342    14.9964 1.4893
reshape_unstack_simple                     4.8106     3.2289 1.4899
read_table_multiple_date_baseline        980.0920   657.3641 1.4909
timeseries_1min_5min_mean                  0.8087     0.5422 1.4915
timeseries_timestamp_tzinfo_cons           0.0210     0.0141 1.4922
match_strings                              0.5718     0.3817 1.4982
timeseries_large_lookup_value              0.0301     0.0201 1.4998
reindex_fillna_pad                         0.1828     0.1217 1.5018
timeseries_to_datetime_iso8601             5.6048     3.7270 1.5038
read_table_multiple_date                2169.5440  1441.5460 1.5050
reindex_daterange_pad                      0.2608     0.1731 1.5066
timeseries_1min_5min_ohlc                  0.8046     0.5331 1.5092
reindex_daterange_backfill                 0.2466     0.1634 1.5094
period_setitem                          1166.9910   772.4509 1.5108
reindex_fillna_backfill                    0.1790     0.1183 1.5130
timeseries_asof_single                     0.0656     0.0434 1.5136
append_frame_single_mixed                  2.0747     1.3697 1.5147
timeseries_slice_minutely                  0.0762     0.0501 1.5208

Columns: test_name | target_duration [ms] | baseline_duration [ms] | ratio

(01bc3e0 against 81169f9)

- a Ratio of 1.30 means the target commit is 30% slower then the baseline.
```

closes #2347


Labels:

Comments:
I'll check on my machine when I get a chance if there's any perf difference.

I also need to figure out why Travis has become totally borked. most likely my fault

yeah, things are failing on py2 as well now, and I don't think that's a problem on the travis side.
they recently started preinstalling numpy on the vm to speed things up, perhaps related?

@wesm, reverting 4a200b8 seems to restore py2 builds on travis.

Oh definitely. I'll fix asap

Vbenchmarks looked fine on my end so I went ahead and merged this one

On Python 3 it looks like some issue with NumPy using `sys.__stdout__`. But I don't know what could have changed (must be something in the travis environment)

it's in the final stages of resolution, should be fixed once travis upgrade to venv 1.8.4 within a couple days
https://github.com/numpy/numpy/issues/ 2761

Great. looks like venv 1.8.4 was cut today
Pytables: bug fixes, code cleanup, and much updated docs
State: closed

Should be completely backwards compatible
mainly bug fixes, code clean, and much updated docs
should merge cleanly
1. added **str** (to do **repr**)
2. added **delitem** to support store deletion syntatic sugar
3. row removal in tables is much faster if rows are consecutive
   (remove also returns the number of rows removed if a table)
4. added Term class, refactored Selection (this is backwards compatible)
        Term is a concise way of specifying conditions for queries, e.g.
   
   ```
       Term(dict(field = 'index', op = '>', value = '20121114'))
       Term('index', '20121114')
       Term('index', '>', '20121114')
       Term('index', ['20121114','20121114'])
       Term('index', datetime(2012,11,14))
       Term('index>20121114')
   
       added alias to the Term class; you can specify the nomial indexers
       (e.g. index in DataFrame, major_axis/minor_axis or alias in Panel)
   ```
   
    this should close GH #1996
5. added Col class to manage the column conversions
6. BUG: added min_itemsize parameter and checks in pytables to allow setting of indexer columns minimum size (current implemenation will truncate indexer columns that are too long in a subsequent append - loss of info)
7. added indexing support via method create_table_index (requires 2.3 in PyTables)
        btw now works quite well as Int64 indicies are used as opposed to the Time64Col which has a bug); includes a check on the pytables version requirement
   
    this should close GH #698
8. signficantlly updated docs for pytables to reflect all changes; added docs for Table sections
9. BUG: a store would fail if appending but the put had not been done before (see test_append)
        this the result of incompatibility testing on the index_kind
10. BUG: minor change to select and remove: require a table ONLY if where is also provided (and not None)
    
      all tests pass; tests added for new features

I have some implementation changes to make to Tables to make writing quite a bit faster
but will do in a future commit


Labels:

Comments:
Closing this, superceded by v4 (you can just keep pushing to the same PR or rebase on master if you don't want to create new pull requests)

thanks

ready to merge when u r ready

On Nov 27, 2012, at 7:31 PM, Wes McKinney notifications@github.com wrote:

> Closing this, superceded by v4 (you can just keep pushing to the same PR or rebase on master if you don't want to create new pull requests)
> 
> —
> Reply to this email directly or view it on GitHub.
Update doc/source/r_interface.rst
State: closed

rpy2-2.3.0 is becoming the current release.
Changes to the API are relatively few. 


Labels:

Comments:
On that topic, I may be willing to port the code to rpy2.3 since I use the conversions a lot. Where are the major API changes?

API changes should be listed [here](http://rpy.sourceforge.net/rpy2/doc-2.3/html/changes.html#new-features)
Fix seconds parsing #2209
State: closed

Creating a milliseconds entry by doing math on the seconds entry:

```
msecs = round(float(seconds[i] % 1)*10**6)
```


Labels:

Comments:
Can you add some test cases for this? Thanks.

Would love to, but I can't find your date parsing tests? Can you give me a pointer? If you don't have any date parsing tests yet, I'm afraid it's out of my league, as I never wrote a unittest before, but I'm sure I could extend existing ones.

Oh sure thing, please check pandas/io/tests/test_date_converters.py
`test_datetime_six_cols` uses the converter that calls `lib.try_parse_datetime_components`
add check for endianess in major constructors
State: closed

apropos #2318


Labels:

Comments:
This should probably get put in a Cython function that uses the NumPy C API before it can ship. The additional overhead (a couple microseconds is significant) is more concerning in the Series constructor than DataFrame

fwiw The relevant vbenchmark is series_constructor_ndarray

No problem, I can do that. is it worth it or is this a non-issue?

It does seem potentially like overkill since the mixed-endianness problem I would expect to be very rare in practice. Hmm

your call, i'll leave this open for now. let me know either way (or just close).

how significant of a performance regression is it?
a possible place where this can happen in practice is when dealing with fits files [1].
fits stores its data in big endian format and it is not unusual to just memory map that in with numpy.

[1] http://fits.gsfc.nasa.gov/

Tabling this til a later time. I did a brief tinker with some big-endian arrays and it's not clear to me where problems might arise-- DataFrame for example casts any input data to little-endian on arrival, though all that conversion might be a bad thing with large data sets. There's a lot of work needing to be done on data type handling in the near future, so it would be good to make a battery of "smoke tests" to ensure responsible handling of big-endian data types. 

This will never get merged. closing. left #2821 as future reminder.
Fixes for #1000, to_string(), to_html() should respect col_space
State: closed

Note that the semantics of `col_space`  are different in each case, characters vs. pixels,
but that's reasonable.

I hope this doesn't presage more html configuration via kwd arguments, yonder way
madness lies.

also, there's a discrepency between the docstring and [io.rst](https://github.com/pydata/pandas/blame/master/doc/source/io.rst#L671) (github doesn't seem to jump to the right line)
the first defines it as the width of the columns, the latter as the number of spaces between columns,
I adopted the former, since space between columns does not translate as easily to html.

_sigh_, also, `colSpace` has already been deprecated in favor of `col_space`, so
I would feel bad about deprecating `col_space` in favor of `min_col_width`.
I'll leave that decision to braver, fearless  souls.

closes #1000.


Labels:

Comments:
oops, breakage.

Travis is experiencing weirdness with numpy on py3, tox runs clean for me on 3.2.

Yeah what's going on with that? hopefully transient

Merged this, thanks
CLN: Dropped python 2.5 support
State: closed


Labels:

Comments:
also, should this test be removed?

https://github.com/pydata/pandas/blob/86ab2e9631faf83f9e9bb3174273249ade598f45/pandas/stats/tests/test_moments.py#L166

I'm not sure why I wrote that note-- should definitely leave the test just in case
Make sparse DataFrame and Series construction more efficient
State: closed

Cleans up SparseSeries and SparseDataFrame initialization code so no objects are created unnecessarily. Currently if you run the (potentially useless) statement:

```
SparseDataFrame(columns=np.arange(100000), index=np.arange(100000))
```

pandas will consume memory and time filling in empty Series objects only to then compress them away.

Added to test to ensure SparseDataFrame always initializes with SparseSeries. Otherwise functionality should be preserved.


Labels:

Comments:
cherry-picked. Thanks for the PR.

Got some merge issues in that last commit?

Yeah in RELEASE.rst. I pushed a fix already.
BUG: Fix long index repr #2319
State: closed

``` python
In [2]: pd.Index(np.arange(10000))
Out[2]: Int64Index([0, 1, 2, ..., 9997, 9998, 9999], dtype=int64)
```

broken by 95678ebf.
configurable via `np.set_printoptions (threshold)`, like the original version which used np.ndarray.**repr**.

closes #2319


Labels:

Comments:
I get a couple of failed tests for cparser which are unrelated.

Oh really? Pls report if you are able

False alarm, it went away with a clean/develop.
BUG: dtype=object should stop conversion from object in frame constructo...
State: closed

...r #2255

don't call maybe_convert_objects in _convert_object_array if dtype is object.

Note that it also stops other type conversions too.


Labels:

Comments:
It's better to add new keyword arguments at the end of the arglist, 
or you risk breaking existing code:

``` python
In [2]: def f(a,b=1):
   ...:    print b
   ...: 
   ...: f(0,'b')
   ...: 
b

In [3]: def f(a,c=2,b=1):
   ...:    print b
   ...: 
   ...: f(0,'b')
1

```
API: fillna method argument should be None by default and raise on both ...
State: closed

...value and method #2027

@wesm what do you think of the API change? If you're ok with it I'll propagate to DataFrame and Panel.


Labels:

Comments:
I agree that `method='ffill'` should not be the default. Shouldn't it raise an exception now if you write `ts.fillna()`? Folks who want that should define `TimeSeries.ffill` to forward fill for them (we could even add this convenience method). Go ahead and tweak out the other classes-- let me know when ready for merging

should be ready to go now.

Added ffill and bfill as convenience functions to Series/DataFrame/Panel

thank you sir
Added support for specifying colors in parallel_plot()
State: closed

Sometimes the random colors chosen in parallel_plot are very unfortunate, so I thought it would be nice to provide an option to pass a list of colors in.


Labels:

Comments:
Looks good to me. Can you please add a test case so I can merge it in? Thanks!

Sure thing. How does this look?

I have one more related pull request I was planning on submitting after this which enables the x-axis to be rescaled when numeric column values are used (e.g. "# hours"). Is there anywhere I should go first to suggest the change, or is it okay to just submit another pull request?

Thanks!

If you already coded it up, feel free to make another PR. What's the current behavior that's not right?

Cherry-picked. Thanks very much!

Great! Currently parallel plots remaps the x values to 0, 1, 2, 3... so the plot is also scaled linearly. It wasn't too hard to change it so that if all of the columns are numeric it uses those as the xticks. This way when you pass it something with columns that are not evenly-spaced the plot is scaled correctly.

I'll go ahead and submit a pull request.

Oh I see, that makes sense. Maybe make that an option instead of doing it automatically? Like a 'use_columns' parameter analogous to 'use_index'.
Also maybe another optional parameter to pass in an external set of xticks so if you already have meaningful column names, you don't have to mangle them to get the spacing right?
pytables selection enhancements (to close GH #1966)
State: closed

in pandas/io/pytables.py
1. added **str** (to do **repr**)
2. row removal in tables is much faster if rows are consecutive
3. added Term class, refactored Selection (this is backdwards compatible)
    Term is a concise way of specifying conditions for queries, e.g.
   
   ```
   Term(dict(field = 'index', op = '>', value = '20121114'))
   Term('index', '20121114')
   Term('index', '>', '20121114')
   Term('index', ['20121114','20121114'])
   Term('index', datetime(2012,11,14))
   Term('index>20121114')
   ```
   
    updated tests for same
   
   this should close GH #1996

also potentially a starting point to do things like this:
  df['df>0']...e.g allowing simple string expressions in criteria (Term is indpendent; I could move to a separete module.... - just a thought)


Labels:

Comments:
see #2264
corrected version of ts_hourly
State: closed

small glitch in the function but mainly fixes in tests


Labels:

Comments:
BTW no need to create a new PR, you can just keep pushing commits to the branch you're working on

@timmie, I implemented an hourly version in pivot_annual, pivot_annual and pivot_annual_h were so similar that it didn't make sense to have separate method and module just to have output in a different orientation. You can always use `.T` on the output.

I didn't merge in the extended_info and the default index. Here's why:
1. default index: having the wrong year is not the right way to do this. I guess you need it to access month/week/day properties. In that case you should make a function that can take a year and an array/index of ints that represents hours and create a correct datetimeindex.
2. extended info: would DataFrame.describe be enough?

I'm moving the rest of this issue to 0.11 if you still want to finish the 2 items I just listed above.

> I implemented an hourly version in pivot_annual, pivot_annual and pivot_annual_h were so similar that it didn't 

Thank you very much for that! Shall we add a similar part for minutely?

> it didn't make sense to have separate method and module just to have output in a different orientation. You can always use .T on the output.

In earlier version I had problems with `.T`. And I flet that format easier to read.

> default index: having the wrong year is not the right way to do this. I guess you need it to access month/week/day properties.

Exactly. 

> In that case you should make a function that can take a year and an array/index of ints that represents hours and create a correct datetimeindex.

Could you please show me an example? Python datetime always needs a year/month/day/hour.

> extended info: would DataFrame.describe be enough?

well, I do not think so:

`describe?`
`Generate various summary statistics of each column`

So this is like the summary of a column. 

extended info would generate the descriptives for all columns and add it to the dataframe

I.e. create long-term hourly averages.

Or would you do this also with DataFrame.describe?

Thanks again!

Is there anything remaining here or can be closed?

@timmie, you can probably just do df.T.describe() to get most of what you want.
- minutely: sure, go ahead and just piggieback off of my code for that
- default index: maybe another way to do this is to generate the field info BEFORE you do the pivoting?
- datetime fields in extended info: I think maybe the right place for that is to make a method `DatetimeIndex.fields` that returns a DataFrame whose columns are the field names. This method should optionally take a list of field names.

I'm going to close this PR. Please make a new PR with the stuff mentioned above if you're interested. 
As always, thanks for the contribution!
BUG: start_time end_time to_timestamp bugs #2124 #2125 #1764
State: closed

@wesm can you review this plz?

The main change is I have to_timestamp default to second frequency now to get the first and last second of the Period. This doesn't **quite** solve #2125 since Timestamp resolution goes down to Nanos. So the question is is it appropriate to add 999999999ns to the end_time when ns doesn't exist asa period freq? 


Labels:

Comments:
ok how about this:

start_time : starting second
end_time : ending second
to_timestamp : defaults to 'D' freq for week or longer and 'S' otherwise. The logic here is to make it more convenient for cases like monthly_period.to_timestamp(how='e') to get 'M' freq and monthly_period.to_timestamp() to get 'MS' freq. 

The alternative is to always default to 'S' so for that whole class of use case above you would need to remember to specify 'D' as the freq (e.g., annual_period.to_timestamp('D', how='e'))

What do you think? Consistency vs convenience
Ts pivot hourly
State: closed

Now I figured out.

Here is the PR for #2153


Labels:

Comments:
@changhiskhan do you have time to review this for 0.9.1?

yeah, doesn't look like too many items left. I'll look at it before release.

Tests don't pass. I'm pushing this off til the next release.

@timmie were you able to get the tests to pass on your machine?

When is 0.92 due?

> were you able to get the tests to pass on your machine?

I tried them line by line in the IPython. They worked.
What command do you use to run them?

It was the first time I write dedicated unit tests for a module in my life.
I am learning a lot with you guys and hope to get as serious with TDD.

I will recheck on Monday.

@timmie, thanks for the fast response!
0.10 is going to be next release (fingers crossed we don't have to do a 0.92), which will probably be out maybe mid-late December?

generally you can run the tests by doing something like `python pandas/tseries/tests/test_util.py` from the root pandas directory. And you can run the entire test suite using `test_fast.sh` (excludes plotting tests) or `test.sh`.
You need nosetests and coverage installed.

When you run it in IPython your environment already has a bunch of stuff in it so that's probably why there's a discrepancy. The tests have to pass from clean slate.

After you get the unit tests to pass, if you really want to make the code clean and nice, try to merge the methods in pivot and util. Your method should be called by the method in `util.pivot_annual` (i.e., I would add a check in the cascading ifs in util.pivot_annual to handle hourly frequency and call your function).

Let me know if you have any other questions. As always, we're grateful for your contributions!

> generally you can run the tests by doing something like 

Thanks for the patient instructions.

> try to merge the methods in pivot and util. Your method should be called by the method in util.pivot_annual

actually this is another point were I would like to receive your feedback.
The result given by my function is transposed in comparison to existing privot_annual

I think that my version is more convenient when viewing results later in Excel.
That's why it was not included but put into a separate file.

Please see [PR 2183](https://github.com/pydata/pandas/pull/2183) instead
BUG: fix unicode handling in plot label, close #2080
State: closed

#2080


Labels:

Comments:
@lodagro, add the test you've already provided [here](https://github.com/pydata/pandas/commit/5fa2ae439f4e02e0d97ff05fbba16337685297f8#L4L905)?

It's already in the repo.

I wonder why travis didn't catch it then.
`nosetests --exe -w /tmp pandas.tests` does fail in test_plot() when I run it locally.

I don't think Travis installs matplotlib so the test is being skipped.

Sent from my mobile device

On Oct 28, 2012, at 6:52 AM, y-p notifications@github.com wrote:

I wonder why travis didn't catch it then.
nosetests --exe -w /tmp pandas.tests does fail in test_plot() when I run it
locally.

—
Reply to this email directly or view it on
GitHubhttps://github.com/pydata/pandas/pull/2122#issuecomment-9844576.

Oops, I caught this myself and took out the utf-8 encode. closing

BTW, i got matplotlib running with travis for python 2.x (based on the .travis.yml file in matplotlib/matplotlib), but it takes sometimes too long to build all dependencies and travis aborts (somewhere between 10 en 15 minutes) :-(
STY: pep8
State: closed

Except for test_\* files i ran every file through pep8, and resolved most of the violations.


Labels:

Comments:
Wow, unexpected and awesome. Thanks. Did autopep8 help at all or was this a manual affair?

Vim was my auto-pilot, driving me around and informing me what was not according to pep8 but the corrections are manual. I never used autopep8, scary to let code touch code, i think i would be more comfortable with a required confirmation of each auto-correct. Do you use it?

autopep8 is good for fixing a large category of errors, but I find that fixing over-indented/under-indented code (errors 125-129, if I recall correctly) is a manual affair. I generally do one error number at a time (using --select) and commit after each autopep8, then squash-merge at the end before pushing.

As a benchmark, I modified 6500 LOC in a 40K LOC never-pep8'd codebase in a day last week. The rest I did in vim: highlight a visual selection, then += to format.
Panel constructed with a None as a dict value throws an error
State: closed

Python 2.7.3 (default, Jun 21 2012, 07:50:29)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.

> > > import pandas
> > > print pandas.**version**
> > > 0.8.1
> > > print pandas.Panel(dict(a = None, b = pandas.DataFrame(index=[1,2,3],columns=[1,2,3])))
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "/usr/local/lib/python2.7/site-packages/pandas-0.8.1-py2.7-linux-x86_64.egg/pandas/core/panel.py", line 219, in **init**
> > >     mgr = self._init_dict(data, passed_axes, dtype=dtype)
> > >   File "/usr/local/lib/python2.7/site-packages/pandas-0.8.1-py2.7-linux-x86_64.egg/pandas/core/panel.py", line 256, in _init_dict
> > >     major = _extract_axis(data, axis=0)
> > >   File "/usr/local/lib/python2.7/site-packages/pandas-0.8.1-py2.7-linux-x86_64.egg/pandas/core/panel.py", line 1396, in _extract_axis
> > >     raw_lengths.append(v.shape[axis])
> > > AttributeError: 'NoneType' object has no attribute 'shape'

same behavior exists in 0.9
added tests and fixes for this case (and the pathological case of constructing with dict(a=None))


Labels:

Comments:
Any idea why the file modes got changed to 755 from 644?

no idea

I can be reached on my cell 917-971-6387

On Nov 2, 2012, at 4:47 PM, Wes McKinney notifications@github.com wrote:

> Any idea why the file modes got changed to 755 from 644?
> 
> —
> Reply to this email directly or view it on GitHub.
Fix for Python 3.3
State: closed

There were two problems with a simple 'import pandas' on python 3.3, that have to do with slots.
For the first one, I converted the class initializer into an instance variable.
For the second one, there seems to be a property to compute 'ndim' so I simply removed it.

(I couldn't figure out quickly how to run the tests under Python 3; a number of problems are in the way.)


Labels:

Comments:
cherry picked this commit. thanks