BrandonFors/pandas-github-issues huggingface dataset:
BUG: DTI.freqstr raises AttributeError when freq is None
State: closed

`DatetimeIndex.freqstr` raises `AttributeError` if `freq/offset` is `None`, even though docstring says "return the frequency object as a string if its set, otherwise None"

```
idx = pd.DatetimeIndex(['2011-01-01', '2011-01-02', '2011-01-03', '2011-01-04'])
idx.freqstr
# AttributeError: 'NoneType' object has no attribute 'freqstr'
```

Also,  `DataFrame/Series.to_period` has a logic to use `inferred_freq` when the freq is not passed, but it doesn't work actually because of the bug. Moved the logic to `DatetimeIndex.to_period` for consistency and made it works.

The fix will simplify #7602 a little.


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
thanks!

fixes r gr8!
BUG: {expanding,rolling}_{cov,corr} don't handle arguments with different index sets properly
State: closed

Closes https://github.com/pydata/pandas/issues/7512.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
needs a release note in v0.14.1 (in bug fix section)

ok, just ensuring that this is a bug fix only (and not actually changing the result; which is technically wrong currently when the indices are not aligned).

thanks
API: DatetimeIndex and PeriodIndex have same representation
State: closed

Closes #7601. Made `PeriodIndex` to have dummy timezone representation to be consistent with `DatetimeIndex`, but this should be ommited?
### Output after the fix

```
# DatetimeIndex (No change)
pd.date_range(start='2011-01-01', periods=10, freq='M')
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2011-01-31, ..., 2011-10-31]
# Length: 10, Freq: M, Timezone: None

# PeriodIndex
pd.period_range(start='2011-01-01', periods=10, freq='M')
# <class 'pandas.tseries.period.PeriodIndex'>
# [2011-01, ..., 2011-10]
# Length: 10, Freq: M
```


Labels:
name: Datetime
description: Datetime data dtype
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Period
description: Period data type

Comments:
doesn't need a tz
what I meant by consistent is the look/feel

OK. Removed tz repr from `PeriodIndex`

can this integrate some of the DatetimeIndex formatting stuff? and out in Base (while leaving the output the same)

in theory could do similarly with Series though might be much trickier 

needs rebase
an u show the examples for both period and datetime in the top of PR

Rebased and examples are attached.

thanks!
BUG: GroupBy.size created by TimeGrouper raises AttributeError
State: closed

Related to #7453 2nd issue:
- `GroupBy.size` created by `TimeGrouper` raises `AttributeError`


Labels:
name: Bug
name: Groupby

Comments:
ok can u add a bench for size in that case?
for both a regular groupby and TimeGrouper?

OK, added benchmarks.

can I post benchmarks for those 2? thxs

Here it is. Cannot measure base performance of `groupby_dt_timegrouper_size` because of the bug, but it looks not slow comparing to `groupby_dt_size`

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_dt_size                              |  42.2956 |  55.4504 |   0.7628 |
groupby_dt_timegrouper_size                  |  34.2657 |  ------- |   ------ |
```

run then with `-H`to test current perf (with your fix) with comparable samples (or do in ipython).

Yep, current perf of `groupby_dt_timegrouper_size` is attached above.
ENH: read_{csv,table} look for index columns in row after header with C engine
State: closed

Closes #6893.

Currently the Python parser can read data with the index columns specified on the first line after the header, e.g.

``` python
In [3]: pd.__version__
Out[3]: '0.14.0-271-gf8b101c'

In [4]: text = """                      A       B       C       D        E
one two three   four
a   b   10.0032 5    -0.5109 -2.3358 -0.4645  0.05076  0.3640
a   q   20      4     0.4473  1.4152  0.2834  1.00661  0.1744
x   q   30      3    -0.6662 -0.5243 -0.3580  0.89145  2.5838"""

In [5]: pd.read_table(StringIO(text), sep='\s+', engine='python')
Out[5]: 
                           A       B       C        D       E
one two three   four                                         
a   b   10.0032 5    -0.5109 -2.3358 -0.4645  0.05076  0.3640
    q   20.0000 4     0.4473  1.4152  0.2834  1.00661  0.1744
x   q   30.0000 3    -0.6662 -0.5243 -0.3580  0.89145  2.5838
```

but the C parser fails:

``` python
In [6]: pd.read_table(StringIO(text), sep='\s+', engine='c')
---------------------------------------------------------------------------
CParserError                              Traceback (most recent call last)
. . .
CParserError: Error tokenizing data. C error: Expected 5 fields in line 3, saw 9
```

This PR patches the C parser to enable this feature.


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv
name: MultiIndex

Comments:
@mcwitt thanks!
Ignore comment lines in read_csv parsing
State: closed

This is the first part of #7470.
closes #2685 


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv

Comments:
@amras1 pls squash, otherwise looks ok

@cpcloud @jorisvandenbossche ?

not familiar with the c parser code, so can't comment on that, but added some doc comments.

@amras1 You updated the io.rst explanation of the parameters, but I think you forgot to also update the parameter explanations in the docstring (it was in your previous PR I think, so probably just have to copy it)

@jorisvandenbossche - Oops, I'll update that now. Let me know if there are any other doc problems.

looking good what the docs is concerned.

Maybe you could also add a test for the combination of using comments with skiprows and header? (or are there already?)

Ok, updated with a skiprows/header test

@jorisvandenbossche ok?

@amras1 thanks!

@jreback - You're welcome!

#7470 can do for 0.15.0 as its an API change

give a check on the docs when they are built (shortly): http://pandas-docs.github.io/pandas-docs-travis/io.html#comments and verify how it looks..thxs

Sure thing

Weird...there's one section in which ipython outputs this:

```
In [15]: data = 'a,b,c\n# commented line\n1,2,3\n#another comment\n4,5,6'

In [16]: print(data)
a,b,c
1,2,3
4,5,6

# commented line
#another comment
```

The rest of the output looks fine.

Indeed strange. I suppose this is an issue with the processing in the ipython directive. If you want to look into it, the code is in https://github.com/pydata/pandas/tree/master/doc/sphinxext/ipython_sphinxext (but which is a copy from ipython itself), or you can open an issue at ipython.
SQL: don't reflect full database GH7396
State: closed

Closes #7396

For now, let PandasSQLAlchemy not reflect anything by default, and `read_sql_table` provides its own reflected meta. Had to change some of the other functions to get this working.

@danielballan @mangecoeur Can someone review? (it feeled a bit messy)


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@danielballan @mangecoeur @hayd can someone review this?

I had a look and it all seems sane, but don't have time to actually run it right now :(

OK, merging then. All tests still pass so should be OK.
DOC: closes gh6838. Breakout options.rst from basics.rst
State: closed

closes #6838. 
closes #3104 
Breakout "options" section for docs, heavily using info already in `basics.rst`. Expands examples, and turns `describe_options` into a table.


Labels:
name: Docs

Comments:
also their is a section in the FAQ which you can incorporate (or maybe just delete)

I the top level you can render this and how it?

What is the best way to get the .html up here?

see here: https://github.com/pydata/pandas/pull/7398

You make a `.png` and then gh will show it

![options and settings pandas 0 14 0-117-g8536e05 documentation](https://cloud.githubusercontent.com/assets/4166648/3403513/03a86608-fd67-11e3-8001-34eb5121b055.png)

Wonderfull! 
I will only be able to give this a review beginning next week.

BTW, as an alternative to show the output, instead of a png, you can also push it to github pages. See eg https://github.com/jorisvandenbossche/example-pandas-docs (I pushed the relevant pages (+ the static folder for css) to a folder in that repo for each pr, and then we can look at it that way). Then you can just give that link, and update the content if changes are made. (But do as you like!)

![options and settings -2](https://cloud.githubusercontent.com/assets/4166648/3438378/e847da02-00c6-11e4-9c7d-d0477879ea52.png)

I'm seeing the same Travis-CI build errors for pull request [#7619](https://github.com/pydata/pandas/pull/7619). These build errors don't seem to be related to the pull requests.

@clham Do you have time the coming days to process the comments?

Yes. Sorry about dropping this,  I started a new job this week and have
been slammed.
On Jul 5, 2014 9:51 AM, "Joris Van den Bossche" notifications@github.com
wrote:

> @clham https://github.com/clham Do you have time the coming days to
> process the comments?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/7578#issuecomment-48086888.

No problem, let know when you find some time to update. 
Pushing this to 0.15 (as it is not critical for the release)

![options and settings -3](https://cloud.githubusercontent.com/assets/4166648/3489569/6bd8ddac-0537-11e4-84e0-e9bd6b40cca5.png)

looking sexy :)

@clham Really thanks a lot! 
Looking very good, so let's merge this, and we can always correct or update later if there are some more issues.
Made read_html ignore empty tables
State: closed

This accompanies #7470.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@cpcloud closes #5107?

possibly, have tocheck

no ... IIRC that error occurs bc of some strange multiindex formatting. not sure if i have the time to get into it by 0.14.1

pushing

@amras1 ?

yes need a release note
and their is a print statement in there

Ok, updated

merged via https://github.com/pydata/pandas/commit/59c175f9e582c5b2466e46b2842d5929fdf7db19
BUG: Error in rolling_var if window is larger than array, fixes #7297
State: closed

fixes #7297

Other rolling window functions rely on the logic in `_check_minp`, and split the iteration into the ranges `[0, minp-1)` and `(minp-1, N)`. Because of the way `rolling_var` handles things, splitting the iteration into `[0, win)` and `(win, N)` makes more sense. Added also some comments as to what is going on elsewhere in the code.

No tests have been added, as @kdiether seems to have that very advanced.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@jaimefrio thanks for the fix, just need tests from #7297 (If you want to add ok too). I don't think cc @kdiether has pull his request (yet)

@jaimefrio can you incorporate those tests in the #7297 (if you want to add more complete tests go for it as discussed there). but want to put this in place for 0.14.1

Sure, hopefully will get it done tonight, tomorrow at the latest. Will ping here once the tests are in and Travis is OKing it.

pls add a release note in v0.14.1.txt as well thxs

hows this coming?

Let's see what Travis thinks, but I think this should be it.

The error is unrelated, some Travis setup failure.

@jaimefrio thanks for the fixes! (sometimes travis get stuck....I just restarted and all good)
BUG: isnull doesnt handle PeriodNaT properly
State: closed

Related to #7485, `isnull` and `notnull` is required to be changed.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Period
description: Period data type

Comments:
see my comments, maybe run a perf check to see if the `is_periodnat` is an issue

Thanks. I've ran perf after changing your point, and found some string related methods take 2 times slower than latest commit. Maybe every call of `is_periodnat` from `checknullobj` is problematic, I'll consider better approach.

ok, maybe run with those problematic tests again, they are somewhat random (but the import _may_ be a problem).lmk

pushing, if you get too it in next day or 2 pls update

@sinhrks perf on this?

@sinhrks perf on this change?

@sinhrks status?

@sinhrks status?

@sinhrks ?

@sinhrks can you rebase / revisit 

closing pls reopen if/when updated

Yep, this is mostly solved by #9134. There is still an issue in object dtype, but currently I don't have good idea to handle it properly without affecting to performance (and I think below example is not popular case...).

```
idx = pd.Index(['x', pd.Period('NaT', freq='M')])
pd.isnull(idx)
# [False False]
```
DOC: Cookbook recipe for emulating R's expand.grid() (#7426)
State: closed

closes #7426 

As suggested by @jreback in #7426, I've added a quick cookbook recipe for emulating R's `expand.grid()`, which creates a dataframe from every combination of the values you give it.

None of the existing sections really seemed to apply, so I've added it in a new section.

Example usage:

``` python
import itertools
import pandas as pd

def expand_grid(data_dict):
    rows = itertools.product(*data_dict.values())
    return pd.DataFrame.from_records(rows, columns=data_dict.keys())

df = expand_grid(
    {'height': [60, 70],
     'weight': [100, 140, 180],
     'sex': ['Male', 'Female']}
)

df
Out[2]: 
       sex  weight  height
0     Male     100      60
1     Male     100      70
2     Male     140      60
3     Male     140      70
4     Male     180      60
5     Male     180      70
6   Female     100      60
7   Female     100      70
8   Female     140      60
9   Female     140      70
10  Female     180      60
11  Female     180      70
```


Labels:
name: Docs

Comments:
@jorisvandenbossche ?

@jreback Alright, I've made the suggested changes and squashed down to a single commit, let me know if there's anything more I'd need to do.

Looking good, thanks!
Minor cleanups for nanops
State: closed

This includes the requested fix for `nanops`, using `common.is_*_dtype` functions whenever possible.


Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
This can be easily split into two pull requests if you want.  This seems like a very small change so I wasn't sure.  

I can also squash the first two commits. I felt it was better at least for the review to have them separate for clarity, but before the actual merge I would guess you would probably want them squashed.

this is ok
just squash and ping me on green

if u r interested ther are lots of dtype comparisons which need fixing to use these functions all over the library (eg dtype == object) is pretty prevalent 

also u can directly import these (instead of using com.is_...)

This seems to be ready now.  travis is green and the first patch has been squashed.

can run a perf check and post anything big
(should be ok but just checking)

There are no major performance regressions.

thanks!
BUG: DatetimeIndex.asobject raises ValueError when contains NaT
State: closed

Closes #7539. 

Also, this fixed #7112 case 3 and case 4 to preserve tz. I've modified the tests for it.

Related to #6469.


Labels:
name: Datetime
description: Datetime data dtype
name: API Design
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls do a perf check make sure nothing really changed

Result of vbench attached.
- 1st time

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_frame_cython_many_columns            |  13.8263 |  12.6497 |   1.0930 |
packers_write_json_date_index                | 249.6490 | 225.3887 |   1.1076 |
packers_write_pack                           |  85.4166 |  75.2767 |   1.1347 |
datetimeindex_normalize                      | 195.3310 | 162.7570 |   1.2001 |
```
- 2nd time

```
write_store_table_mixed                      |  74.6583 |  64.0423 |   1.1658 |
frame_isnull                                 |   2.7107 |   2.2360 |   1.2123 |
reindex_daterange_backfill                   |   2.4064 |   1.9484 |   1.2351 |
dtype_infer_int32                            |   3.3810 |   2.7177 |   1.2441 |
stat_ops_frame_mean_float_axis_0             |   2.3917 |   1.8989 |   1.2595 |
series_drop_duplicates_string                |   1.1253 |   0.8860 |   1.2702 |
```

this is ok...ping when green

@jreback now green.
BUG: expanding_{cov,corr} for objects of different lengths
State: closed

Closes https://github.com/pydata/pandas/issues/7512 and https://github.com/pydata/pandas/issues/7514.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
I'm not sure how/why the merge https://github.com/seth-p/pandas/commit/f5934261e3bcbb35fdbaf0d24bf55db88b59c8a0 got included, whether it's a problem, or what to do about it...

have to rebase it away but don't worry for now
see the wiki

https://github.com/pydata/pandas/wiki

ESP section in using git

I am closing this and will resubmit.
should fix issue #7520, test_nanargmin fails with datetime64 objects
State: closed

I think this will fix the issue #7520.  The issue appears to be that older versions of `numpy`, probably less than `1.7.0`, can't handle `datetime64` dtypes very well.  I have added simple check to see if `numpy` can handle a standard operation with `datetime64` dtypes, and if not it skips those tests.

I can't test this personally since I don't have the older version of `numpy`.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
all datetime64/timesela64 HAVE TO be cast to i8 before operations (and cast back after)

it is an assertion error if this is not the case

you can simply set up a virtual env to test numpy 1.6.1

Yes, `nanops` casts internally.  The problem with this bug is with the reference function.  All the `nanops` tests compare to a reference function to make sure the calculations are correct.  In this case, it is a `numpy` function.  The current version of the `numpy` function, and the version that travis tests against, handle `datetime64` objects fine.  Older versions, however, apparently don't.

So this change just tests the `numpy` reference function and if it fails, it skips the test.

that's fine then
BUG: Timestamp.tz_localize resets nanosecond
State: closed

`Timestamp.tz_localize` resets `nanosecond`.

```
t = pd.Timestamp('2011-01-01') + pd.offsets.Nano(1)
t.tz_localize('US/Eastern')
# Warning: discarding nonzero nanoseconds
#2011-01-01 00:00:00-05:00
```

Even though `DatetimeIndex.tz_convert` can preserve it.

```
idx = pd.date_range('3/11/2012 04:00', periods=10, freq='N')
idx.tz_localize('US/Eastern')
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2012-03-11 04:00:00-04:00, ..., 2012-03-11 04:00:00.000000009-04:00]
# Length: 10, Freq: N, Timezone: US/Eastern
```


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
Ah, the subject and release note are incorrect. Problem is tz_localize, not tz_convert. As I'm out, will fix later.

thanks!

Sorry, corrected subject and sent another PR to fix release note.
BUG: df.reset_index loses tz
State: closed

Closes #3950.


Labels:
name: Bug
name: Timezones
description: Timezone data dtype
name: MultiIndex

Comments:
OK, modified locs.

needs a rebase

thanks!

Don't use `range(5)` in building test frames, instead use `np.arange(5,dtype='int64')` to avoid dtype test issues on windows!

thanks

https://github.com/pydata/pandas/commit/1a6d483b096104a98e3abca39ab7d25953a5ab14
WIP/ENH: allow read_excel to accept URLs (GH6809)
State: closed

closes #6809

This borrows the mechanisms already used in read_csv to enable reading of Excel objects via URLs.

Not sure what tests are appropriate, and I'm not certain what the various decorators do, so I tried to mirror the read_csv tests as best I could.


Labels:
name: Enhancement
name: IO Excel
description: read_excel, to_excel

Comments:
looks good.... @jorisvandenbossche @cpcloud comments?

(Removed vestigial `import xlrd`.)

looks good

thanks @dsm054 glad to have you back on the wagon!

https://travis-ci.org/pydata/pandas/jobs/30344018

any idea why the network decorator didn't catch this? (maybe need to add this exact error message to the error list)...
TST: io.data.Options changes to avoid test failures on bad data.
State: closed

Added try, except to underlying price / quote_time function.

Changed to RemoteDataError in the event that some tables are found, but the correct table is not available.

Fixes  #7524.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
thanks!
BUG: DatetimeIndex comparison handles NaT incorrectly
State: closed

Derived from #7485. Comparison with `DatetimeIndex` result incorrectly if `NaT` is included.

```
idx1 = pd.DatetimeIndex(['2011-01-01', pd.NaT], freq='M')
idx2 = pd.DatetimeIndex([pd.NaT, '2011-01-01'], freq='M')
print(idx1 > idx2)
# [True, False]
# it must be [False, False]
```


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
thanks!
BUG: area plot raises ValueError with tz-aware data
State: closed

Closes #7471.

Must be revisit after #7322.


Labels:
name: Bug
name: Visualization
description: plotting
name: Timezones
description: Timezone data dtype

Comments:
#7322 was merged, pls revisit..thanks!

@sinhrks this looks green now....ok to merge?

Yes, fixed to be consistent with #7322.
cc @rosnfeld
BUG, TST: Fix pandas.core.strings.str_contains when handling regex=False and case=False
State: closed

`pandas.core.strings.str_contains` does not match in a case insensitive fashion _at all_ when given `regex=False` and `case=False`.  This PR should fix the situation.  Additionally, there is test coverage for `pandas.core.strings.str_contains` case insensitive matching both with and without regular expressions enabled.


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
pls add a release note in v0.14.1 (bug fix section), you can ref this PR

@cpcloud @jorisvandenbossche ok by you?

@cpcloud @jorisvandenbossche ok?

looking good. @benjwadams can you squash the two commits?

@benjwadams thanks!
BUG: offsets.apply may return datetime
State: closed

Currently, `offsets.appy`, `rollforward` and `rollback` returns `Timestamp` if argument is `Timestamp` or `np.datetime64`. 

If input is `datetime`, these functions return `datetime` or `Timestamp` inconsistently depending on internal process. It may better to always return `Timestanp`? 
### Affected Offsets
- 'pandas.tseries.offsets.Day'
- 'pandas.tseries.offsets.MonthBegin'
- 'pandas.tseries.offsets.FY5253Quarter'
- 'pandas.tseries.offsets.FY5253'
- 'pandas.tseries.offsets.Week'
- 'pandas.tseries.offsets.WeekOfMonth'
- 'pandas.tseries.offsets.Easter'
- 'pandas.tseries.offsets.Hour'
- 'pandas.tseries.offsets.Minute'
- 'pandas.tseries.offsets.Second'
- 'pandas.tseries.offsets.Milli'
- 'pandas.tseries.offsets.Micro'


Labels:
name: Bug
name: API Design
name: Frequency
description: DateOffsets

Comments:
these should _always_ return `Timestamp`, this is checked, but guess not for _other_ types of input...