BrandonFors/pandas-github-issues huggingface dataset:
PERF: improve perf of index iteration (GH7683)
State: closed

closes #7683

`PeriodIndex` creation is still in python space, not much help

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
timeseries_iter_datetimeindex_preexit        |  12.7254 | 3657.9890 |   0.0035 |
timeseries_iter_datetimeindex                | 679.8913 | 3726.6284 |   0.1824 |
timeseries_iter_periodindex_preexit          |  69.0370 |  62.8881 |   1.0978 |
timeseries_iter_periodindex                  | 6941.9633 | 6024.2947 |   1.1523 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [142718e] : PERF: DatetimeIndex.__iter__ now uses ints_to_pydatetime with boxing
Base   [f9493ea] : Merge pull request #7713 from jorisvandenbossche/doc-fixes3

DOC: fix doc build warnings
```


Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance

Comments:
5 times faster! that's great!

@yrlihuan thanks for the fixes on this!
BUG: PeriodIndex constructor doesn't work with Series objects
State: closed

closes #7701 


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Period
description: Period data type

Comments:
@jreback i think this should go in 0.14.1, since this is a bug

found this updating the examples in wes's book

is this #7701 ? (which @jorisvandenbossche beat you to it!)

releas note & can you indicate the issue in the test? squash and looks ok, ping when passes

@cpcloud side issue: https://travis-ci.org/pydata/pandas/jobs/29543656

prob just need to add that error code tot he network decorator?

yeah not sure, all urlerrors should be caught because they're a subclass of one of hte classes that network checks ... i'll see if i can't find out what the issue is

pushed this: https://github.com/pydata/pandas/commit/66e1763f8dd0d92d2d36b071569fc7d2a313a647

rebase and give it a whilrl

@cpcloud can you update?

updated ... waiting on travis

minor comment, pls squash and merge (travis already passed, so good 2 go)

alright i'll add some docs and then merge it

oh....doesn't really need any docs, just the issue number on the test (well maybe the doc-string might need to be verified)

oh ok ... i've already added a bug fix in v0.14.1.txt

a bug fix _doc note_ i should say

@cpcloud as an aftermath, can you also update the docstring of PeriodIndex? (it actually says only 'array' and not series)

sure :) aftermath :fire:
BUG: offset normalize option may not work in addition/subtraction
State: closed

Closes problem found in #7375.
@jreback Is this for 0.15 (needs release note)?


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
its for 0.14.1 (trying to release in next day or 2). @jorisvandenbossche noticed it. 

that's why I wanted a separate PR (you can update an existing entry if you want, or new is ok too)

Understood. Appreciate your review.

can you add in the doc examples explicity (i am sure they overlap, but just to make sure), the ones right above this: http://pandas-docs.github.io/pandas-docs-travis/timeseries.html#parametric-offsets (e.g. the section where it uses normalize)

OK, added doc description.

ok ping on green
BUG: DatetimeIndex.__iter__ creates a temp array of Timestamp (GH7683)
State: closed

closes #7683 


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
needs:
- tests
- release note needs to go in v0.15.0.txt
- needs a vbench (in vb_suite)

Better to add `DatetimeIndexOpsMixin` in base.py, and remove `__iter__` from `DatetimeIndex` and `PeriodIndex`.

`DatetimeIndex` behaves a bit differently from `PeriodIndex` when boxing. Instead of passing raw values into `_box_func`, it firstly reinterprets them as integers (`asi8`). Could this be made consistent?

Believe so like `DatetimeIndexOpsMixin.asobject` (and `_box_values`) does. `PeriodIndex._box_func` should accept `asi8` and return correct object representation.

@yrlihuan I see you did a new PR, that's fine, normally you don't need to do that though (just update the existing on), making sure you force push

see here: https://github.com/pydata/pandas/wiki/Using-Git
DOC: table keyword missing in the docstring for Series.plot() and DataFr...
State: closed

...ame.plot()


Labels:
name: Docs

Comments:
HTML doc for this can be found here: http://pandas.pydata.org/pandas-docs/stable/visualization.html#plotting-tables 

@Sinhrks this is correct?

Thans for this. Only one comment.

@jorisvandenbossche looks ok to me

@mortada Thanks a lot!
BUG: Fix conditional for underlying price in io.data.options.
State: closed

Refactor and regression test.

Fixes #7685


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
@dstephens99 if you could make that small change asap would be gr8

I just merged this in, thanks!

Sorry, didn't have access to git at work.  

I tested that code though, is not np.nan works.  I think np.isnan is probably faster though.
FIX: to_sql takes the boolean column as text column
State: closed

In the original code, `com.is_bool(arr_or_dtype)` checks whether `arr_or_dtype` is a boolean value instead of a boolean dtype.

A new function `is_bool_dtype` is added to `pandas.core.common` to fix this bug.


Labels:
name: Bug
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@yelite Thanks for the contribution!

Can you add a test for it (look in io/tests/test_sql.py)? You can for example take the example you were working with where you experienced the issue, and slim this down to a small test case. Just ask if something is not clear for this.

@jreback There isn't already a function like this somewhere?

this is already in master (recently added)
core/common iirc

@yelite indeed, it is already here: https://github.com/pydata/pandas/blob/master/pandas/core/common.py#L2255

@jorisvandenbossche 
Great, I removed my function from the commit and added a simple test.

@yelite I wanted to say your test would fail because of the index, but you already adapated it!

Can you add a line in doc/source/v0.14.1.txt in the bug fixes section for this?

@jreback 

```
AssertionError: attr is not equal [inferred_type]: 'string' != 'unicode'
```

https://travis-ci.org/pydata/pandas/jobs/29317980

Does this difference between `string` and `unicode` really matter? Should we omit it?

so the returned data is unicode. so 2 solns:
- on the test calls tests with (and w/out) the server returning unicode (IIRC they usually and by default do, but can be turned on/off 

e.g. in sqlalchemy (sort of manuall), I usually do

`engine.dialect.description_encoding=None` (with postgresql),
- add an option to `assert_frame_equal` to ignore this different (better soln I think)

@jorisvandenbossche ?

@jreback 
A mistake. I thought `check_column_type` means checking the type of each column. Actually it checks the type of column block. So it's not important in this test. I set it to False now.

@jorisvandenbossche 
I added a line at the end of  `doc/source/v0.14.1.txt`. Is that ok or it should be organized by section?

For the whatsnew note, that is OK.

And the column names of the original dataframe and the result from `read_sql` are indeed different in type ... (but indeed, not important to test that here). 

Travis is green, so merging. 
Thanks a lot! (and if you want to tackle other issues, don't hesitate :-))

https://travis-ci.org/pydata/pandas/jobs/29341707

I have seen this happen if something weird is passed to an `is_***_dtype` (`np.dtype` craps out).

let's see if it repeats

hmm, the next travis builds have the same (but strange that for the PR it had not this issue)
Create DOC: Xls Visualization
State: closed


Labels:

Comments:
what is this?

A demonstration of how to use pandas to graph data from a xls sheet.

On Mon, Jul 7, 2014 at 11:38 AM, jreback notifications@github.com wrote:

> what is this?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/7676#issuecomment-48195287.

possibly could go in here: http://pandas.pydata.org/pandas-docs/stable/10min.html#excel
this doesn't need a whole section

Is there any reason not using pandas plotting functions and calling mp directlyl?

closing this. not sure of the purpose here.
TST: skip buggy tests on debian (GH6270, GH7664)
State: closed

closes #6270 
closes #7664 


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
could I have mislead you more? ;)  it was 64bit :-/ on 32bit it was that other issue, so it didn't get to test with 3.2 -- tested now in 32bit chroot -- indeed the same problem.  So just please skip regardless of the arch just based on numpy/python version

ok, that's fine...will change.thanks

ok, this is merged in master. pls give a recheck and lmk if anything else (or these break).
thanks :)

if all set on your end going to release friday.
Add some documentation on gotchas related to pytz updates #7620
State: closed

closes #7620 


Labels:
name: Docs

Comments:
can u make this pytz/dateutil (or just say tz library)

as this is a function. of the time zone definitions and not the tz library itself

Done.

Also, the ipython slowness was actually due to how completion is implemented in PyDev.  It does dir(object) and then iterates through the attributes calling getattr in order to call inspect.getdoc.  With a large DatetimeIndex, getattr for properties evaluates the property and some of those take 15 seconds.  Trying to work out a solution to that on the PyDev side.

@rockg, if that's of any help, I've recently pushed a [fix](https://github.com/ipython/ipython/pull/4827) into ipython introspection utilities to avoid evaluating properties in the last step of the lookup when doing question-mark queries, e.g. `obj.this_is_evaluated.this_is_not?` It ended up pretty simple, maybe you can borrow some inspiration from there.

@jorisvandenbossche ?

looks ok. Does this close #7620 ?

@immerrr Thank you for the reference.  I was doing something similar.  I simply did:

```
try:
   obj = getattr(type(parent_obj), attr)
except AttributeError:
   obj = getattr(parent_obj, attr)
except:
   continue
```
DOC: remove mention of TimeSeries in docs
State: closed

As a Series with a DatetimeIndex is no longer presented as a `TimeSeries`, I think we also should not longer mention it in the docs as a 'seperate object', so removed the last few mentions.


Labels:
name: Docs

Comments:
maybe add this to the Deprecatoon list on 0.16 (the aliasing of the TimeSeries object itself)

Do you mean #6581?

yep

actually this needs to be deprecated first, so pls create an issue - can do it in 0.15

@jorisvandenbossche moving to 0.16.0 but ok if you wanto do in 0.15.0

@jorisvandenbossche ?

Looking back at this, I don't really see a problem with already changing the docs, before actually deprecating/removing it. As normal users won't see it anymore (a Series with DatetimeIndex does not give you a TimeSeries back, but just a Series).

Or what was the reason we didn't merge this?

@jorisvandenbossche I think this is fine. the deprecation issue (for actual `TimeSeries` object is already noted)
TST/CLN: Refactor io.data.options class to improve testing
State: closed

Removed most references to month and year (passing expiry date around instead).
Also protected with RemoteDataError checking for the expiry month links.

Fixes #7648


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
https://travis-ci.org/pydata/pandas/jobs/29115841

looks like need to trap that failure as well

thanks!

There's no way to tell what the variables where at the time of the error on Travis CI is there? I don't want to trap too wide of an error - I'm guessing it failed there because it wasn't able to retrieve the underlying price.  

since mkt is closed, you _might_ be able to reprod. You can 'trap' the error, then just use print statements (it will print to the log)

I recreated it.  A missing underlying price.  Checked to make sure there is an underlying before reducing the dataframe.

thanks!
FIX: Scalar timedelta NaT results raise
State: closed

Currently, coercion of scalar results from `float` to `timedelta64[ns]` passes through `int`, which raises when attempting to coerce `NaN` to `NaT`.

To reproduce:

``` python

import pandas as pd
import numpy as np
pd.Series([np.timedelta64('NaT')]).sum()
# TypeError: reduction operation 'sum' not allowed for this dtype
```


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
needs tests (put in `tests/test_series.py`, after the `timdelta` operations sections. Need to systematically tests these types of ops (both with and w/o `NaT`). Might also be some tests in `tseries/tests/test_timeseries.py/test_timedeltas.py`

e.g. `sum,mean,max,min`, and that raises on things like `var,std`

Oh, the problem is that test_nanops.py only covers NaN and ignores NaT.  Hrm.  Any bets on how much breakage this change is going to reveal?

It looks like `np.timedelta64('NaT')` creates an object of dtype `m8` instead of `m8[ns]`, and the `isnull` can't handle that.  Instead of accepting `m8`, it would probably be better to coerce `np.timedelta64('NaT')` to `np.timedelta64('NaT', 'ns')` whenever it's being added to a Pandas object, if that's feasible.

you don't handle m8 like that
it's an integer array; use tslib.iNaT

I don't understand what you mean by that.  I will see if I can get back to it sometime over the weekend.

OK, clearly I need to wait until I have time to really read this and figure out what's going on.

have a look at `tseries/test_timedeltas.py`. timedeltas are an odd duck. You generally work on an int64 based series, then take a `timedelta64[ns]` view. note that almost all tests are skipped for numpy < 1.7 (as its broken) for timedeltas.

@ischwabacher status?

Sorry, hectic time at home.  I will try to make time for this soon.

closing in favor of #8268 

this is already fixed in master with Timedelta refactor
PERF: fix perf issue in tz conversions w/o affecting DST transitions
State: closed

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
datetimeindex_normalize                      |   3.3297 |  83.0923 |   0.0401 |

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [fc88541] : PERF: allow slice indexers to be computed faster
Base   [160419e] : TST: fixes for 2.6 comparisons
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
related #7633
FIX: don't create 32 bit int type for int64 column (GH7433)
State: closed

Closes #7433

Replace the sqlalchemy `Integer` type with `BigInteger` as the default integer type in pandas is int64. 
Probably we could be more specific in the type conversion (check for the exact integer type).


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Tests are passing. @mangecoeur @danielballan OK with this?

Looks good.

Could someone test it with oracle?

I see that there are on travis a lot of warnings after merging this. So the tests pass, but raise warnings like: `/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/sqlalchemy/engine/reflection.py:46: SAWarning: Did not recognize type 'BIGINT' of column 'index'`

I didn't see this warning, and after testing, I don't see it with sqlalchemy 0.9.6 and 0.8.3, but I do see it with 0.8.1 (the version used on travis for build 3).

As the tests pass, I assume this is not a problem. But anybody an idea to prevent this warning is raised when users have an older version of sqlalchemy?

so u want to put in place one/both of those newer versions on Travis? which ones should I replace?

Well, already different version are used on different builds on travis. The older version 0.7.1 is used on py2.6, 0.8.1 is used on py2.7 and 0.9.1 is used on py3.3. So the older version of 0.7.1 should be kept to test compatibility with older sqlalchemy, the other ones could be updated to the latest sqlalchemy version. 

This issue was solved in 0.8.2 (https://bitbucket.org/zzzeek/sqlalchemy/issue/2764/make-sqlite-handle-bigint).

So, or we live with the fact that users with older versions of sqlalchemy get this warning, or we can also try to 'fix' it ourselves with something like: https://review.openstack.org/#/c/23883/4/nova/db/sqlalchemy/__init__.py (from, a [similar nova bug](https://bugs.launchpad.net/nova/+bug/1147879))

I wouldn't worry about in that case; changing the 3.4 build to use 0.9.6

You can also change the 2.7 build to use 0.9.6, then the warning don't noise the travis test output (and the older 0.7.1 is still tested in 2.6)

both done (Travis still building master though)
PERF: optimize MultiIndex.from_product
State: closed

This PR speeds up MultiIndex.from_product employing the fact that operating on categorical codes is faster than on the values themselves.  

This yields about 2x improvement in the benchmark

``` python
In [1]: import pandas.util.testing as tm

In [2]: data = [tm.makeStringIndex(10000), tm.makeFloatIndex(20)]

In [3]: %timeit pd.MultiIndex.from_product(data)
100 loops, best of 3: 10.6 ms per loop

In [4]: %timeit pd.MultiIndex.from_arrays(pd.tools.util.cartesian_product(data))
10 loops, best of 3: 23.4 ms per loop
```

It's only marginally slower in small size cases:

``` python
In [1]: data = [np.arange(20).astype(object), np.arange(20)]

In [2]: %timeit pd.MultiIndex.from_product(data)
1000 loops, best of 3: 317 µs per loop

In [3]: %timeit pd.MultiIndex.from_arrays(pd.tools.util.cartesian_product(data))
1000 loops, best of 3: 308 µs per loop

In [4]: data_int = [np.arange(20), np.arange(20)]

In [5]: %timeit pd.MultiIndex.from_product(data_int)
1000 loops, best of 3: 285 µs per loop

In [6]: %timeit pd.MultiIndex.from_arrays(pd.tools.util.cartesian_product(data_int))
1000 loops, best of 3: 269 µs per loop
```

And this case came as a surprise because the cartesian product is blazingly fast both in old and new versions, but profiling showed that factorization is a lot faster when done on a smaller array:

``` python
In [7]: data_large = [np.arange(10000), np.arange(20)]

In [8]: %timeit pd.MultiIndex.from_arrays(pd.tools.util.cartesian_product(data_large))
100 loops, best of 3: 9.88 ms per loop

In [9]: %timeit pd.MultiIndex.from_product(data_large)
100 loops, best of 3: 2.74 ms per loop
```


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
Looks like a nice speedup, but could you please verify that #6439 (cartesian product of a DatetimeIndex) is still fixed?

e.g.:

``` python
import pandas as pd
idx = pd.MultiIndex.from_product([[1, 2], pd.date_range('2000-01-01', periods=2)]).values 
print [x.day for _, x in idx]
# should print [1, 2, 1, 2]
```

In retrospect, I should have added a test for MultiIndex.from_product in #6451.

@shoyer You can always still add a test for that in a new PR

@shoyer works for me

``` python
In [24]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:import pandas as pd
:idx = pd.MultiIndex.from_product([[1, 2], pd.date_range('2000-01-01', periods=2)]).values 
:print [x.day for _, x in idx]
:--
[1, 2, 1, 2]

```

IIRC their are tests for using DatetimeIndex in MultiIndex.from_product. @immerrr if not enough test coverage, pls add (otherwise ok)

@immerrr looks ok, pls verify test coverage then can merge

Ok, added the test

ok, ping when green

good to go

thansk!

@immerrr Thanks for adding that test!

@shoyer you're welcome :)
BUG: Timestamp.__new__ doesnt preserve nanosecond
State: closed

When `Timestamp.__new__` accepts a timestamp with nanosecond, nanosecond is not preserved properly.

```
# create Timestamp with ns (OK)
t = pd.Timestamp('2011-01-01') + pd.offsets.Nano(5)
t, t.value
# (Timestamp('2011-01-01 00:00:00.000000005'), 1293840000000000005)

# If it is passed to Timestamp.__init__, ns is not displayed even though internal value includes it. (NG)
t = pd.Timestamp(t)
t, t.value
# (Timestamp('2011-01-01 00:00:00'), 1293840000000000005)

# If offset is added to above result, ns is displayed properly (OK)
t = t + pd.offsets.Nano(5)
t, t.value
# (Timestamp('2011-01-01 00:00:00.000000010'), 1293840000000000010)
```

NOTE: Unrelated to this issue, test_tslib had `test_timedelta_ns_arithmetic` method duplicatelly. Thus renamed.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
Modified the subject. The problem is not `__init__` , but `__new__`

ok then...thanks
BUG: PeriodIndex.min/max returns int
State: closed

Related to #7279. `PeriodIndex.min/max` should return `Period` ignoring `NaT`.

```
pidx = pd.PeriodIndex(['2011-01', 'NaT'], freq='M')
pidx.min()
# -9223372036854775808
pidx.max()
#492
```


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Period
description: Period data type

Comments:
is their still a need for min/max in DatetimeIndex (or were we using numpy min/max which doesn't work with NaT)?

#7279 made `DatetimeIndex.min/max` to handle `NaT` properly. This PR moves the logic to mixin to cover `PeriodIndex`.

thanks!
BUG: DatetimeIndex.delete with tz raises ValueError
State: closed

Found and fixed `DatetimeIndex.delete` (#7302) results in `ValueError` when it has `tz` and `freq` is less than daily..

```
idx = pd.date_range(start='2011-01-01 09:00', periods=40, freq='H', tz='Asia/Tokyo')
idx.delete(0)
# ValueError: Inferred frequency None from passed dates does notconform to passed frequency H
```

`insert` (#7299) and `take` don't have problems, but changed test case to use hourly frequencies to detect this kind of problem.


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
@sinhrks update?

@jreback updated to use tz_convert_with_transition.

gr8 thanks!
CLN: Simplify Period Construction / Resolution
State: closed

Simplified `Period` construction a little by adding `millisecond` resolution.


Labels:
name: Period
description: Period data type
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
modify this to add Period/PeriodIndex https://github.com/pydata/pandas/blob/master/pandas/io/tests/generate_legacy_pickles.py

run it on 0.14.0 and add generated pickles to the repo
then test with this change to see if pickle works ok (if u can do more than 1 version/platform would be gr8)

if u want to add some for prior versions of pandas ok too (this scipy is meant to be independent of pandas versions for this reason)

the test suite will test the pickles on current version for back compat tests

Thanks. As `Period` can't be pickled, going to confirm `PeriodIndex` only.

```
import pickle
pickle.dumps(pd.Period('2011-01-01'))
# TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled
```

ahh right.....that is a separate issue (and should be resolved as well)

OK. Added `PeriodIndex` pickles taken in 0.13.0 and 0.14.0.  

gr8 thanks!