BrandonFors/pandas-github-issues huggingface dataset:
TST: harmonize testing namespace in TestCase (GH8023)
State: closed

closes #8023 

Merged via https://github.com/pydata/pandas/commit/ace3c46be0ebf12f0eb4b63f22843d76b67ccfde


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@papaloizouc Are you interested in trying to add the other `assert_` functions to the `TestCase`?

@jorisvandenbossche sure will do

I think we should try to do the adding of the methods a bit more automatically (so also the docstring are kept as the original).

Eg something like `setattr(TestCase, 'assert_frame_equal', staticmethod(assert_frame_equal))` adds the method to the class keeping the docstring and signature

@jreback 

sure easy enough to inspect the module namespace and auto add the methods to TestCase via a generated wrapper

Merged via https://github.com/pydata/pandas/commit/ace3c46be0ebf12f0eb4b63f22843d76b67ccfde

@papaloizouc Thanks!
ENH: adds ability to generate bq schema from df
State: closed

I need this function for some other work I have in a branch (to_gbq w/ a new table and to_gbq with staging in gcs).  I saw issue #8325 and thought this might be a good intermediate step. 


Labels:

Comments:
seems reasonable. Maybe put in a doc mention in io.rst (gbq section)?

and a release note in v0.15.2 (referencing that issue)

Ok, made the updates, hopefully all goes well with travis.

@tshauck thanks!
ENH: make Series work with map objects the same way as generators
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: API Design
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@jreback I looked around before to see if there was something like types.MapType but I guess there isn't. It's similar with python3's range that actually works fine on Series because you can call len on it, which is not the case with python3's map.

- pls add a release note in v0.15.2 (enhancements section, use this PR as the refernce issue)
- squash down

@mcsalgado thanks!

you're welcome!
WIP: add multi-multi index support to join
State: closed

closes #6360 


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
you need to remove the NotImplemented from the test suite to see if it passes the tests: https://github.com/pydata/pandas/blob/master/pandas/tools/tests/test_merge.py#L1193

also, think about adding a couple of more tests to exercise this (e.g. say an empty frame with the proper index on the merging axis), maybe a couple of simple examples too.

closing pls reopen if/when updated

Any chance to reopen this issue? It would be really useful to have this feature in pandas
Cheers
@harisbal this is a pull-request to fix an issue. you can certainly build on this.
modified:   pandas/io/ga.py
State: closed

```
        ENH
    I've made a number of alterations to this module to improve the
    usability of it.

    1.  A feature has been added to the 'filters' keyword, in that
        compound filters can now be created with AND and OR combinations
        as in the API, while still allowing the user to create
        the filters via lists of strings without the 'ga:' prefix. In
        particulare, compound filters can be formed as:

        filters = ['filter_00', 'and', 'filter_01', 'or', 'filter_02']

        where no defuault behaviour has been created, thus ensuring that
        the compound item is formed the way the user intends it to be.
        The 'and' and 'or' separators are case-insensitive.

    2.  The 'sort' keyword is now passed to the query, and works as
        it should with the API. That is, the current 'sort' keyword only
        acts on the retrieved data when the parser is constructing the
        local DataFrame object. This behaviour is now changed, in that
        the 'sort' keyword is included in the query, as well as acting
        as previous on the parser. Reverse-ordering as in the API is
        maintained by simply including negaitve sign in front of the
        metric / dimension that is being sorted on.

        This is a sublte but important difference, in that the user can
        now include something like:

        sort = '-pageviews'

        to obtain the top N-many results in a query, which is currently
        not possible without obtaining all results and sorting them
        locally.

        Defualt behaviour has been altered, in that the defualt sort
        is now None (sent to query) while the local dataframe is still
        sorted on it's index.

    3.  I didn't like the combination of positional and named keyword
        arguments in the methods, so I have altered the bahaviour of
        the module to use only named keyword aguments.  Thus, the user
        is now able to form a query as a single dictionary item and pass
        it to, say, the read_ga() method.
```


Labels:

Comments:
This really needs some tests. With either some fake data and a mock object and/or testing authentication. The problem is your changes may work now and with you testing. But cannot be tested for regressions, or API changes, or any kind of impl changes.

So I would ideally like to have a set of tests for the existing behavior, _then_ make changes.

No problem; I'm new to this so I'm not sure how to provide / do that and will need a bit of instruction though...

(as in, I cannot share the secrets.json file I'm working with, and don't know how do create a dummy one to demo with)

here are existing tests: https://github.com/pydata/pandas/blob/master/pandas/io/tests/test_ga.py

so need tests for anything new.

these are only 'local' tests. meaning, that Travis the CI will skip on all of them as we don't have credentials to test this. Ideally like to fix that probelm too.

Great!  I'll get to this soon, and may have a solution for the credentials as well.

@DeliciousHair closing this. pls reopen if you can provide some tests. thanks!
fix to_datetime default error suppression
State: closed


Labels:
name: Datetime
description: Datetime data dtype
name: API Design

Comments:
- its not nice to change kwarg names without a deprecation cycle.
- why do you think this is a good idea?
- I think you could only realistically do this with an option that you could optionally set,
  e.g. change the kwarg to `None` then if its `None` use the default value of the option (I am talking
  about `pd.set_option(....)` here)
- this would need a fair amount of tests for this change

I can see how switching to a boolean can break things, but ignoring errors by default is quite bad. How about I start by just changing the default?

@cowpig well, the problem is `pd.to_datetime` will not barf on valid but non-parseable datelikes now.

eg. 

```
In [27]: pd.to_datetime('19MAY11',errors='raise')
TypeError: 'NoneType' object is not iterable

In [28]: pd.to_datetime('19MAY11',errors='raise',coerce=True)
Out[28]: NaT

In [29]: pd.to_datetime('19MAY11',errors='raise',format='%d%b%y')
Out[29]: Timestamp('2011-05-19 00:00:00')
```

So that error message is pretty bad. If you chane the default, then see what error message happens, and could think about doing that. I am not averse, its just that this affects lots of things possibly in subtle way.

I think it belongs here:

date format `"%Y%m%d"` does not raise errors even when `errors='raise'` and silently returns `datetime.datetime` for out of bounds timestamp:

```
>>> pd.to_datetime('1300-01-01', format='%Y-%m-%d', errors='raise')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users//.virtual_envs/system/lib/python2.7/site-packages/pandas/tseries/tools.py", line 318, in to_datetime
    return _convert_listlike(np.array([ arg ]), box, format)[0]
  File "/Users//.virtual_envs/system/lib/python2.7/site-packages/pandas/tseries/tools.py", line 306, in _convert_listlike
    raise e
pandas.tslib.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1300-01-01 00:00:00
>>> pd.to_datetime('13000101', format='%Y%m%d', errors='raise')
datetime.datetime(1300, 1, 1, 0, 0)
```

closing pls reopen if/when updated
BUG/ENH: cleanup for Timedelta arithmetic
State: closed

Fixes #8813 
Fixes #5963
Fixes #5436

If the other argument has a dtype attribute, I assume that it is ndarray-like
and convert the `Timedelta` into a `np.timedelta64` object. Alternatively, we
could just return `NotImplemented` and let the other type handle it, but this
has the bonus of making `Timedelta` compatible with ndarrays.

I also added a `Timedelta.to_timedelta64()` method to the public API. I
couldn't find a listing for `Timedelta` in the API docs -- we should probably
add that, right?

Next up would be a similar treatment for `Timestamp`.

CC @immerrr


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
Regarding `to_timedelta64`, I suppose it already exists as the `asm8` attribute. But maybe adding such a method could be useful for visibility? 

`to_timedetta64()` is fine (and adding `to_datetime64()` ok too). These are corresponding to `to_timedelta()/to_datetime()` and return the underlying object (and not the value, which is what `asm8` does).

#5963 & #5436 might already be fixed / related. So might be able to just add in a couple of tests (or if they are enough there, lmk; e.g. pretty sure offsets already work).

@jreback Yep, those issues are indeed fixed. I added a bunch of tests and everything is passing now.

@shoyer looks good, some minor doc-like fixes.

FYI, you need to list each issue sepately with (closes/fixes) in order to have it close them when merged.

pls squash and ping when ready

#5963 was actually already fixed prior to this PR, but I'll note it anyways :).

@jreback looks like we're ready.

thanks
REF: refactor **kwargs usage
State: closed

related issue: https://github.com/pydata/pandas/issues/8597. 

among changes:

[tests/test_generic.py](https://github.com/pydata/pandas/blob/0a2ea0ac39a74771d54b91522728f6092b0ef897/pandas/tests/test_generic.py#L603) `downcast` is misspelled `donwcast`, but no error is raised, because of how `**kwargs` is used.

[core/categorical.py:sort](https://github.com/pydata/pandas/blob/0a2ea0ac39a74771d54b91522728f6092b0ef897/pandas/core/categorical.py#L945) `na_position` is not passed to `self.order`.

[core/categorical.py:fillna](https://github.com/pydata/pandas/blob/0a2ea0ac39a74771d54b91522728f6092b0ef897/pandas/core/categorical.py#L981) `Categorical.fillna` does not support `inplace` as opposed to say `Series.fillna` or `DataFrame.fillna`, but `**kwargs` being there causes below code to just silently ignore passed argument:

```
>>> ca = pd.Categorical([0, 0, 2, 2, 3, np.nan])
>>> ca.fillna(0, inplace=True)
[0, 0, 2, 2, 3, 0]
Categories (3, int64): [0 < 2 < 3]
>>> ca
[0, 0, 2, 2, 3, NaN]
Categories (3, int64): [0 < 2 < 3]
```

[This test](https://github.com/pydata/pandas/blob/0a2ea0ac39a74771d54b91522728f6092b0ef897/pandas/tests/test_groupby.py#L4418) is basically testing that you can specify _any_ keyword argument and it is just silently ignored. I cannot think of any case that this would be a good idea, or has any use case.

multiple instances where `*args` or `*kwargs` are not passed to inner function or have any use at all, as can be seen in the PR diff.


Labels:
name: API Design
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@behzadnouri some of these changes are ok, others I have commented on. These need some tests, otherwise someone else will come along one day and change it back because they need to pass an argument thru. So don't need a lot but a small set of API compat tests that test passing unknown / numpy compat args (e.g. out/keepdims) as appropriate / as needed.

We would need a release note documenting any actual API changes here (one that a user could see, e.g. the ones you enumerated at the top of the PR)

I reverted some of the changes mainly regarding decorating numpy functions, etc. and added a test.

other changes seems ok to me:

[this](https://github.com/pydata/pandas/blob/e9ee47c863a68acfaa3d824d88c450c1e48de473/pandas/core/generic.py#L1911) is the only place i see where defines `__finalize__`. as far as i can say, there is no need for `**kwargs`, and it would not cause any api compatibility issue. i would suggest removing `**kwargs`.

as for `argsort`, other instances of `argsort` pass in arguments; for example: [here](https://github.com/pydata/pandas/blob/26129f56bf0cc6ec91b2f694d6ef341ded2d1e84/pandas/core/index.py#L1148) in the same file; or [here](https://github.com/pydata/pandas/blob/26129f56bf0cc6ec91b2f694d6ef341ded2d1e84/pandas/core/categorical.py#L843) in `categoricals.py`. similarly for `apply`.

I would suggest to keep these changes;

@behzadnouri can you rebase this

@jreback rebased

merged via https://github.com/pydata/pandas/commit/38021ca83d5c85d27236ebc04865f936d2a82f99

thanks!
DOC: Suppress warnings in visualization.rst
State: closed

Closes #8234. Sorry to take a long.

I understand there are 2 warnings:
1. Warning in /Users/sin/Documents/Git/pandas/doc/source/visualization.rst at block ending on line 1275
   This is raised from `mpl` as plotting empty axes for demonstration purpose. Thus simply suppressed.
2. Warning in /Users/sin/Documents/Git/pandas/doc/source/visualization.rst at block ending on line 1312
   This is a warning displayed when passing multiple axes. Added explanations.


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
@jorisvandenbossche @TomAugspurger ?

@sinhrks these warnings also appear when running the tests, so probably there are also some of these calls that can be fixed (see https://travis-ci.org/pydata/pandas/jobs/42314766)

@sinhrks @jorisvandenbossche this ready to merge?

just the tests, it would be good to take a look at that as well

@sinhrks can you see if you can fix the test warnings as well?

@jorisvandenbossche @sinhrks ?

@jorisvandenbossche ?

bumbed to 0.16, as you don't visually see it in the docs (warning is not shown by sphinx), so no blocker

Sure, will fix tests also.

Updated, and prepared a separate fix #9278 for a program change.

@jorisvandenbossche OK, now all the warnings should disapear. Pls check.

Just looking at the travis log, I still see some warnings. Eg (in the log of the third job: https://travis-ci.org/pydata/pandas/jobs/48128130):

```
..................S.................................................../home/travis/build/pydata/pandas/pandas/tools/plotting.py:3225: UserWarning: To output multiple subplots, the figure containing the passed axes is being cleared

"is being cleared", UserWarning)

........../home/travis/build/pydata/pandas/pandas/tools/plotting.py:3208: UserWarning: When passing multiple axes, sharex and sharey are ignored.These settings must be specified when creating axes

"These settings must be specified when creating axes", UserWarning)

./home/travis/miniconda/envs/pandas/lib/python2.7/site-packages/matplotlib/axes.py:4747: UserWarning: No labeled objects found. Use label='...' kwarg on individual plots.

warnings.warn("No labeled objects found. "
```

It is not clear to me if you can see which test triggered the warnings.

@sinhrks add some questions

@TomAugspurger can you also give this a brief look?

@TomAugspurger Thanks to confirm. I think everything done now.

OK. I've only been able to give it a quick glance, but from that everything looks fine.

Regarding my comment above (https://github.com/pydata/pandas/pull/8877#issuecomment-71401640) the `No labeled objects found. Use label='...' kwarg on individual plots.` is still present in the tests, but this is more a leftover not addressed in #9278, so see if you can look at it

Yes, will do. I originally misunderstood you're referring to different line.

@sinhrks This is ready to merge for you?

In any case, thanks for your endurance, this has been a long one ..

@jorisvandenbossche Yes. I expect all warnings should be disappears after this and #9464, otherwise will implement a fix in #9464.

Thanks! 
Maybe then rebase #9464 to be sure
BUG: Panel4D setitem by indexer to Series fails for mixed-dtype
State: closed

This is another tweak to `_setitem_with_indexer` that allows it better to comprehend higher dimensional frames of mixed-dtype. With this tweak setting by indexer to a Series now works. Unfortunately setting to a DataFrame is still broken. I plan to open an issue for that.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls squash, otherwise looks good

closing pls reopen if/when updated
BUG: type change breaks BlockManager integrity
State: closed

closes https://github.com/pydata/pandas/issues/8850

on master:

```
>>> cols = MultiIndex.from_tuples([('1st', 'a'), ('2nd', 'b'), ('3rd', 'c')])
>>> df = DataFrame([[1.0, 2, 3], [4.0, 5, 6]], columns=cols)
>>> df['2nd'] = df['2nd'] * 2.0  # type change in block manager
/usr/lib/python3.4/site-packages/numpy/lib/function_base.py:3612: FutureWarning: in the future negative indices will not be ignored by `numpy.delete`.
  "`numpy.delete`.", FutureWarning)
>>> df.values
...
  File "/usr/lib/python3.4/site-packages/pandas-0.15.1_72_gf504885-py3.4-linux-x86_64.egg/pandas/core/internals.py", line 2392, in _verify_integrity
    tot_items))
AssertionError: Number of manager items must equal union of block items
# manager items: 3, # tot_items: 4
>>> df.blocks
...
  File "/usr/lib/python3.4/site-packages/pandas-0.15.1_72_gf504885-py3.4-linux-x86_64.egg/pandas/core/internals.py", line 2392, in _verify_integrity
    tot_items))
AssertionError: Number of manager items must equal union of block items
# manager items: 3, # tot_items: 4
```

`._data` is also broken:

```
>>> df._data
BlockManager
Items:       
1st  a
2nd  b
3rd  c
Axis 1: Int64Index([0, 1], dtype='int64')
FloatBlock: slice(0, 1, 1), 1 x 2, dtype: float64
IntBlock: slice(1, 3, 1), 2 x 2, dtype: int64
FloatBlock: slice(1, 2, 1), 1 x 2, dtype: float64
```

integer block is bigger than what it should be and overlaps with one of the float blocks.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
cc @immerrr

Yeah, I guess I didn't expect loc to be a slice in `BlockManager.set`.

Such corruption would be also a lot easier to prevent if blkloc invalidation (`self._blklocs[blk.mgr_locs.indexer] = -1`) used `np.info(self._blklocs.dtype).min` instead of `-1` which is a valid indexer most of the time.

@behzadnouri thanks

@immerrr I'd u would like to post an issue about this invalidation go ahead
DOC GH3508 (bis) added basic documentation of google analytics  in remot...
State: closed

added the GA doc to the correct file: "remote_data.rst"
closes #3508


Labels:
name: Docs

Comments:
actually if you would add a release note (in Enhancements) in v0.15.2, with a pointer to the docs (e.g. see here)....then people will know about the new docs.

thanks

Added entry in whatsnew for v15.0.2.

For the rest, looks good to me! 
Can you squash both commits into one?

merged via https://github.com/pydata/pandas/commit/9dd675b7a7f664f0d7773180c77d6f722d0647a5

thanks!

@benoitpointet can you run this PR

https://github.com/pydata/pandas/pull/8785

and see if it removes the warnings that were previously generated. 

lmk

Tested, it does indeed. It however does not remove the index_col bug described in #8773.

k thanks...

#8773 is not a bug, or did this use to work? (e.g. has the format changed)? The user needs to specify the `index_col`?

it used to work without​ having to provide the index_col argument.
b.

if you want to somehow infer that the index_col is 0 that would be great!

(e.g if `index_col=None` is the default), then just try it, if it fails, then you can try with `index_col=0`). IF its unambiguous.

Ideally would have tests for this.....

I'm unsure about hacking the `read_ga` function to infer such things.

If you look at the behaviour of the `index_col` param of the `read_csv` function for example, it support values such as `index_col=['start', 'end']` or `index_col=1`. That for me should be the default behaviour of the index_col param in read_ga. I'm still having a hard time figuring out why it does not behave so ...
BUG: Bug in csv parsing when passing dtype and names and the parsed data is a different data type (GH8833)
State: closed

closes #8833 

```
In [5]: data = """1.0,a
   ...: nan,b
   ...: 3.0,c
   ...: """

In [8]: read_csv(StringIO(data),sep=',',names=['A','B'])
Out[8]: 
    A  B
0   1  a
1 NaN  b
2   3  c

In [9]: read_csv(StringIO(data),sep=',',names=['A','B'],dtype={'A' : int})
ValueError: cannot safely convert passed user dtype of <i8 for float64 dtyped data in column 0
```


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
@cpcloud 
ok, I put a more informative error message up (so your original example will pass thru just fine, converting to int, this is a non-convertible case)

Course numpy is annoying (this is 1.9.1). You would think that the casting a == float to an int is safe :)

```
In [3]: np.array([1.0,2.0,3.0]).astype('i8',casting='unsafe')
Out[3]: array([1, 2, 3])

In [4]: np.array([1.0,2.0,3.0]).astype('i8',casting='safe')
TypeError: Cannot cast array from dtype('float64') to dtype('int64') according to the rule 'safe'

In [5]: np.array([1.0,np.nan,3.0]).astype('i8',casting='unsafe')
Out[5]: array([                   1, -9223372036854775808,                    3])

In [6]: np.array([1.0,np.nan,3.0]).astype('i8',casting='safe')
TypeError: Cannot cast array from dtype('float64') to dtype('int64') according to the rule 'safe'
```

cc @mwiebe 

The tricky thing here is that NumPy only has the types to make this determination, not the values, so from the perspective of only knowing that it is float -> int, it is not a safe cast. The error mode in dynd is an attempt to provide a value-based way to handle this, by inserting code to check the values as float is converted to int.

@mwiebe and that is exactly what I do

will be trying out dynd for real with ?int64 after pydata
when I have some time!
Fix unrecognized 'Z' UTC designator
State: closed

closes #8771 


Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
This time PR is green ;-)

@broessli update on this?

@jreback I've been very busy and I think some thoughts and refactoring are
in order to fix all this properly (I don't like the idea of reusing
out_local), but unfortunately I don't have time to do it right now. I've
corrected the behavior of the internal parser regarding 'Z' specifier, but
the issue raised here has to do with the way the internal and
fallback/external parser are interacting which to me needs some non trivial
refactoring

Le vendredi 21 novembre 2014, jreback notifications@github.com a écrit :

> @broessli https://github.com/broessli update on this?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/8832#issuecomment-64034328.

## 

Benoît.

@broessli ok, pls squash / rebase. Let's push thru this change as is, then pls add another issue (which I''ll mark for the future), to possibly handle the 2 error cases that we discussed above (list them in the top of the issue so you or whomever gets to it won't forget).

thanks

@jreback rebase done, PR green, sorry for the delay, have been away with no access to my home computer

thanks!
BLD: fix odd usage of cython imports, xref (GH8602)
State: closed

closes #8602 

easy enough, that import should not have been there. we don't import from other pyx files unless its explicity exported (which is in very few cases).


Labels:
name: Build
description: Library building on various platforms

Comments:
closed via https://github.com/pydata/pandas/commit/d881ba259a98af23f58cd278dce2e32a3e8da05e