BrandonFors/pandas-github-issues huggingface dataset:
GH 9016: Bitwise operation weirdness
State: closed

Series now supports bitwise op for integral types. 

I have made the changes in wrapper() itself instead of na_op() because it looked to me like wrapper is controlling the input and output fill and dtype. Once that is taken care of, na_op() seems to be doing the right thing by itself and so I did not have to change anything to get back the expected behavior. Happy to change if things need to be altered toward better design etc.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
This PR intends to resolve #9016

can u move the tests near the other boolean tests (right below) in test_series

also need some test for ops with NaN and series containing nan and ops with other dtypes
(with other dtypes should prob raise TypeError) 

@shoyer, @jreback  thanks, I am working on the suggested changes. In the meantime I noticed this. (using master without this patch).

```
In [4]: Series([0,1,2,3]) & Series([True, True, True, True])
Out[4]:
0    False
1     True
2    False
3     True
dtype: bool
```

I was expecting output to be `Series([False, True, True, True])` as RHS has dtype bool. 
Figured Numpy converts bool to 0/1 and executes bitwise op. For e.g.,

```
In [5]: import numpy as np

In [6]: np.array([0,1,2,3]) & np.array([True, True, True, True])
Out[6]: array([0, 1, 0, 1])

In [7]: np.array([0,1,2,3]) | np.array([True, True, True, True])
Out[7]: array([1, 1, 3, 3])
```

In the Series example, should the behavior be preserved? In that case should the output dtype be bool or int64? Otherwise it looks confusing.

Looks like if changed, `test_logical_with_nas()` in test_frame.py would break.

made the following changes:
- Code refactored to remove some duplication
- Moved test function near other comparison tests
- Added more tests with NaN and other dtypes
- Fixed code so it works with list,tuple and np.ndarray types correctly.

@tvyomkesh this seems like a lot of branching for just slightly different dtype handling. see what you can do about that.

@jreback thanks for pointing to `core.common.is_interger_dtype()`. Made the changes. Also cut down on branching a bit. Updated release note with a small example.

I still think this is confusing.

```
In [3]: pd.Series([2]) & pd.Series([True])
Out[3]:
0    False
dtype: bool
```

Either both the inputs should be converted to dtype boolean in which case output value should be True  or if both the inputs are converted to dtype int64 implictly and bitwise & is carried out per current behavior, then the output dtype should remain int64 and not be coerced to bool. The current patch can be modified to fix this. Thoughts?

PR review changes are done. Good to merge?

@jreback thanks. I have made the changes.

@jreback and @shoyer I have pushed the changes. Thanks.

thanks @tvyomkesh
Fix missing text on Index documentation page.
State: closed


Labels:
name: Docs

Comments:
Thanks! However, I think, looking at the fix in sphinx 1.2.2 (https://bitbucket.org/birkenfeld/sphinx/pull-request/218/closes-1374-autosummary-generates-overly/diff), it should be enough to put a full stop at the end of the sentence to have the same effect and not having to go over the 80 char line width.

Nice, I'll do that now

@graingert thanks!
Fix docs in groupby.tail
State: closed

The old docs is wrong where head() & tail() return the same result.
Change input of the example to see group data clearer.


Labels:
name: Docs
name: Groupby

Comments:
@kidphys ping when you update this.

@kidphys so can you remove the `as_index=False`? And can you then also squash the commits? Apart from that, ready to merge!

Thanks @kidphys
FIX: Fixed some issues around vb_suite
State: closed

I'm new to vb_suite, and ran following command according to [this document](https://github.com/pydata/pandas/wiki/Performance-Testing).

```
./test_perf.sh -b master -t HEAD
```

But I got some errors (`InvalidGitRepositoryError` and Benchmark duplication errors), and finally I got tests run by fixing following issues.
- VB_DIR pointed wrong directory
- temporary variable `bmark` in groupby was collected as Benchmark
- indexing module had same name ('series_getitem_scalar') Benchmarks

I'm not so familier with vb_suite, and any feedbacks are welcome ! Thanks.


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
was it hanging on something particular?

fyi, you can also run with `-r group` for example to only run some of the benches

Sorry for mashing up multiple modifications in one pull request. Problems I got are like commented above. As to `indexing` module, I got unexpected result now, and so I'll investigate on it and comment about it later. **It seems that, if you already have a correct database, you doesn't get UNIQUE Constraint issue above.**

> fyi, you can also run with -r group for example to only run some of the benches

thanks.

> As to indexing module, I got unexpected result now, and so I'll investigate on it and comment about it later.

Investigated, and added commit for that.

@ledmonster thanks!
BUG 9188: concat of all-nan with empty frame produces object dtype
State: closed

My first stab at this issue. Would wait for comments and inputs. This is the behavior without the fix.

```
In [4]: df_1 = pd.DataFrame({"Row":[0,1,1], "EmptyCol":np.nan, "NumberCol":[1,2,3]})

In [5]: df_2 = pd.DataFrame(columns = df_1.columns)

In [6]: df_concat = pd.concat([df_1, df_2], axis=0)

In [7]: df_1.dtypes
Out[7]:
EmptyCol     float64
NumberCol      int64
Row            int64
dtype: object

In [8]: df_2.dtypes
Out[8]:
EmptyCol     object
NumberCol    object
Row          object
dtype: object

In [9]: df_concat.dtypes
Out[9]:
EmptyCol      object
NumberCol    float64
Row          float64
dtype: object

In [10]: df_concat
Out[10]:
  EmptyCol  NumberCol  Row
0      NaN          1    0
1      NaN          2    1
2      NaN          3    1

In [11]: df_1
Out[11]:
   EmptyCol  NumberCol  Row
0       NaN          1    0
1       NaN          2    1
2       NaN          3    1

In [12]: df_2
Out[12]:
Empty DataFrame
Columns: [EmptyCol, NumberCol, Row]
Index: []
```

Seeing this after the fix.

```
In [3]: df_1 = pd.DataFrame({"Row":[0,1,1], "EmptyCol":np.nan, "NumberCol":[1,2,3]})

In [4]: df_2 = pd.DataFrame(columns = df_1.columns)

In [5]: df_concat = pd.concat([df_1, df_2], axis=0)

In [6]: df_1.dtypes
Out[6]:
EmptyCol     float64
NumberCol      int64
Row            int64
dtype: object

In [7]: df_2.dtypes
Out[7]:
EmptyCol     object
NumberCol    object
Row          object
dtype: object

In [8]: df_concat.dtypes
Out[8]:
EmptyCol     float64
NumberCol    float64
Row          float64
dtype: object

In [9]: df_1
Out[9]:
   EmptyCol  NumberCol  Row
0       NaN          1    0
1       NaN          2    1
2       NaN          3    1

In [10]: df_2
Out[10]:
Empty DataFrame
Columns: [EmptyCol, NumberCol, Row]
Index: []

In [11]: df_concat
Out[11]:
   EmptyCol  NumberCol  Row
0       NaN          1    0
1       NaN          2    1
2       NaN          3    1
```

However this is causing `test_partial_setting_mixed_dtype` test to fail because after the fix dtypes change for df.

```
In [2]: df = DataFrame(columns=['A','B'])

In [3]: df.loc[0] = Series(1,index=range(4))

In [4]: df1 = DataFrame(columns=['A','B'],index=[0])

In [5]: df.dtypes
Out[5]:
A    float64
B    float64
dtype: object

In [6]: df1.dtypes
Out[6]:
A    object
B    object
dtype: object
```

whereas before the fix, the test was happy as all dtypes were same.

```
In [2]: df = DataFrame(columns=['A','B'])

In [3]: df.loc[0] = Series(1,index=range(4))

In [4]: df1 = DataFrame(columns=['A','B'],index=[0])

In [5]: df.dtypes
Out[5]:
A    object
B    object
dtype: object

In [6]: df1.dtypes
Out[6]:
A    object
B    object
dtype: object
```

Is it expected to preserve the test behavior?


Labels:

Comments:
freezing this.
BUG: Fixes #9281: fixes to tseries.tests.test_tslib.TestTimestamp
State: closed

closes #9281 

The last test of both test_now and test_today was passing for anyone with a timezone
of UTC-5 or greater (but failed, for example, in US/Pacific).  The test was not
testing what the original author meant it to (which is that the times are very close
together) so I added abs(.) around the Timedeltas and also fixed the errant test.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks!

I don't think abs() is necessary here given the order that the times are created so the difference will always be positive.  Nevertheless, thanks for fixing.

@rockg, yeah, I wasn't too sure about the `abs`, but I was thinking it makes the tests more likely to catch some problematic code that someone writes in the future.  For example, take

```
self.assertTrue(abs(ts_from_method_tz - ts_from_string_tz) < delta)
```

If someone ends up rewriting `Timestamp.now(tz='US/Eastern')` and it erronesously causes `ts_from_string_tz` to have a value corresponding to the user's local time instead of US/Eastern, then because of `abs`, the test will fail not only for users with a timezone less than UTC-5, but rather for everyone.
Update plotting.py (issue #2916)
State: closed

Added ax docstring. 
Address issue generated by `scripts/find_undoc_args.py`:
[+2892 tools/plotting.py boxplot_frame_groupby()](https://github.com/pydata/pandas/blob/master/pandas/tools/plotting.py#L2892): Missing[1/9]=['ax']


Labels:
name: Docs

Comments:
Thanks!
TST: tests for GH4862, GH7401, GH7403, GH7405
State: closed

closes https://github.com/pydata/pandas/issues/4862
closes https://github.com/pydata/pandas/issues/7401
closes https://github.com/pydata/pandas/issues/7403
closes https://github.com/pydata/pandas/issues/7405

minor code change to https://github.com/pydata/pandas/pull/9061; otherwise only tests. the code change is to avoid https://github.com/pydata/pandas/issues/9170 issue.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: MultiIndex

Comments:
@jreback took `_shallow_copy/Index` to a separate functions, and removed `check_dtype=False`. 

@behzadnouri ok this looks good. and tests ok on windows.

can you add a release note (you can put all the issues on the same one I think). ping when pushed.

@jreback done!

thanks! a 4-some! excellent
BUG: Adding nano offset raises TypeError
State: closed

Closes #9284.


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
this looks good
ready to merge?

Let me add explicit tests for backward compat and return type checks (pointed in #9309). I expect to finish it today.

@jreback It's ready. Pls review.

can u test a pickle from a previous version working?

you will have to use say 0.15 and run the test script to generate data 

you may need to modify the test script to pickle with a freq on the Timeseries

this is the generate_test_data in the pandas/io/tests/generate_legacy_pickles

thanks @sinhrks
GH 9273: Timedelta constructor should accept nanoseconds keyword.
State: closed

closes #9273 

This patch adds nanoseconds keyword support in Timedelta constructor.

<pre>
In [1]: from pandas import Timedelta

In [2]: td = Timedelta(nanoseconds=1)

In [3]: td1 = Timedelta(microseconds=1)

In [4]: td.components
Out[4]: Components(days=0, hours=0, minutes=0, seconds=0, milliseconds=0, microseconds=0, nanoseconds=1)

In [5]: (td + td1).components
Out[5]: Components(days=0, hours=0, minutes=0, seconds=0, milliseconds=0, microseconds=1, nanoseconds=1)
</pre>


Labels:
name: API Design
name: Timedelta
description: Timedelta data type

Comments:
minor comment. ping when you have pushed

yes, much better that way. done. thanks.

thanks!
ENH: Corrects to_html print spacing GH4987
State: closed

Closes #4987 

This is my first non-doc PR, so please bare with me.  Is it worth my while to go through the tests and standarize all of the test HTML to all block-strings, or is the mix of individual lines/blocks ok?


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
does this actually change the way the output looks? 

As far as I know it only strips off the leading/trailing spaces in the HTML data rows. Those spaces shouldn't be rendered by a browser?

@jorisvandenbossche @cpcloud @TomAugspurger look ok?

@clham can you add a release note? (put it under API changes section in whatsnew, just a single line and reference the underlying issue)

 @jreback, Should be good to go. I added a plain api section. Current version only
lists breakages.

@TomAugspurger @jorisvandenbossche any problem with this?

Seems OK to me

But I think this should not be listed in API changes, but can be considered as a bug fix. 
And it needs a rebase.

merged via https://github.com/pydata/pandas/commit/ed45e6eeab43c1abf7b541fd659dada12172b52c

thanks!

Thanks a lot for this, I had some problems letting pandoc convert such html and was doing a `re.sub(' +',' ', html)`but after this it isn't nessesary anymore :-)
ENH: StringMethods supports is_xxx methods
State: closed

Derived from #9111.

Add following methods to be compat with standard `str`.
- StringMethods.isalnum
- StringMethods.isalpha
- StringMethods.isdigit
- StringMethods.isspace
- StringMethods.islower
- StringMethods.isupper
- StringMethods.istitle


Labels:
name: API Design
name: Strings
description: String extension data type and string data

Comments:
since these functions are quite similar. can you change impl to use a function generator to create these? (with the appropriate doc strings)

It may also be possible to include some of the existing functions in a similar generator.

Right, how about this?

looks good. can you add the methods here as well?

http://pandas.pydata.org/pandas-docs/stable/text.html#method-summary (maybe we want to auto-generate this as well from the StringMethods?)

@jreback Sure. Did it.

this needs to go after #9322 (and might need a rebase after that) as the docs are changing a bit)

OK, rebased.

thanks @sinhrks
BUG: Subplotting boxplot shows unnecessary warnings
State: closed

Related to #8877 and 80a730c93717e7fc01ae2f880109bc752519cecf

Because `legend=True` is default, `DataFrame.plot(kind='box', subplots=True)` shows unnecessary warnings.

```
# OK, no legend / no warnings
df.plot(kind='box')
```

```
# NG: no legend and warnings
df.plot(kind='box', subplots=True)
# UserWarning: No labelled objects found. Use label='...' kwarg on individual plots.
#   warnings.warn("No labelled objects found. "
```


Labels:
name: Visualization
description: plotting

Comments:
Looking at the travis doc build, the warning is still there.

Looks the same warning is raised by hexbin also. Let me fix.

Looks fixed. Other warnings are unrelated to this change, and being fixed in #8877.

https://travis-ci.org/pydata/pandas/jobs/47391584

Great. It does look like all the no labelled objects errors have been fixed.

Thanks!
allow for empty SparseSeries SparsePanel constructors
State: closed

Closes #9272.

Note that my solution is to provide a default `data=[]` for SparseSeries and `frames={}` for SparsePanel. This differs from the dense implementations that have `None` defaults that are then changed to the above. Doesn't seem like a big deal to me, but I can do that here if needed.


Labels:
name: API Design
name: Sparse
description: Sparse Data Type
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls add a release note as well (call this a bug fix).

I'm guessing that this was buried under more important things, but let me know if you'd like any additional changes.

merged via https://github.com/pydata/pandas/commit/391f46af129d214056ec9109caa4c53b7dddfb04

thanks!
FIX: Add endianness missing flag when reading data
State: closed

Added endianess flat to data type to allow data to be read cross platforms

closes #8688


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Stata
description: read_stata, to_stata

Comments:
@yarikoptic @jreback I think this one is ready.

@yarikoptic ok on these platforms?

@bashtage IIRC the tests are disabled on non-small so is this actually tested?

All of the tests were re-enabled for big-endian a few releases ago.

There are only three tests that are skipped on any platform:

https://github.com/pydata/pandas/blob/master/pandas/io/tests/test_stata.py#L256

I would like to kill these in a different PR since they rely on proprietary data and so will never be re-enabled.

ok, ping on green

@jreback It is ready.

thanks!
API: restore full datetime.timedelta compat with Timedelta w.r.t. seconds/microseconds accessors (GH9185, GH9139)
State: closed

closes #9185
closes #9139 


Labels:
name: API Design
name: Timedelta
description: Timedelta data type
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@shoyer @jorisvandenbossche 

I think this should bring us back to the previous behavior.

I think we also need to update the timedelta docs: http://pandas.pydata.org/pandas-docs/stable/timedeltas.html

all comments have been addressed.

OK, looks good to me!

Thanks Jeff!
BUG: bug in multi-index where insert fails
State: closed

closes https://github.com/pydata/pandas/issues/9250

on master:

```
>>> df
         3rd
1st 2nd     
a   b      0
b   d      1
>>> df.loc[('b', 'x'), '3rd'] = 2  # this works!
>>> df
         3rd
1st 2nd     
a   b      0
b   d      1
    x      2
>>> df.loc[('b', 'a'), '3rd'] = -1  # fails! sets everything to -1
>>> df
         3rd
1st 2nd     
a   b     -1
b   d     -1
    x     -1
>>> df.loc[('b', 'b'), '3rd'] = 3  # erros!
NotImplementedError: Index._join_level on non-unique index is not implemented
```

on branch:

```
>>> df
         3rd
1st 2nd     
a   b      0
b   d      1
>>> df.loc[('b', 'x'), '3rd'] = 2
>>> df.loc[('b', 'a'), '3rd'] = -1
>>> df.loc[('b', 'b'), '3rd'] = 3
>>> df
         3rd
1st 2nd     
a   b      0
b   d      1
    x      2
    a     -1
    b      3
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
this feels like it should also fix #9250 (just reported), but does not...if you have a chance

@jreback updated the pr to handle https://github.com/pydata/pandas/issues/9250 as well

@behzadnouri thanks! gr8 as usual!
ENH: add Timestamp.to_datetime64
State: closed

This PR adds a `Timestamp.to_datetime64()` method to complement the `Timedelta.to_timedelta64()` method I added in #8884. It is a continuation of the aborted #8916.

Arguably, there should also be the alias `Timestamp.values` to complement the series property but I haven't added that yet.


Labels:
name: Datetime
description: Datetime data dtype
name: API Design

Comments:
hmm, its called `Timestamp.value`; you think `.values` should also be there? why? (its not a big deal, but curious why)

`Timstamp.value` is an integer; `.values` would be a `np.datetime64` object, like the datetime64 `.values` on arrays.

hmm that could be useful

Any comments on this PR? (Tests are green)

trivial comment. looks good otherwise. you are picking up pieces from those other pr's yes?

@jreback By other PRs do you mean #8916? Yes, probably, eventually. Not entirely sure it's possible though given numpy datetime64 limitations :).

I wrote this one because I need it for the next PR I'm submitting tonight.

@shoyer yep. you had some good stuff in there IIRC.
FIX: Fix encoding to allow StataReader to read urls
State: closed

Fix encoding so that StataReader can correctly read URLs

closes #9231


Labels:
name: Unicode
description: Unicode strings
name: IO Stata
description: read_stata, to_stata

Comments:
This fix is not finished - mostly using it to document the issue.  The StringIO is returning unicode while unpack expects bytes (or string on Python 2).  This simply encodes the unicode to a new StringIO object, and is very ugly.

@jreback @jseabold Should be fixed now. I left in the url read for now simply to to verify once that it works.  I will delete this basic test once it goes green.

One timed out, the others all passed, so it seems ready, barring other issues in the method I used.

minor comment, I restarted the stalled job

@jreback Looks good.  Let me know if you need me to rebase.

@bashtage thanks!
DOC: update docstring of DataFrame.append
State: closed

I always thought that `append` was for appending rows, not columns. So, unless I am completely missing something, the docstring was saying the wrong thing.

So I reworded it a bit, see below. 
Apart from that, there were some other things that are not fully clear to me in the docstring:
- The docstring says that `other` should be a "list of Series/dict-like objects" (or a Dataframe):
  - Does it need to be a list? It also works with just a Series. Or what are the differences between both?
  - With a simple example, I don't see a difference, although in implementation, there is.
- The docstring says for the `ignore_index` kwarg: "Useful for gluing together record arrays"
  - How does this look in practice? If I try to append a record array, I get "TypeError: cannot concatenate a non-NDFrame object" -> so I will remove this


Labels:
name: Docs

Comments:
see #8295 / #8337 

the axis argument _should_ be added to `append` to make this symmetric with `concat`.

The reason for the accepting of say a list is that if it _can_ be interpreted correctly in the context of a Dataframe (e.g. it matches the columns and can be directly converted; this is NOT true for concat).

I don't know about record arrays, that may be a 'feature' that was not implemented. 

I updated this (for how the function now is, so without the axis argument). Only the first sentence (about adding rows) will have to be reverted when the other PR is rebased/merged

Removed the sentence about record arrays (does not make sense as far as I can see) 
Added a see also and example, and some other small things.

:+1: 

@jorisvandenbossche lgtm
COMPAT: Need to read Bytes on Python
State: closed

Reading from s3 on Python 3 returns a Bytes object.


Labels:

Comments:
Yep, thanks.

I just tested this out, and Py3 needs it. I'm not sure this can be tested well with CI... but could you add a release note in `doc/source/whatsnew/v0.16.0.txt`?

Sure. Done.

Thanks.